Skip to content
Welcome and Introduction (5 minutes)Greet participant and make them comfortableExplain the purpose of the test and overall processThis group has had the overall concept explained to them beforeConsent and Pre-Survey (10 minutes)Have participant sign consent form and liability waiver if anyAdminister pre-survey questionnaireVR Setup (5 minutes)Help participant put on VR headsetGuide through basic controls and interactionsEnsure comfort and proper fitVR Experience (15 minutes)Allow participant to go through the Advanced Water Purification experienceObserve and take notes using the Observational Data Collection WorksheetIntervene only if absolutely necessaryPost-Experience Debrief (20 minutes)Ask immediate reactions and feelingsAddress any discomfort or issues experiencedPost-Survey and Wrap-up (10 minutes)Administer post-survey questionnaireAnswer any final questions
It seems possible to automatically test VR apps for ease of use, without needing someone to watch the whole timeCombining different types of data (movements, speech, etc.) works better than just using one typeMore research is needed, but this is a promising start for testing VRNovices: Those who had never or hardly used VR before.Experts-people who had lots of experience with VR
Both beginners and experts found the tutorial easy to use and enjoyable.Beginners needed the instructions repeated more often than experts.Experts felt more confident moving around in the virtual world.Beginners found the tutorial more mentally challenging, but also more fun (probably because it was new to them).Most people could do the tasks they learned pretty easily after the tutorial.Very few people felt sick or uncomfortable from using VR.
The VR experience was generally rated positively by participantsFeelings of presence and social presence were strongly related to most aspects of user experienceHealth-related measures like simulator sickness didn't show strong relationships with user experienceSome measurements taken during the experience didn't match up well with those taken after
How did you feel during the VR experience?
Follow upWere there any moments when you felt particularly immersed or engaged, as if you were physically present in the experience? Did you experience motion sickness, dry eye, head ache or dizziness?What did you think about the 3D art? can you explain that more? Can you share your favorite things about the experience? Can you share the things you didn’t like about the experience? Follow up: Was there anything you felt was distracting in the experience?Where did the experience feel difficult to navigate or understand?Can you tell me more about what made it difficult? (try to get people to elaborate on the more difficult navigation or interaction moments) Did you have ever find yourself disinterested in the story, or did you experience difficulty relating to the characters or events? (pause for response) If so, at what point?Do you think the decisions you made within the experience (game play, exploring the space, touching buttons) had an effect on what came next? Did you feel like the characters acknowledged you and responded to you in the VR?Was there a particular character that you really liked? Why did you like them? What new information did you learn about the water treatment process?How do you think this knowledge will impact your views on recycled water?Optional: What’s another thing you learned about water treatment? (this question is here if you need more information from the user, skip it if already answered by participant)How do you feel about drinking recycled water? Did the VR experience change your perspective on drinking recycled water?If yes, in what ways?Is there anything about the game that would make you recommend it to others?Did you feel that the VR element added significantly to the gameplay?
How did you feel during the VR experience?
Follow upWere there any moments when you felt particularly immersed or engaged, as if you were physically present in the experience? Did you experience motion sickness, dry eye, head ache or dizziness?Did the 3D art feel realistic? What were some of your favorite things about the experience? What were some things you didn’t like about the experience? Think back to the net gun, contaminant capture game. Was that game fun and interesting to you?Could you easily understand how it worked, or did it take you a while to figure out what you were supposed to do? How do you think we could have made that game more fun or easier to understand?What else would you like to share about your experience with this game?Think back to the reverse osmosis, water pressure game. Was that game fun and interesting to you?Could you easily understand how it worked, or did it take you a while to figure out what you were supposed to do? How do you think we could have made that game more fun or easier to understand?What else would you like to share about your experience with this game?Think back to the UV Light Saber Filtration Game. Was that game fun and interesting to you?Could you easily understand how it worked, or did it take you a while to figure out what you were supposed to do? How do you think we could have made that game more fun or easier to understand?What else would you like to share about your experience with this game?Did you like the way the VR Experience ended? What new information did you learn about the water treatment process?How do you think this knowledge will impact your views on recycled water?How do you feel about drinking recycled water? Did the VR experience change your perspective on drinking recycled water?If yes, in what ways?If so, why? If no, why?
AWP VR Testing Protocol
Test Flow: This is the outline of the VR user test for Advanced Water Purification experience. It includes a brief introduction, consent and pre-survey, VR setup and experience, post-experience discussion, and a final survey. The process is designed to gather detailed feedback on the VR experience.
For Each Participant (60 minutes total)
Literature Review: To learn about the best ways to test, what problems people face, and what new ideas are emerging. To guide us in creating better tests that focus on user engagement and effectiveness, making the VR experience more fun for users.
Testing VR Experiences
Article/Paper Title
Aim
Methods
Results
To explore if it's possible to automatically test how easy virtual reality (VR) apps are to use, without needing a human observer watching the whole time.
Created a simple VR app with basic tasks like math problems and shape recognition
Had 14 people try out the app while wearing a VR headset
Tracked their head and hand movements, brain activity, speech, and how fast they did tasks
Had them fill out a survey about their experience afterwards
Using all the data together, they could automatically detect when there were issues in the VR app about 84% of the time
Key Takeaways:
Test how easy or difficult it was for people to use and learn from a VR tutorial.
Find out any differences between:
The Authors have prepared a VR tutorial teaching one how to perform an elementary set of actions within a virtual world, including walking by "teleportation," picking up and dropping objects, using buttons and levers, and gazing at objects for some time.
They invited 23 college students, novices and experts, to try out this tutorial.
After the tutorial, they got the students to do the following: Answer questionnaires concerning their feelings. They made these participants do some work in VR to see how much they had learned.
Explore how different ways of measuring user experience in virtual reality (VR) relate to each other.
The researchers wanted to see if traditional methods for evaluating 2D interfaces could be combined with VR-specific measures to create a better way to evaluate large-scale, multi-user VR experiences.
72 participants tried out a multi-user VR adventure in groups of 3
Participants filled out questionnaires before, during, and after the VR experience
The questionnaires measured things like presence (feeling of being there), social presence (feeling of being with others), user experience, and simulator sickness
The researchers looked at how the different measurements related to each other using statistical analysis
There are no rows in this table
Pre-Test Questionnaire
The pre-survey will include background information about the participant's experience with VR technology, comfort levels associated with headsets, interests of the participant, and their current understanding of the water treatment process.
Pre-Survey Questions
Section
Question
Type
Options
Demographics
Age, location, gender
Single Select
VR Experience and Comfort
On a scale of 1-5, how comfortable are you with using VR headsets?
Likert Scale
1-I’ve never used it before
2- I’ve used it once
3- I use it less than once a month
4- I use it a few times per month
5- I use VR weekly
If you’ve experienced VR before, what type of VR experiences do you typically enjoy?
Short Answer
Ex: Games, Educational, Entertainment etc.
Would you be interested in using VR for educational purposes?
Likert Scale
1- Not at all interested
2 -Sllightly interested
3- Neutral, neither interested nor uninterested
4- Moderately Interested
5- Extremely Interested
Water Treatment Knowledge
You are probably familiar with recycling when it comes to paper, aluminum cans, and cardboard, etc. Did you know that water can be recycled too?
Single Select
No, That sounds familiar but I don’t know much about it
Yes
What is your current level of understanding of the water treatment process?
Single Select
I know nothing
I have a basic understanding, I know a lot about it
I have an advanced understanding on it
How comfortable are you with the idea of drinking recycled water?
Likert Scale
1-Very uncomfortable
2- Somewhat uncomfortable
3 - Neutral
4- Somewhat comfortable
5-Comfortable
What is your definition of low quality water?
Short Answer
Expectations and Challenges
What do you want to see in a VR experience? Please rank your choices by order of most important to least important.
Grid Multiple Select
Immersion (defined as a feeling of being surrounded by and deeply engaged in an experience)
Realistic (like things would appear in real life)
Animated, Educational, Artistic, Easy to Use,
Interactive (defined as a two-way exchange where the user's actions influence or change the outcome)
Other: Short Answer]
There are no rows in this table
Post-Test Questionnaire
The post-survey will measure the effectiveness in VR through means of immersion, understandability, and comfort.
Post-Test Questions
Section
Question
Type
Options
Immersion and Comfort
How immersive did you find the VR experience? Note: Immersion in VR is defined as a sense that you are surrounded by or fully involved in a virtual space.
Likert Scale
1-Not Immersive at all
2-Slightly Immersive
3- Moderately Immersive
4- Very Immersive
5- Completely Immersive
Did you experience any discomfort during the experience?
Single Select
Short Answer
Yes
No
Explain
How would you rate the overall quality of this experience compared to other headset experiences you’ve seen?
Likert Scale
1 very low quality
2-Low quality
3-average quality
4- high quality
5- very high quality
Learning and Understanding
Has your understanding of the water treatment process changed after the VR experience?
Single Select
Yes/No
a. If yes, How?
b. If no, Why?
Now that you’ve experienced an educational VR, do you find it helpful as a tool for education?
Likert Scale
1- Not at all interested
2 -Sllightly interested
3- Neutral, neither interested nor uninterested
4- Moderately Interested
5- Extremely Interested
How would you rate the effectiveness of this VR in teaching you the Advanced Water Purification process?
Likert Scale
Not Effective, Somewhat Effective, Moderately Effective, Very Effective
Has your comfort level with drinking recycled water changed?
Single Select
Short Answer
Yes/No
a. If yes, How?
How easy or difficult was it to get used to this new form of technology?
Likert Scale
1- Very easy
2- Easy
3- Neutral
4- Difficult
5- Very Difficult
Engagement
Can you describe any memorable moments from this experience?
Short Answer
What aspects of the VR experience did you find most engaging or informative?
Short Answer
Usability and Navigation
Were there any parts of the experience that felt confusing or difficult to navigate?
Short Answer
How easy was it to follow the instructions or prompts within the VR experience? If yes, why?
Likert Scale
1- Very easy
2- Easy
3- Neutral
4- Difficult
5- Very Difficult
How intuitive did you find the controls?
Likert Scale
Not intuitive, Somewhat intuitive, Neutral, Intuitive, Very intuitive
Expectations and Improvements
Did anything during the VR experience turn out differently than you expected?
Short Answer
Was there anything in the experience you would have changed or improved?
Short Answer
There are no rows in this table
Debrief Session Scripts: Discuss verbally with the participants about the test session. This includes overall feelings, likes, dislikes, pain points and the VR game specific questions.
Script A: Holistic insights
Follow up:
Follow up:
Follow up:
Follow up
Follow up
Script B: game interaction focused
Follow up
Follow up
11. Were there moments where you felt you could predict what was coming next, or did events catch you off guard?
12. Did any tasks require more concentration than others? If so, can you tell us why?
13. Did interacting with virtual characters feel natural?
Follow Up
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.