Skip to content
Gallery
GIT 592- AWII Fall 2024
Table of Contents
Arizona Water Blueprint
Water SIMmersive Exhibit
AWP VR Testing Protocol
Water Game Data
Comparing AI-generated and human-written setup & troubleshooting instructions in a VR experience
Share
Explore

icon picker
AWP VR Testing Protocol

flow-chart
Test Flow: This is the outline of the VR user test for Advanced Water Purification experience. It includes a brief introduction, consent and pre-survey, VR setup and experience, post-experience discussion, and a final survey. The process is designed to gather detailed feedback on the VR experience.
For Each Participant (60 minutes total)
Welcome and Introduction (5 minutes)
Greet participant and make them comfortable
Explain the purpose of the test and overall process
This group has had the overall concept explained to them before
Consent and Pre-Survey (10 minutes)
Have participant sign consent form and liability waiver if any
Administer pre-survey questionnaire
VR Setup (5 minutes)
Help participant put on VR headset
Guide through basic controls and interactions
Ensure comfort and proper fit
VR Experience (15 minutes)
Allow participant to go through the Advanced Water Purification experience
Observe and take notes using the Observational Data Collection Worksheet
Intervene only if absolutely necessary
Post-Experience Debrief (20 minutes)
Ask immediate reactions and feelings
Address any discomfort or issues experienced
Post-Survey and Wrap-up (10 minutes)
Administer post-survey questionnaire
Answer any final questions

⁠

book-shelf
Literature Review: To learn about the best ways to test, what problems people face, and what new ideas are emerging. To guide us in creating better tests that focus on user engagement and effectiveness, making the VR experience more fun for users.
Testing VR Experiences
Article/Paper Title
Aim
Methods
Results
⁠
Usability Testing of Virtual Reality Applications-The Pilot Study
⁠
⁠
To explore if it's possible to automatically test how easy virtual reality (VR) apps are to use, without needing a human observer watching the whole time.
Created a simple VR app with basic tasks like math problems and shape recognition
Had 14 people try out the app while wearing a VR headset
Tracked their head and hand movements, brain activity, speech, and how fast they did tasks
Had them fill out a survey about their experience afterwards
Using all the data together, they could automatically detect when there were issues in the VR app about 84% of the time
Key Takeaways:
It seems possible to automatically test VR apps for ease of use, without needing someone to watch the whole time
Combining different types of data (movements, speech, etc.) works better than just using one type
More research is needed, but this is a promising start for testing VR
⁠
Usability Testing of a Virtual Reality Tutorial
⁠
⁠
Test how easy or difficult it was for people to use and learn from a VR tutorial.
Find out any differences between:
Novices: Those who had never or hardly used VR before.
Experts-people who had lots of experience with VR
The Authors have prepared a VR tutorial teaching one how to perform an elementary set of actions within a virtual world, including walking by "teleportation," picking up and dropping objects, using buttons and levers, and gazing at objects for some time.

They invited 23 college students, novices and experts, to try out this tutorial.

After the tutorial, they got the students to do the following: Answer questionnaires concerning their feelings. They made these participants do some work in VR to see how much they had learned.
Both beginners and experts found the tutorial easy to use and enjoyable.
Beginners needed the instructions repeated more often than experts.
Experts felt more confident moving around in the virtual world.
Beginners found the tutorial more mentally challenging, but also more fun (probably because it was new to them).
Most people could do the tasks they learned pretty easily after the tutorial.
Very few people felt sick or uncomfortable from using VR.
⁠
Assessing User Experience in Virtual Reality – A Comparison of Different Measurements
⁠
⁠
Explore how different ways of measuring user experience in virtual reality (VR) relate to each other.

The researchers wanted to see if traditional methods for evaluating 2D interfaces could be combined with VR-specific measures to create a better way to evaluate large-scale, multi-user VR experiences.
72 participants tried out a multi-user VR adventure in groups of 3
Participants filled out questionnaires before, during, and after the VR experience
The questionnaires measured things like presence (feeling of being there), social presence (feeling of being with others), user experience, and simulator sickness
The researchers looked at how the different measurements related to each other using statistical analysis
The VR experience was generally rated positively by participants
Feelings of presence and social presence were strongly related to most aspects of user experience
Health-related measures like simulator sickness didn't show strong relationships with user experience
Some measurements taken during the experience didn't match up well with those taken after
There are no rows in this table
⁠

⁠

google-forms

Pre-Test Questionnaire

The pre-survey will include background information about the participant's experience with VR technology, comfort levels associated with headsets, interests of the participant, and their current understanding of the water treatment process.


Pre-Survey Questions
Section
Question
Type
Options
Demographics
Age, location, gender
​
VR Experience and Comfort
On a scale of 1-5, how comfortable are you with using VR headsets?
1-I’ve never used it before 2- I’ve used it once 3- I use it less than once a month 4- I use it a few times per month 5- I use VR weekly
​
If you’ve experienced VR before, what type of VR experiences do you typically enjoy?
Ex: Games, Educational, Entertainment etc.
​
Would you be interested in using VR for educational purposes?
1- Not at all interested 2 -Sllightly interested 3- Neutral, neither interested nor uninterested 4- Moderately Interested 5- Extremely Interested
Water Treatment Knowledge
You are probably familiar with recycling when it comes to paper, aluminum cans, and cardboard, etc. Did you know that water can be recycled too?
No, That sounds familiar but I don’t know much about it Yes
​
What is your current level of understanding of the water treatment process?
I know nothing I have a basic understanding, I know a lot about it I have an advanced understanding on it
​
How comfortable are you with the idea of drinking recycled water?
1-Very uncomfortable 2- Somewhat uncomfortable 3 - Neutral 4- Somewhat comfortable 5-Comfortable
​
What is your definition of low quality water?
​
Expectations and Challenges
What do you want to see in a VR experience? Please rank your choices by order of most important to least important.
Immersion (defined as a feeling of being surrounded by and deeply engaged in an experience) Realistic (like things would appear in real life) Animated, Educational, Artistic, Easy to Use, Interactive (defined as a two-way exchange where the user's actions influence or change the outcome) Other: Short Answer]
There are no rows in this table
⁠

⁠

google-forms

Post-Test Questionnaire

The post-survey will measure the effectiveness in VR through means of immersion, understandability, and comfort.
Post-Test Questions
Section
Question
Type
Options
Immersion and Comfort
How immersive did you find the VR experience? Note: Immersion in VR is defined as a sense that you are surrounded by or fully involved in a virtual space.
1-Not Immersive at all 2-Slightly Immersive 3- Moderately Immersive 4- Very Immersive 5- Completely Immersive
​
Did you experience any discomfort during the experience?
Yes No Explain
​
How would you rate the overall quality of this experience compared to other headset experiences you’ve seen?
1 very low quality 2-Low quality 3-average quality 4- high quality 5- very high quality
Learning and Understanding
Has your understanding of the water treatment process changed after the VR experience?
Yes/No a. If yes, How? b. If no, Why?
​
Now that you’ve experienced an educational VR, do you find it helpful as a tool for education?
1- Not at all interested 2 -Sllightly interested 3- Neutral, neither interested nor uninterested 4- Moderately Interested 5- Extremely Interested
​
How would you rate the effectiveness of this VR in teaching you the Advanced Water Purification process?
Not Effective, Somewhat Effective, Moderately Effective, Very Effective
​
Has your comfort level with drinking recycled water changed?
Yes/No a. If yes, How?
​
How easy or difficult was it to get used to this new form of technology?
1- Very easy 2- Easy 3- Neutral 4- Difficult 5- Very Difficult
Engagement
Can you describe any memorable moments from this experience?
​
​
What aspects of the VR experience did you find most engaging or informative?
​
Usability and Navigation
Were there any parts of the experience that felt confusing or difficult to navigate?
​
​
How easy was it to follow the instructions or prompts within the VR experience? If yes, why?
1- Very easy 2- Easy 3- Neutral 4- Difficult 5- Very Difficult
​
How intuitive did you find the controls?
Not intuitive, Somewhat intuitive, Neutral, Intuitive, Very intuitive
Expectations and Improvements
Did anything during the VR experience turn out differently than you expected?
​
​
Was there anything in the experience you would have changed or improved?
​
There are no rows in this table
⁠

⁠

collaboration
Debrief Session Scripts: Discuss verbally with the participants about the test session. This includes overall feelings, likes, dislikes, pain points and the VR game specific questions.

Script A: Holistic insights

How did you feel during the VR experience? Follow up
Were there any moments when you felt particularly immersed or engaged, as if you were physically present in the experience?
Did you experience motion sickness, dry eye, head ache or dizziness?
What did you think about the 3D art?
can you explain that more?
Can you share your favorite things about the experience?
Can you share the things you didn’t like about the experience?
Follow up: Was there anything you felt was distracting in the experience?
Where did the experience feel difficult to navigate or understand?
Follow up:
Can you tell me more about what made it difficult? (try to get people to elaborate on the more difficult navigation or interaction moments)
Did you have ever find yourself disinterested in the story, or did you experience difficulty relating to the characters or events? (pause for response)
Follow up:
If so, at what point?
Do you think the decisions you made within the experience (game play, exploring the space, touching buttons) had an effect on what came next?
Did you feel like the characters acknowledged you and responded to you in the VR?
Follow up:
Was there a particular character that you really liked? Why did you like them?
What new information did you learn about the water treatment process?
Follow up
How do you think this knowledge will impact your views on recycled water?
Optional: What’s another thing you learned about water treatment? (this question is here if you need more information from the user, skip it if already answered by participant)
How do you feel about drinking recycled water?
Did the VR experience change your perspective on drinking recycled water?
Follow up
If yes, in what ways?
Is there anything about the game that would make you recommend it to others?
Did you feel that the VR element added significantly to the gameplay?

⁠

Script B: game interaction focused

How did you feel during the VR experience? Follow up
Were there any moments when you felt particularly immersed or engaged, as if you were physically present in the experience?
Did you experience motion sickness, dry eye, head ache or dizziness?
Did the 3D art feel realistic?
What were some of your favorite things about the experience?
What were some things you didn’t like about the experience?
Think back to the net gun, contaminant capture game.
Was that game fun and interesting to you?
Could you easily understand how it worked, or did it take you a while to figure out what you were supposed to do?
How do you think we could have made that game more fun or easier to understand?
What else would you like to share about your experience with this game?
Think back to the reverse osmosis, water pressure game.
Was that game fun and interesting to you?
Could you easily understand how it worked, or did it take you a while to figure out what you were supposed to do?
How do you think we could have made that game more fun or easier to understand?
What else would you like to share about your experience with this game?
Think back to the UV Light Saber Filtration Game.
Was that game fun and interesting to you?
Could you easily understand how it worked, or did it take you a while to figure out what you were supposed to do?
How do you think we could have made that game more fun or easier to understand?
What else would you like to share about your experience with this game?
Did you like the way the VR Experience ended?
What new information did you learn about the water treatment process?
Follow up
How do you think this knowledge will impact your views on recycled water?
How do you feel about drinking recycled water?
Did the VR experience change your perspective on drinking recycled water?
Follow up
If yes, in what ways?
11. Were there moments where you felt you could predict what was coming next, or did events catch you off guard?
12. Did any tasks require more concentration than others? If so, can you tell us why?
13. Did interacting with virtual characters feel natural?
Follow Up
If so, why?
If no, why?
Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.