Explore

Comparing AI-generated and human-written setup & troubleshooting instructions in a VR experience

Goal: To compare the individual strengths of AI-generated and human-written technical documentation specific to instructions for setup and troubleshooting for an existing VR experience. The aim is to specifically focus on how these instructions impact the user experience in a semi-moderated test environment by analyzing factors like time taken, error rate, success rate, number of interventions, etc.

Procedure & Setup:

Participant Recruitment and Consent: Recruit participants and get consent for recording and participation. Explain the semi-moderated structure of the VR test. Randomly assign participants to either the AI-generated audio instructions group or the human-read instructions group, considering they are split equally between the two.

Instructions:

AI-generated Group: Participants receive AI-generated audio instructions for setting up the VR experience. The moderator is present during the test but does not provide additional instructions or assistance unless the participant explicitly requests help or faces a clear technical issue. The moderator can rewind or replay the audio instructions as needed without any verbal assistance.

Human-written Group: The moderator reads the human-written instructions aloud to the participant/verbally narrates the instructions (worried that this is leaning toward the moderated side). The moderator can re-read sections as needed based on questions that the participant has.

VR Setup: Participants attempt to set up the VR experience following the instructions they receive.

Moderator Intervention (if needed): The moderator observes the participant's progress. If the participant struggles or asks for help, the moderator can give clarification, replay sections of the audio (AI group), reread sections (human group) or offer assistance. We will keep in mind to record when and why they intervened.

Methods:

Think-Aloud (Recorded): Ask participants to think aloud while they are setting up the VR and record the session.

Debriefing: After the test, have a way for participants to ask questions or give additional feedback.

Parameters to compare:

Time taken for each task

Error rates and success rates

Number of interventions required

User satisfaction scores

Qualitative feedback from participants about the process, limitations, and issues faced

Literature Review

Literature Review

Paper 1

The History of Technical Communication and the Future of Generative AI

Paper 2

On the Current Moment in AI: Introduction to Special Issue on Effects of Artificial Intelligence Tools in Technical Communication Pedagogy, Practice, and Research, Part 1

Paper 3

Surveillance Work in (and) Teaching Technical Writing with AI

Paper 4

Beyond Academic Integrity: Navigating Institutional and Disciplinary Anxieties About AI-Assisted Authorship in Technical and Professional Communication

Paper 1

Paper Title

The History of Technical Communication and the Future of Generative AI

Details of Publication

Proceedings of the 42nd ACM International Conference on Design of Communication, 2024

Description

Download PDF

Paper 1- The History of Technical Communication and the Future of AI.pdf

Overview

Understand intersection of generative AI and Technical and Professional Communication (TPC).

Focuses on four key areas: How writing processes work, theories about power and control in communication, the roles of humans and technology in communication, and issues of fairness and equity.

Encourages Tech Comm. professionals to ensure ethical and inclusive use of AI technologies.

Key Theories and Research

Iteration and Process: The paper discusses how generative AI might change traditional writing processes by reducing iterations, which might affect learning outcomes. It examines frameworks like Agile Software Development and design thinking as models for integrating AI into TPC.

Theory, Agency, Power: How AI affects who has control and influence in communication. It looks at theories to understand how texts created by AI might change the role and power of humans in creating and interpreting written content. This involves questioning whether AI-generated texts are merely copies or if they have their own unique qualities that could alter human work in communication.

Actors and Activity: Using Actor-Network Theory (ANT), the paper talks about the roles of human and non-human actors in technical communication, highlighting the importance of maintaining human role in AI-driven processes.

Social Justice: The paper addresses the potential for AI to replicate biases present in training data, leading to outcomes that might be discriminating. It says that it is important to have careful observation and justice-oriented approaches.

Areas of Controversy and Claims

Impact on Writing Processes: There is debate over whether generative AI's ability to reduce iterations is beneficial or harmful to learning and creativity.

Agency and Power Dynamics: The extent to which AI affects human control is talked about, with concerns about AI reducing human roles in decision-making.

Bias and Discrimination: The potential for AI to build on existing biases is an area of concern. There are different views on how these issues can be managed.

Gaps in Research

Long-term Effects on Learning: More research is needed on the long-term impacts of reduced iterations in writing processes due to AI.

Ethical Frameworks for AI Integration: There is a need for complete ethical frameworks for the integration of AI into TPC.

Case Studies on Social Justice Impacts: Additional case studies are required to understand how generative AI affects Underrepresented populations and what measures can be taken to ensure equal outcomes.

⁠

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.