Examination of users' interactions with Conversational Artificial Intelligence
Goal
While considering users' autonomy and technology acceptance, we wanted to understand how their interactions with artificial intelligence agents are affected when they collaborate during problem-solving tasks.
Approach
We measured the impact of factors such as autonomy and technology acceptance by constructing a gamified UX experiment environment that positioned AI as a vibrant “teammate” while working in timed tasks.
What did we do?
Process
Process
An environment was created for users who were asked to solve problem-solving tasks in a bomb disposal game, requiring them to interact with a specially developed GPT-4 assistant and collaborate and solve tasks together. In order to reveal how user autonomy shapes human-artificial intelligence cooperation, basic psychological factors were evaluated through validated scales (ACS-30 and UTAUT2) and the written interactions of users with the artificial intelligence agent were examined.
Results
Results
Users with high autonomy achieved faster and more successful task completion, even when AI made errors. Participants perceived AI as an active second player and teammate rather than a passive tool. Users with low autonomy showed greater reliance on AI guidance and expressed a clear need for support. Although technology acceptance was generally high, it was not a significant factor in determining task performance. It was observed that AI systems, in their current state, have a high likelihood of making errors.
Design Recommendations
Design Recommendations
Recommendations regarding psychological design principles for AI interfaces were proposed. It was suggested that AI systems should dynamically adjust their level of support based on users' confidence and expertise in using technology; specifically, by offering direct, step-by-step guidance to users with low autonomy while enabling those with high autonomy to take independent initiative, and by supporting these interactions with clear visual cues that inform users about potential AI biases and errors.
Measurable Outcomes
Higher user autonomy directly increased task speed and success, even during AI errors, with users perceiving AI as a teammate rather than a tool. Findings established that AI interfaces must provide adaptive support while transparently surfacing system limitations.
Looking to scale your product conversions?
We can increase your conversion rates with psychology-based design methods and optimize your user experience using scientific methodologies.