AI-mate: Evaluating AI chatbot
Evaluating an AI-enabled real-time intervention to improve student engagement in group discussions
Overview
-
The pandemic led to a significant shift towards online education and increased barriers to social interactions of students.
-
According to a survey conducted by Inside Higher Ed in 2020 with 97 university presidents in the U.S., 81% found it challenging to maintain student engagement during online classes.
-
In this study, we aim to design a chatbot-based solution and investigate if it can serve to increase engagement during online group discussions.
My Role
Duration
-
I designed and implemented a research method to explore people's interactions with the Wizard of Oz prototype of the AI chatbot.
-
I mapped out research insights to design appropriate chatbot interventions.
12 weeks (Feb - Apr 2022)
Tools
Figma, Zoom, Miro
Team
3 researchers and designers
Process
Research
Goal
With an initial goal to evaluate AI-based solutions for addressing social challenges in online education, the project began by scoping the research goal within the domain of AI for social concerns in online education.
Consequently, we chose to focus on online group discussions in classrooms and collected relevant user needs. In research phase II, we evaluated the impact of an AI chatbot on student engagement in group discussions.
Scoping Research
How do current AI-enhanced online education tools aim to address social challenges of students?
Gathering Needs
What are the pain points of middle school students in online group discussions?
Evaluating Concept
How can we evaluate the impact of real-time AI interventions on student engagement in online group discussions?
Phase I - Discovery
Phase II - Evaluation
Phase I - Discovery
How do current AI-enhanced online education tools aim to address social challenges of students?
Market Analysis
What are the offerings of existing AI-driven products for enhancing social skills and engagement?
From our market analysis, we discovered a broad categories of AI-enhanced social tools for children, focusing on mental health, social skills, and communication skills.
Literature Review
What are the current solutions, challenges, and potential applications of AI in enhancing online education, facilitating social interactions, and utilizing chatbots for group discussions?
Through literature review we gained an in-depth understanding of AI-enhanced solutions, such as chatbots for group discussions.
What are the pain points of middle school students in online group discussions?
After conducting secondary research, we narrowed our focus to the challenge of group discussion experience in online education.
Interviews
We investigated pain points of middle school students navigating online education during the pandemic?
We interviewed five children alongside their parents to better understand the user's needs. The interview questions were tailored to comprehend students' experiences with group discussions, and parents attitudes towards child's social interactions.
Key Findings
Chatbots promote group discussions, by taking roles such as group members, moderators.
Inadequate knowledge, motivation, confidence, nervousness, and a lack of self-reflection lead to reduced student engagement in group discussions.
Lack of interactive and individualized feedback, due to lack of connection between instructor and students, is precieved as major barrier in online learning experience.
Design implications
A personalized AI-chatbot enabling engagement through relevant interventions
We designed AI-mate, a personalized chatbot designed to engage with participants individually during group discussions, through tailored interventions inspired by research.
Validate participation
Validate participation by encouraging contributing participants.
Feedback on participation
Feedback on participation by sharing relative engagement based on time spoken and turns taken, to drive self-reflection.
Suggestion Prompts
Suggestion prompts to help initiate conversation, taking the role of an instructor.
Scenarios and Interventions
.png)
Phase II - Evaluation
Investigating the impact of AI-enabled real-time intervention and feedback on student engagement in group discussions
Wizard of Oz Study Design
-
At the beginning of the study, we asked participants to complete a pre-session survey to better understand their personalities in group discussions.
-
We then simulated group discussion sessions with and without the AI mate and observed the chatbot interactions and participant engagement.
-
After each discussion, we conducted a post-discussion survey to measure the participants' attitudes toward the discussion.
-
Finally, we administered a post-study survey to gauge the participants' perception of the chatbot.
-
After the survey, the three researchers individually spoke with each participant in breakout rooms to gather qualitative feedback and ask follow-up questions.
I simulated the AI mate experience by communicating with the participants from behind the screen based on defined interventions.
Me
Findings
AI intervention through real-time feedback on the performance improved motivation and overall satisfaction.
Participants who consider themselves active contributors in group discussions showed a similar level of engagement in both discussions(with and without AI). However, they showed higher satisfaction ratings in the discussion where AI Mate provided performance validation.
Timely and relevant suggestion prompts by the AI Mate were most valuable to participants.
Those who identified themselves as passive participants in group discussions, emphasized that such prompts would facilitate their active participation by reducing barriers and making involvement more accessible.
Friction due to curiosity on how the AI mate is interacting with others
Participants brought up AI-mate at times during the discussion mentioning its reply or seeking other's experiences. The curiosity of individuals to know how the system is interacting with others in the group might distract the conversation.
Reflections
Getting innovative with research methods
This project helped me improve crafting novel research methods considering time and study constraints, such as utilizing wizard of oz alongside experiment design and leveraging qualitative survey questionnaires to gather feedback true to the real usage environment.
Multi-participant moderation
I honed my skills in recruiting, managing, and moderating studies with multiple participants for the group discussion, similar to the focus group studies.
Transparency on study methodology
It is crucial to make sure that the methodology of the study is transparent and accepted by the participants while maintaining the designed environment. We followed a two-phase consent process. The first was at the start of the study, where participants gave their consent to participate. The second was at the end of the study, where we informed them about the use of a human simulation for AI chatbot and sought their continued consent to use their data for research.