top of page

AI-mate: Evaluating AI chatbot

Evaluating an AI-enabled real-time intervention to improve student engagement in group discussions

Group 8 (2).png

Overview

Overview
  • The pandemic led to a significant shift towards online education and increased barriers to social interactions of students.

  • According to a survey conducted by Inside Higher Ed in 2020 with 97 university presidents in the U.S., 81% found it challenging to maintain student engagement during online classes.

  • In this study, we aim to design a chatbot-based solution and investigate if it can serve to increase engagement during online group discussions. 

My Role 

Duration

  • I designed and implemented a research method to explore people's interactions with the Wizard of Oz prototype of the AI chatbot.

  • I mapped out research insights to design appropriate chatbot interventions.

12 weeks (Feb - Apr 2022)

Tools

Figma, Zoom, Miro

Team

3 researchers and designers

Process
Research

Research

Goal

With an initial goal to evaluate AI-based solutions for addressing social challenges in online education, the project began by scoping the research goal within the domain of AI for social concerns in online education. 

 

Consequently, we chose to focus on online group discussions in classrooms and collected relevant user needs. In research phase II, we evaluated the impact of an AI chatbot on student engagement in group discussions.

Scoping Research

How do current AI-enhanced online education tools aim to address social challenges of students?

Gathering Needs

What are the pain points of middle school students in online group discussions?

Evaluating Concept

How can we evaluate the impact of real-time AI interventions on student engagement in online group discussions?

Phase I - Discovery

Phase II - Evaluation

Phase I - Discovery

How do current AI-enhanced online education tools aim to address social challenges of students?

Market Analysis

What are the offerings of existing AI-driven products for enhancing social skills and engagement?

From our market analysis, we discovered a broad categories of AI-enhanced social tools for children, focusing on mental health, social skills, and communication skills.

Screenshot 2024-03-03 at 4.44.43 PM.png

Literature Review

What are the current solutions, challenges, and potential applications of AI in enhancing online education, facilitating social interactions, and utilizing chatbots for group discussions?

Through literature review we gained an in-depth understanding of AI-enhanced solutions, such as chatbots for group discussions.

What are the pain points of middle school students in online group discussions?

After conducting secondary research, we narrowed our focus to the challenge of group discussion experience in online education.

Interviews

We investigated pain points of middle school students navigating online education during the pandemic?

 

We interviewed five children alongside their parents to better understand the user's needs. The interview questions were tailored to comprehend students' experiences with group discussions, and parents attitudes towards child's social interactions.

Screenshot 2024-03-03 at 4.46.59 PM.png
Key Findings

Chatbots promote group discussions, by taking roles such as group members, moderators.

Inadequate knowledge, motivation, confidence, nervousness, and a lack of self-reflection lead to reduced student engagement in group discussions.

Lack of interactive and individualized feedback, due to lack of connection between instructor and students, is precieved as major barrier in online learning experience.

Design implications

A personalized AI-chatbot enabling engagement through relevant interventions

We designed AI-mate, a personalized chatbot designed to engage with participants individually during group discussions, through tailored interventions inspired by research.

Validate participation

Validate participation by encouraging contributing participants.

Feedback on participation

Feedback on participation by sharing relative engagement based on time spoken and turns taken, to drive self-reflection.

Suggestion Prompts

Suggestion prompts to help initiate conversation, taking the role of an instructor.

Scenarios and Interventions

Flow (1).png
Phase II - Evaluation

Investigating the impact of AI-enabled real-time intervention and feedback on student engagement in group discussions

Wizard of Oz Study Design

  • At the beginning of the study, we asked participants to complete a pre-session survey to better understand their personalities in group discussions.

  • We then simulated group discussion sessions with and without the AI mate and observed the chatbot interactions and participant engagement.

  • After each discussion, we conducted a post-discussion survey to measure the participants' attitudes toward the discussion.

  • Finally, we administered a post-study survey to gauge the participants' perception of the chatbot.

  • After the survey, the three researchers individually spoke with each participant in breakout rooms to gather qualitative feedback and ask follow-up questions. 

Untitled.jpg
Screenshot 2024-03-03 at 5.33.43 PM.png

I simulated the AI mate experience by communicating with the participants from behind the screen based on defined interventions.

Group 1000000879 (1).png

Me

Findings

AI intervention through real-time feedback on the performance improved motivation and overall satisfaction.

Participants who consider themselves active contributors in group discussions showed a similar level of engagement in both discussions(with and without AI). However, they showed higher satisfaction ratings in the discussion where AI Mate provided performance validation.

Timely and relevant suggestion prompts by the AI Mate were most valuable to participants.

Those who identified themselves as passive participants in group discussions, emphasized that such prompts would facilitate their active participation by reducing barriers and making involvement more accessible.

Friction due to curiosity on how the AI mate is interacting with others

Participants brought up AI-mate at times during the discussion mentioning its reply or seeking other's experiences. The curiosity of individuals to know how the system is interacting with others in the group might distract the conversation.​

Reflections
Reflections

Getting innovative with research methods

This project helped me improve crafting novel research methods considering time and study constraints, such as utilizing wizard of oz alongside experiment design and leveraging qualitative survey questionnaires to gather feedback true to the real usage environment.

Multi-participant moderation

I honed my skills in recruiting, managing, and moderating studies with multiple participants for the group discussion, similar to the focus group studies.

Transparency on study methodology

It is crucial to make sure that the methodology of the study is transparent and accepted by the participants while maintaining the designed environment. We followed a two-phase consent process. The first was at the start of the study, where participants gave their consent to participate. The second was at the end of the study, where we informed them about the use of a human simulation for AI chatbot and sought their continued consent to use their data for research.

bottom of page