Conversational AI-Driven Coach: A Personalized Digital Coach for Enhancing Student Performance and Goal Achievement
Piloted a GPT-based Socratic tutor in a University of Bern statistics course, comparing tutoring modes and deriving design guidelines.
Duration: January 2025 – December 2025
Status: Completed
Educational Level: Tertiary Level
Topic: Artificial Intelligence AI, Digital Tools
Keywords: genAI, Coaching, Socratic, AI, Tutoring
Initial Situation
Students in introductory university statistics courses vary strongly in prior knowledge and study habits. Traditional support (exercises, office hours, forums) does not scale to the volume and timing of questions, and many learners need help that targets reasoning steps rather than final answers. Large language models can offer personalised tutoring at scale, but uncontrolled generation conflicts with course-specific terminology, notation and didactic sequencing. A deployable solution therefore needs both a pedagogically grounded tutoring strategy (Socratic questioning) and technical guardrails that keep responses aligned with the course materials.
Objectives
- Build a course-specific conversational tutor for a University of Bern statistics course using GPT-4-class models (later GPT-5).
- Implement and compare two tutoring strategies: Socratic tutoring and standard instructional explanations.
- Engineer a controllable architecture that respects theexact wording and notation of the course while remaining robust to unexpected learner paths.
- Evaluate feasibility and preliminary learning outcomes in a real deployment and derive transferable design recommendations.
Method
We combined
- (a) literature review on tutoring and behavioural change techniques with
- (b) iterative prototyping of multiple system architectures.
The final system used a checkpoint-based workflow mixing hard-coded constraints (terminology, references, forbidden paraphrases) with generative components. We deployed the tutor in a Uni Bern statistics course in a counterbalanced two-week field experiment: students were randomly assigned to receive either the Socratic or instructional mode first and then switched. We analysed chat logs and pre/post quizzes.
Results
We delivered a functioning tutor and deployed it successfully in a live university course. The main technical finding was that real deployments require strong controllability: course stakeholders expected strict adherence to the official wording and notation, while students expected the robustness of consumer AI platforms. A checkpoint-based hybrid architecture proved effective for enforcing terminology while keeping the dialogue adaptive. In the field evaluation, overall pre-to post changes were small; differences between tutoring modes were not conclusive in week 1, while week 2 showed a small trend favoring the Socratic mode (higher mean improvement than the instructional mode). The study also highlighted high missingness and strong dependence on students’ starting level, motivating larger-sample follow-up evaluations.
Implemented Translation
The tutor was implemented and piloted as part of a University of Bern statistics course. Students could use it for self-study and problem solving, and the course team provided concrete constraints on terminology, notation and allowed solution steps. Based on observed failure modes and stakeholder feedback, we iteratively refined guardrails (e.g., discouraging synonyms, enforcing references to the course material, and handling unexpected solution paths). The deployment served as a proof of concept for integrating an LLM tutor into authentic teaching settings and established a workflow that can be reused in future course iterations.
