Comparing Tutor vs. Socratic LLM-driven dialogue strategies to quantify engagement, goal attainment, and long-term learning in diverse cohorts.
Students in specialized study programs often possess diverse academic backgrounds, leading to varying prior knowledge and preparedness. This variation poses challenges to delivering personalized instruction and support. While personalized approaches are known to improve learning outcomes, educators often struggle to provide individualized support due to time limitations and cognitive overload. Although goal-setting is an effective strategy for increasing focus, motivation, and learning efficiency, implementing it at scale remains difficult under these constraints.
Leveraging Large Language Models (LLMs) for digital coaching presents a scalable way to deliver personalized, motivational guidance tailored to diverse learner needs. However, key gaps remain. First, it is unclear how different dialogue strategies – such as Tutor-based content delivery versus Socratic-based questioning – impact student engagement, goal attainment, and retention. Second, the long-term effects of these strategies on behavioral change and learning are poorly understood. This research addresses these gaps by comparing these strategies within an LLM-driven digital coach through a controlled laboratory evaluation and a longitudinal field study.
The expected outcomes include empirical insights into the effectiveness of each dialogue strategy, contributing to a theoretical framework for LLM-based digital coaching. This work will advance understanding of how personalized, LLM-driven coaching can improve student learning, engagement, and long-term success. By integrating coaching theories, such as the GROW model and behavioral change techniques, with advanced AI techniques, this project aims to develop scalable digital coaching solutions that significantly benefit educators and students.
The research collaboration between the Bern University of Applied Sciences, the University of St. Gallen, and the University of Pennsylvania has already resulted in publications in leading academic outlets, demonstrating their expertise in chatbot development, AI-driven feedback systems, and coaching theory. This study is focused on advancing student learning in higher education in Switzerland.
Starts January 2025
Usage Analytics and Engagement Metrics in Real Courses:
By integrating the digital coach into existing learning platforms (e.g., Moodle) and tracking key engagement indicators -such as the frequency of tool use, number of goal-setting sessions initiated, time spent interacting with the coach, and completion rates of guided learning tasks- you get a direct, quantifiable measure of adoption and sustained use.
Instructor and Administrator Feedback via Surveys and Interviews:
Post-implementation interviews and surveys with educators and program heads offer qualitative insights into ease of integration, perceived value, and alignment with teaching goals. This feedback helps confirm whether the tool is not only used but also meeting practitioners’ needs and encouraging them to continue or expand its use.
Comparative Pre-Post Student Performance and Retention Data:
Analyzing changes in student outcomes before and after deploying the digital coach—looking at goal attainment, improvements in coursework, and retention rates—provides a direct link between the research-based intervention and actual educational improvements. This evidence demonstrates whether the innovation delivers meaningful, lasting benefits in real classroom conditions.
Website:
https://romanrietsche.github.io/
Andreas Göldi
Prof. Dr. Lyle Ungar
