Sketch to Pipes: Modeling, Understanding, and Supporting Mental Rotation Skills in AR
Mental rotation abilities are crucial in the VET context. AR can support students in improving these skills while learning technical drawing.
Duration: September 2025 – August 2026
Status: Ongoing
Educational Level: Upper Secondary Level – Vocational Education
Topic: Artificial Intelligence AI, Digital Tools
Keywords: Augmented Reality, Technical Drawing, Mental Rotation Abilities, Machine Learning
Initial Situation
Spatial reasoning is an important skill in many professions (e.g., carpenters, plumbers, electricians). Architectural draftsmen for example need to model buildings in 3D environments. Electricians need to install systems fitting within complex structures. Heating and sanitary installers need to build 3D systems from 2D drawings. These tasks require professionals to mentally rotate objects and anticipate how they will appear from different viewpoints, a skill called mental rotation (MR). The development of mental rotation abilities is therefore central to support learning in vocational education and training (VET). Unfortunately, teaching mental rotation skills is difficult as there are no standardized strategies that can be verbalized or imitated. Mental rotation skills are therefore often implicitly learned through practicing on real-world tasks. In a VET teachers often lack the possibility to closely monitor all apprentices in their practice and provide individual feedback.
Objectives
Our project has two main objectives: First, understanding AR’s affordances for improving mental rotation skills. We’ll collect empirical data across Swiss language regions, analyze learning processes using multi-modal data (eye-tracking, hand-movement, task behavior) with data mining techniques, and link process measures to learning outcomes. Second, enabling classroom adoption by developing a teacher’s desktop cockpit for exercise management and localizing the app in three national languages. Teacher interviews will assess perceptions and classroom needs.
Method
We will conduct a longitudinal experimental study with 80-100 VET apprentices (heating/sanitary installers) from German- and Italian-speaking Switzerland. The design includes:
- baseline assessment (T0) with MR and technical drawing tests;
- four weekly AR training sessions (T1-T4) using AR (Sketch-to-Pipes app) on HoloLens 2, with experimental and control groups accessing AR (with vs without real-time 2D projection updates);
- post-test (T5) repeating initial assessments plus a transfer task.
We’ll collect multi-modal data (eye-tracking, hand-movement, task behavior), cognitive load and motivation self-reports. Machine learning techniques will analyze behavioral patterns and predict learning outcomes. Teacher interviews will assess pedagogical value and classroom integration potential.
Planned Translation
SPAI Locarno, gibb Bern and BB Zürich have expressed strong interest in using Sketch-to-Pipes due to its demonstrated effectiveness. Indeed, a previous project tested the prototype in two phases: a one-month pilot and a two-month longitudinal study, which showed significant improvements in technical drawing and mental rotation skills of apprentices with large effect sizes. Building on these findings, this project will employ a longitudinal study to understand how different AR affordances impact learning outcomes and investigate underlying learning mechanisms. We will extend to different language contexts and refine the application based on teachers’ needs. The study will run directly in schools in collaboration with teachers, ensuring translation into practice from the beginning.
