Dr. Jessica Herzing
- Interfaculty Centre for Educational Research (ICER)
Menu
PANDA analyzed process data to advance computer-based assessments in multilingual contexts.
PANDA scrutinized students› usage patterns in computer-based assessments using process data, aiming to improve future assessments in multilingual contexts. Drawing on data from a Swiss assessment involving 4’500 students, PANDA analyzed students› usage patterns while solving the test items.
The findings highlighted that later test items are associated with reduced test performance in specific domains, especially in certain language regions. Additionally, the study identified suspicious test items, shedding light on disparities in audio quality and translation across assessment languages.
Therefore, PANDA showcased how process data within computer-based assessments can bolster quality assurance.
Project description:
Unlike traditional assessments, computer-based assessments gather process data (also named paradata), encompassing students› actions during the assessment and their interactions with the test platform. PANDA analyzed these actions — comprising response times, frequency of answer changes during test-solving, patterns of skipping questions throughout the assessment, engagement with audio players, and the completion status of all test items—providing valuable insights into students› approaches to tackling test items. Through the analysis of these usage patterns, valuable insights were acquired regarding the methodologies students employed during assessments but also regarding the test setting (e.g., test rotations, breaks, test language etc.) and test items itself to inform future assessments in multilingual contexts, as well as, to show how process data can inform quality assurance in computer-based assessments.
Method:
PANDA utilized process data from the assessment of national educational standards (known as Überprüfung der Grundkompetenzen) conducted in Swiss compulsory schools on a large scale. The analysis specifically centered on the field trial data collected in 2022, involving approximately 4’500 second-grade students who participated in an assessment covering all language regions of Switzerland. PANDA’s primary objective was to reveal usage patterns across diverse school languages using learning analytics.
Results:
This study focused on investigating usage patterns among students in computer-based assessments and their implications for improving future assessments. It aimed to comprehend how students engage with assessment tasks and how various factors influence their behavior during testing.
The analysis revealed that later test items generally led to lower student performance in certain domains. Additionally, booklet position significantly impacted student performance, especially in some language regions of Switzerland. The study identified various suspicious test items, highlighting issues like audio quality and translation disparities across assessment languages.
Publications:
Hlosta, M., Herzing, J. M. E., Nath, S., Seiler, S., Keller Zai, F., Bergamin, P., Erzinger, A. B. (forthcoming). Analysis of process data to advance computer-based assessments in multilingual contexts. In Ifenthaler, D. & Sahin, M. “Assessment Analytics in Education – Designs, Methods and Solutions”. Springer.
PANDA’s key outcome was understanding the log files and extracting meaningful insights. This understanding highlighted prerequisites for effectively utilizing process data, suggesting improvements for future studies related to test settings (e.g., test rotations), specific test items (e.g., translation deviations, quality of audio help), or test administrator procedures (e.g., duration of breaks).
Dr. Jessica Herzing
Prof. Dr. Per Bergamin
Dr. Martin Hlosta
Dr. Sukanya Nath
