PANDA – Pattern ANalysis of Digital-Based Assessments

Duration: February 2023 – June 2023
Status: Completed
Educational Level: Primary Level
Topic: Data Science for Education
Keywords: Large-Scale Assessments, Learning Analytics, Process Data, Test Development

Intitial Situation

While traditional assessments primarily generate outcome data, computer-based assessments additionally collect process data (paradata). This data includes response times, frequency of answer changes, interactions with audio players, or the skipping of items. In Switzerland’s multilingual education system, the analysis of such process data holds significant potential for gaining insights into students’ test-taking behavior as well as ensuring the quality and fairness of assessment items.

Objectives

The PANDA project aimed to systematically investigate students’ test-taking behaviour in computer-based assessments using process data. The project addressed two key questions:

• How do factors such as test language, item position, or breaks influence students’ performance and engagement during testing?
• How can process data be leveraged to strengthen quality assurance in multilingual assessments?

Method

PANDA analysed process data from a large-scale Swiss educational monitoring. Specifically, field trial data from 2022 was examined, involving approximately 4,500 lower secondary students from all Swiss language regions. The analyses included:

• Response times and frequency of answer changes
• Patterns of skipping questions
• Engagement with audio elements
• Effects of item position and test settings on performance

Learning analytics approaches were applied to reveal usage patterns across languages and contexts.

Results

The study yielded several key findings:

• Item position effects: Later items in the test were associated with lower performance in certain subject domains.
• Regional variation: Position effects were particularly pronounced in some Swiss language regions.
• Quality issues: Suspicious items were detected, linked to translation inconsistencies and variations in audio quality.

Overall, PANDA demonstrated how process data provides valuable insights not only into students’ strategies but also into structural aspects of test design and implementation.

Implemented Translation

The insights from PANDA translated into concrete recommendations for future assessments:

• Test settings: Optimizing item rotations and structuring breaks more effectively
• Test items: Enhancing translation processes and ensuring consistent audio quality
• Test administration: Providing targeted training for test supervisors regarding assessment procedures.

PANDA contributes to the quality assurance of computer-based assessments in multilingual contexts. The systematic use of process data enables:

• Precise identification of problematic test items
• Improved comparability across language regions
• Evidence-based refinement of test design and administration.

Participating Institutions