Measuring Authenticity in Assessment

We propose a method for defining and measuring the authenticity of assessment by evaluating the online replication of paper exams. We find that by using advanced question types, 66% to 94% of the original exam can be authentically replicated, whereas less than 26% can be replicated using strictly multiple-choice questions.

While educators and researchers agree that the best classroom assessments are authentic, little consensus remains as to what defines ‘authentic assessment’.¹ If ‘authentic’ generally means to be ‘real’ or ‘true,’ what distinguishes an assessment as such? How can authenticity be measured? And why are authentic assessments considered pedagogically the best?

Defining Authenticity

The distinction of an assessment as authentic is not only in how it presents questions but also in how it enables questions to be answered. Compelling students to engage with concepts and solve problems as an academic would, authentic assessments enhance the learning process for students to develop their skills in a realistic manner by working through each applied steps to supply a precise answer in correct notation.

Assessments are considered authentic if they can replicate or simulate the diverse contexts of an academic field.² In the quantitative subjects, these assessments require students to find patterns, check generalizations, make models, and extend processes to resemble the actions of an academic applying concepts to everyday life.³

Measuring Authenticity: Methodology

Our team undertook a research initiative evaluating a select group of quantitative papers to discern what percentage of mark-weighted written exam questions could authentically be replicated using either only multiple-choice questions (MCQs) or using our platform's advanced question types. These exams include an A-Level Maths Edexcel paper, a GCSE HT Maths Edexcel paper, and an A-Level Physics Edexcel paper.

Questions were evaluated on a scale of authenticity, with below 50% rated as poor, between 50% and 75% as okay, and above 75% as authentic. Accumulative percentages reflect how much of the original, paper exam's authenticity could be preserved using each format in the transition to online testing.

Our Findings

Our findings are demonstrated in the table below. Listed per paper is the number of marks, the percentage of questions replicable using MCQ only, and the percentage of questions replicable using MCQ as well as using question types offered by our platform, i.e., numerical and quantitative inputs such as graphs, tables, formulas, equations, intervals, inequalities, and more:

Measurements in the first column show that relying exclusively on multiple-choice can only authentically replicate a limited portion of the original paper. By contrast, the second column demonstrates that by expanding the capabilities of assessments to include a wide range of numerical and quantitative types and answer inputs, our platform's assessment capabilities can authentically preserve most of the original paper experience.

Why Authenticity is Our Goal

At Summatic, our aim is to continue incorporating a wider range of advanced question types to allow students to engage with an increasingly authentic assessment experience, while also facilitating a smoother and more authentic marking process for educators to see results and evaluate student performance. We uphold that the best of online assessment is always striving to be more authentic, and we believe this is reflected in all that we do and will continue to do for those we support.