Journal article 271 views 49 downloads
A Bayesian active learning approach to comparative judgement within education assessment
,
Tom Crick ,
Stephen Lindsay
Computers and Education: Artificial Intelligence, Volume: 6, Start page: 100245
Swansea University Authors:
Andrew Gray, Alma Rahat , Tom Crick
-
PDF | Version of Record
This is an open access article under the CC BY 4.0 Creative Commons Attribution license.
Download (1.36MB)
DOI (Published version): 10.1016/j.caeai.2024.100245
Abstract
Assessment is a crucial part of education. Traditional marking is a source of inconsistencies andunconscious bias, placing a high cognitive load on the assessors. One approach to address these issues is comparative judgement (CJ). In CJ, the assessor is presented with a pair of items of work, and as...
| Published in: | Computers and Education: Artificial Intelligence |
|---|---|
| ISSN: | 2666-920X 2666-920X |
| Published: |
Elsevier BV
2024
|
| Online Access: |
Check full text
|
| URI: | https://cronfa.swan.ac.uk/Record/cronfa66575 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract: |
Assessment is a crucial part of education. Traditional marking is a source of inconsistencies andunconscious bias, placing a high cognitive load on the assessors. One approach to address these issues is comparative judgement (CJ). In CJ, the assessor is presented with a pair of items of work, and asked to select the better one. Following a series of comparisons, a rank for any item may be derived using a ranking model, for example, the Bradley-Terry model, based on the pairwise comparisons. While CJ is considered to be a reliable method for conducting marking, there are concerns surrounding its transparency, and the ideal number of pairwise comparisons to generate a reliable estimation of the rank order is not known. Additionally, there have been attempts to generate a method of selecting pairs that should be compared next in an informative manner, but some existing methods are known to have created their own bias within results inflating the reliability metric used within the process.As a consequence, a random selection approach is usually deployed.In this paper, we propose a novel Bayesian approach to CJ (which we call BCJ) for determiningthe ranks of a range of items under scrutiny alongside a new way to select the pairs to present tothe marker(s) using active learning, addressing the key shortcomings of traditional CJ. Furthermore,we demonstrate how the entire approach may provide transparency by providing the user insightsinto how it is making its decisions and, at the same time, being more efficient. Results from oursynthetic experiments confirm that the proposed BCJ combined with entropy-driven active learningpair-selection method is superior (i.e. always equal to or significantly better) than other alternatives,for example, the traditional CJ method with differing selection methods such as uniformly random,or the popular no repeating pairs where pairs are selected in a round-robin fashion. We also find thatthe more comparisons that are conducted, the more accurate BCJ becomes, which solves the issuethe current method has of the model deteriorating if too many comparisons are performed. As ourapproach can generate the complete predicted rank distribution for an item, we also show how thiscan be utilised in probabilistically devising a predicted grade, guided by the choice of the assessor.Finally, we demonstrate our approach on a real dataset on assessing GCSE (UK school-level) essays,highlighting the advantages of BCJ over CJ. |
|---|---|
| Keywords: |
Comparative judgement, bayesian learning, active learning, machine learning, assessment, Bradley-Terry model (BTM) |
| College: |
Faculty of Science and Engineering |
| Funders: |
EP/S021892/1 |
| Start Page: |
100245 |