Comparing examination standards without graded candidate scripts

Comparative judgement methods are commonly used to explore standards in examination papers over time. However, studies are limited by a paucity of graded candidate scripts from previous years, as well as the expense and time required to standardise scripts. We present three studies that attempted, without the use of graded candidate scripts, to replicate and extend previous results about standards in mathematics examination papers. We found that re-typesetting examination papers into a consistent format was necessary, but that comparative judgement of examination papers without an archive of graded candidate scripts offered a reliable and efficient method for revealing relative demand over time. Our approach enables standards comparison where previously this was not possible. We found a reasonable correlation between judgments of actual student scripts and judgments of the items only, meaning that conclusions may be drawn about the demand of examination papers even when graded candidate scripts are not available.