Convergence of validity for the results of a summative assessment with confidence measurement and traditional assessment

This research examines the use in IT education of an innovative online assessment tool that incorporates confidence measurement. The tool is based on the traditional Multiple Choice Question (MCQ) format with an additional component that permits the user to register their confidence for each answer. The tool is referred to as the Multiple Choice Question with Confidence Measurement (MCQCM). A cohort of 52 Data Communication students utilized the MCQCM as their primary revision tool throughout a semester and then for a test in class. As part of the review they were then asked to give feedback on using the MCQCM as a formal summative assessment tool. The test was graded using the traditional method as well as by calculating a further grade from the student’s registered confidence. The results demonstrated a good correlation and convergence of validity between the dual marks supporting the use of the system as a summative assessment option. It was also observed that the majority of high achievers scored less for the MCQCM grade than for the traditional MCQ. In addition, the students’ perception of the MCQCM as a graded assessment task was quite positive. These results are very encouraging and will be further investigated as part of this ongoing research.