Using computer-aided assessment to test higher level learning outcomes

This paper sets out an approach using a revised Bloom's taxonomy of learning objectives for the careful design of objective questions to assist in the assessment of higher learning outcomes (HLO’s) and details the creation and evaluation of a variety of such questions. This has been done within the context of two constraints; the use of popular, commercially available computer-aided assessment (CAA) software, specifically Question Mark Perception and Half-Baked Hot Potatoes, and the assumption of limited learning technologist support. It examines the problems inherent in the design of objective questions for HLO’s (specifically at the levels of Application, Analysis and Evaluation) and introduces a framework by which systematic design may be carried out. It examines the mode of design, construction and evaluation of 22 such objective questions devised for formative assessment for two post-graduate course units in Information Systems, involving two groups of students, 37 in total. The results from the trialing of these questions raises issues of crucial differences in CAA software for feedback, scoring and delivery. The paper examines key statistical indicators of question quality (facility and discrimination indices), as well as the results of interviews with students, to drawn conclusions about the best use of question types, student preparation for tests, and discusses key issues in question preparation. It concludes with an examination of the main advantages and disadvantages of using CAA for HLO’s referring to the resource overheads needed and the problems inherent in the process.