Evaluation of a small-scale implementation of Computer-assisted Assessment

Due to the sheer number of students who are enrolled for the “Introduction to Computer Applications” module in the Department of Computer Science and Information Systems at Chester College of Higher Education, the use of computerassisted assessment (CAA) was investigated during 1999/2000. This investigation began as an MSc project of a member of staff to implement and evaluate the introduction of CAA into the Department. The Department is keen on encouraging an e-Learning environment and seized the opportunity to use the research to evaluate the use of computer-assisted assessment on the “Computer Applications” module in the first semester of 2000/2001. The purpose of the introduction of CAA into the Department was to evaluate the benefits and/or difficulties likely to be experienced on such a small scale. The purpose of the evaluation of CAA on this module was to measure the students’ attitude towards CAA and their perceived confidence pre and post treatment. A database of appropriately coded questions was developed, some of the questions having been tested (for discrimination index and facility value). CAA software was then investigated and quizzes were designed in Coursebuilder, a plug-in to Macromedia’s web authoring application, Dreamweaver. These quizzes were uploaded onto the department web server and administered to one of three sets of students taking the first year module. Since the number of students taking the module was expected to be high it was appropriate to select a between-subjects design for the evaluation of the CAA use, using a ‘treatment’ group and a ‘non-treatment’ group, the latter group receiving the computer-assisted assessment. Data collection methods used included pre- and post-test confidence logs, together with an attitude survey and comparison of the final marks of the two groups. To further improve validity, a focus group interview was carried out with some of the students in the treatment group and the tutor was interviewed separately. This paper outlines the difficulties encountered during the implementation of the technology, describes the methodology and analyses the results obtained.