Are course evaluations subject to a 'halo effect'?

Many course evaluations including those used in schools by OFSTED, and colleges and universities employ a number of scales as a means of evaluating various aspects of the educational experience of the student. It tends to be assumed students consider the scales independently. This paper argues students are influenced by a ‘halo effect’ when making judgements about the merits of aspects of a course. In this study student evaluations were obtained from 161 university lecturers attending probationary training courses. With a Likert style structured scale a favourable evaluation on one aspect of a course was shown to be linked with favourable evaluations on other aspects. Similarly with unfavourable evaluations different aspects were shown to correlate. This was not the case with an open ended evaluation format. There were no links between reactions on the two types of evaluations. Implications for the interpretation of course evaluations are discussed. Student evaluations of courses using scaled responses are commonly used in school by OFSTED inspectors and in colleges and universities as part of their external grading process. Research studies of course evaluations have a long history. An early example by Bassin (1974) included a series of Likert scales concerned with five aspects of teaching. These were lecture quality, exam quality, text suitability, participation and consideration. It was found that instructors of quantitative courses received lower ratings than those of non-quantitative courses. Pohlmann (1975) evaluated five aspects of courses, namely an overall view of how good the course was, how interested the tutor was in the student, how difficult the student found the course, whether assignments were clearly marked and how good the tutors actual presentation was. Pohlmann found undergraduate student’s evaluations were better on elective than required courses. This use of scales has continued with Rae (1997 p 113-125), and Shevlin et al (2000) recommending using structured scales. These researchers in common with many others assume the scales used are independent. This assumption is central to many course evaluations. Different aspects of courses are usually compared by looking at mean scores of scales which are thought to be independent. Sadly these may have little real value if the individual scores which make up those means are a result of responses on one scale being influenced by those on another. In an early review of the literature by Cohen (1981) there was evidence of some attempt to look at correlations between evaluation scales. This tended to be limited to very specific and predicted areas such as that between a favourable student rating of instructors’ skills and the student having received better grades. The issue of the independence of scales in general has, been neglected in more recent texts on course evaluation methodology (eg Holcomb 1998, Rae 2002, and Salas et al 2003), also in the more broadly based research methodology texts (eg. Fowler 2002, Hayes 2000 and Shaughnessey et al 2000). The present study questions the independence of measures on evaluation scales in the light of the ‘halo effect’, which is well known in the field of person perception but is not a concept which has been commonly applied to course evaluations. Blum and Naylor (1968 p. 200) see the ‘halo effect’ simply as the; ‘tendency to let our assessment of an individual on one trait influence our evaluation of that person on other specific traits’. This definition allows for any influence to be positive or negative.