An experiment to review the quality of Web-based multimedia material

Currently, many educators are focusing on the development of Web-based material and the quality of such material needs to be evaluated. Expert review, as a formative evaluation method, is an important method to evaluate the material prior to release. Other studies have described the use of Subject-Matter Experts (SMEs) and also domain students, with enough knowledge in the subject area, as a way of enhancing the quality of the reviewed material. In addition to the SMEs and domain students a lecturer and graduate student, both in the subject discipline area, were used to review Web-based lectures on Advanced Computer Architectures. Both additional reviewers had sufficient knowledge of the discipline. The research investigates three main hypotheses: I) whether a review conducted by domain reviewers (SME and domain student) or discipline reviewers improves the quality of material, 2) whether using discipline reviewers improves the quality of the material more than domain reviewers and 3) whether there are differences in the quality of the material resulting from students' and lecturers' review. Five versions of the material were developed and used as inputs to an experiment that was designed to test the hypotheses by using questionnaires and tests. In summary, the findings of the research were that: SME reviewer and discipline reviewers were effective in producing higher quality than the unreviewed material. However, rather interestingly, the domain student was shown to be the least effective in enhancing the material. The investigation also found that there was no significant difference in the quality of the material resulting from students' and lecturers' review. Furthermore, it was found that the media used to present the material was more useful when the quality of that media was perceived to be high. Finally 90% of the subjects were willing to study Web-based lectures as part of the course.