Design factors affecting relevance judgment behaviour in the context of metadata surrogates

This paper reports on the findings of a user study which investigated the effects of structure, highlighting and quantity of information on university students’ interaction with metadata surrogates of learning object repositories (LORs) during the relevance judgement process. The literature review revealed a lack of studies investigating how metadata surrogates should be designed to meet users’ needs and improve the effectiveness of relevance judgement. In order to address this issue, different versions of a prototype system, called METALOR 2, were developed and evaluated in terms of the time needed for users to find relevant information, the accuracy of their relevance judgements and their satisfaction. To evaluate the prototype, a user study was conducted where participants were asked to complete a set of tasks and fill in satisfaction questionnaires. The findings showed that participants performed better and were more satisfied with those versions of the prototype which highlighted metadata elements that included the query terms, used clusters or categories to organize metadata elements in the surrogates and included metadata elements which were relevant to the query or task at hand. The paper concludes with some recommendations for improving the design of metadata surrogates in search result interfaces.