Loughborough University
final_inspire_april-14.pdf (96.86 kB)

The need for evidence innovation in educational technology evaluation

Download (96.86 kB)
conference contribution
posted on 2014-09-03, 12:54 authored by Melanie KingMelanie King, Ray DawsonRay Dawson, Firat BatmazFirat Batmaz, Steve Rothberg
More complex and chaotic methods are being adopted in the development of technology to enhance learning and teaching in higher education today in order to achieve innovation in teaching practice. However, because this type of development does not conform to a linear process-driven order, it is notoriously difficult to evaluate its success as a holistic educational initiative. It is proposed that there are five factors that impact on effective educational technology evaluation, which contributes to insubstantial evidence of positive outcomes, these being: premature timing; inappropriate software evaluation techniques and models; lack of shared understanding of the terminology or the semantics of education technology; the growing complexity of agile and open development; and the corporatisation of higher education. This paper suggests that it is no longer helpful for policy makers to evaluate whether educational technology project outcomes were successful or unsuccessful but instead they should use agile evaluation strategies to understand the impact of the product, process and outcomes in a changing context. It is no longer useful to ask the question, ‘did the software work?’ The key is for software developers and policy-makers to ask ‘what type of software works, in which conditions and for whom?’ To understand this, the software development community needs to look at adopting evaluation strategies from the social science community. For example, realist evaluation supplies context driven and evidence-based techniques, exploring outcomes that tend towards the social rather than technical. It centres on exploring the ‘mechanisms’, ‘contexts’ and ‘outcomes’ associated with an intervention and is a form of theory-driven evaluation that is the theory and reasoning of its stakeholders that is rooted in practitioner wisdom.



  • University Academic and Administrative Support

Research Unit

  • Centre for Engineering and Design Education (CEDE)

Published in

International Conference on Software Process Improvement Research and Education (INSPIRE 2014) Proceedings of INSPIRE XIX: Global Issues in IT Education


9 - 23 (15)


KING, M ... et al., 2014. The need for evidence innovation in educational technology evaluation. IN: Uhomoibi, J. ... et al. (eds.) Proceedings of INSPIRE XIX: Global Issues in IT Education, Southampton, UK, 15 April 2014, pp. 9 - 23.


British Computer Society


  • AM (Accepted Manuscript)

Publisher statement

This work is made available according to the conditions of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) licence. Full details of this licence are available at: https://creativecommons.org/licenses/by-nc-nd/4.0/

Publication date



This is a conference paper. It was presented at the 19th Annual INSPIRE conference.




  • en



Usage metrics

    Loughborough Publications


    No categories selected