The science of recordkeeping systems - a realist perspective

The mass adoption of email systems in the 1990s brought with it a revolution in communications. For the first time in human history the majority of written communications were sent by digital rather than analogue means. This communications revolution had a huge impact on recordkeeping theory and practice. It stimulated a mini-renaissance of records management theory but was also disruptive and contentious.

Rival paradigms emerged within the recordkeeping profession. One paradigm argued that email systems were purely communication tools, email accounts were little more than electronic pigeon holes, and that an email became a record only if (and when) a decision was made to capture it into a dedicated records repository. A rival paradigm argued that email systems formed part of an organisation’s record system, that email accounts would function as record aggregations (whether organisations wanted them to or not) and that emails were records from the moment they were sent from, or received into, an email system.

The points of contention between these rival paradigms touch upon some fundamental questions of archival science, such as what it means to create and capture a record, and how records should be organised.

These points of contention cannot be resolved by existing recordkeeping theory. In the 1990s Frank Upward developed a records continuum model that conceptualised how recordkeeping processes worked across an entire society. This is an enormously useful middle-range theory whose breadth and flexibility has enabled it to outlive most of the other theoretical outputs of the 1990s in the recordkeeping domain. But Upward’s model does not provide a close enough view of an organisational record system to enable the consequences of different policy choices to be assessed and compared.

This study starts with a selective critical review of the history of ideas on how recordkeeping systems work. It attempts to place the ideas of the major theorists of recordkeeping systems (Jenkinson, Schellenberg, Scott, Bearman, Duranti and Upward) within the communications and recordkeeping context prevailing at the time they wrote.

The study proceeds to evaluate rival records management approaches towards email, using Ray Pawson’s realist methodologies (realist evaluation and realist review). A key insight of realism in the social sciences is that no social programme works for all stakeholders at all times. The study identifies the theoretical, empirical and legal points of contention between the main rival policy approaches to email, and adjudicates on them by means of thought experiment and/or literature review.

On the basis of the review of recordkeeping theory, and the evaluation of rival policies towards email, this study develops two new models of how an organisational record system works - the record system matrix, and the record system trade-offs model.

The record system matrix ‘zooms-in’ Upward’s record continuum model to show the record system of an organisation as an open system embedded in a recordkeeping society. The matrix depicts the sixteen step implementation chain of an organisational record system, presented as a four by four matrix, which shows the actions needed from social actors (policy makers, practitioners, and end-users) in order for an organisation to be able to conduct communication transactions, capture records of those transactions, aggregate those records, and apply rules to the records via those aggregations.

The record system trade-offs model shows the features a record system must possess in order to be seen to be reliable, efficient, precise and predictable enough to meet the needs of an originating organisation and its stakeholders. The model shows the trade-offs that organisations are faced with in circumstances where it is not possible to optimise a record system to meet all four of these imperatives.

The study uses the two new models to assess the likely impact on recordkeeping of the possibilities offered by advances in artificial intelligence (AI). It looks at the possibility that AI could be used to re-aggregate records within a record system at any point in their lifecycle. This study predicts, via a series of thought experiments, that the original aggregation of email correspondence into email accounts, because it sets the expectations of end-users as to how their correspondence will be managed for access and retention purposes, will continue to influence and constrain recordkeeping policy towards email even after such AI capabilities are acquired.

Insights from the two models developed in this study, and from the AI specific thought experiments conducted in this study, are used to chart an incremental route whereby AI capabilities could be applied within email accounts with a view to progressively improving the precision and efficiency with which emails are managed, whilst minimising the risk of acting counter to end-user expectations.