Beyond validation: assessing the legitimacy of artificial neural network models
Artificial neural network models have been used extensively for prediction and forecasting over the last 25 years. As the data used to develop ANNs contain important information about the physical processes being modelled, it is generally implied that a model that has been calibrated (trained) and performs well on an independent set of validation data represents the underlying physical processes of the system being modelled.
However, this is not necessarily the case, most likely due to problems with equifinality, where different combinations of model parameters (e.g. connection weights) result in similar predictive performance.
Consequently, there is also a need to check the behaviour of calibrated ANN models as part of the validation process, which is commonly referred to as structural, conceptual or scientific validation (Figure 1). This checks whether the input-output relationship captured by the model is plausible in accordance with a priori system understanding.
In this paper, the importance of considering structural validation is demonstrated. This is achieved by developing ANN models with different numbers of hidden nodes for two environmental modelling case studies from the literature namely, salinity forecasting in the River Murray in South Australia and the prediction of treated water turbidity at a water treatment plant based on raw water quality and the administered alum dose.
The validation errors are then compared with corresponding model behaviours. This was done using the validann R-package, which caters to a range of structural validation approaches. Results show that ANN models producing the best fit to the data do not necessarily result in models that behave in accordance with underlying system understanding.
- Computer Science
Published inProceedings of the 24th International Congress on Modelling and Simulation
Pages85 - 91
Source24th International Congress on Modelling and Simulation
PublisherModelling and Simulation Society of Australia and New Zealand (MSSANZ)
- AM (Accepted Manuscript)
Rights holder© The Authors and Modelling and Simulation Society of Australia and New Zealand Inc. (MSSANZ)
Publisher statementThese proceedings are licensed under the terms of the Creative Commons Attribution 4.0 International CC BY License (http://creativecommons.org/licenses/by/4.0), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you attribute MSSANZ and the original author(s) and source, provide a link to the Creative Commons licence and indicate if changes were made. Images or other third party material are included in this licence, unless otherwise indicated in a credit line to the material.