Feature selection of dynamic systems based on model parsimony: Analysis of a chaotic system using synergistic mutual information and simplified probability functions

The first major task in any statistical modelling endeavour is to specify the model’s structure. For complex systems about which formal models are unavailable or insufficient, this represents a substantial task. This is the specific class of “black-box” problems in the domain of dynamic systems modelling.

In such cases, a recommended objective of structural specification is model parsimony. Such an approach minimises the expected cost of a model in out-of-sample estimation. This thesis investigates the first essential step of parsimonious model specification: the selection of information rich features with which to build dynamic models.

The following work demonstrates a synergistic, information-theoretic framework for feature selection of non-linear systems which is based on Joint Mutual Information (JMI). It is used to identify feature sets with which to model a state derivative of the Lorenz Attractor, a chaotic dynamic system that is selected due to its non-linearity and multi-colinearity which make a linear correlation analysis difficult. The methodology also varies the degree of partitioning for the target variable’s probability mass function. This is used to identify the relationship between the feature selection result and partitioning complexity.

The results of this analysis show that the optimal feature set of the attractor may be identified via binary partitioning of the target variable, centred at a value of zero. The initial choice of a binary partition is heuristic, and based on control engineering intuitions. To further investigate this result, a sensitivity analysis is used to demonstrate that the best feature set maximises normalised JMI with respect to partitioning complexity of the target variable distribution, reducing the uncertainty about the target variable by 96% under a 2-bin discretisation.

These findings demonstrate that notions of parsimony can be applied to certain dynamic feature selection problems, in addition to the model structuring process. This may be enabled by a synergistic, non-linear selection criterion, with some limited but informed assumptions about the structure of target variable probability distributions.