entropy-24-00067.pdf (5.22 MB)
Download file

Evaluating the learning procedure of CNNs through a sequence of prognostic tests utilising information theoretical measures

Download (5.22 MB)
Deep learning has proven to be an important element of modern data processing technology, which has found its application in many areas such as multimodal sensor data processing and understanding, data generation and anomaly detection. While the use of deep learning is booming in many real-world tasks, the internal processes of how it draws results is still uncertain. Understanding the data processing pathways within a deep neural network is important for transparency and better resource utilisation. In this paper, a method utilising information theoretic measures is used to reveal the typical learning patterns of convolutional neural networks, which are commonly used for image processing tasks. For this purpose, training samples, true labels and estimated labels are considered to be random variables. The mutual information and conditional entropy between these variables are then studied using information theoretical measures. This paper shows that more convolutional layers in the network improve its learning and unnecessarily higher numbers of convolutional layers do not improve the learning any further. The number of convolutional layers that need to be added to a neural network to gain the desired learning level can be determined with the help of theoretic information quantities including entropy, inequality and mutual information among the inputs to the network. The kernel size of convolutional layers only affects the learning speed of the network. This study also shows that where the dropout layer is applied to has no significant effects on the learning of networks with a lower dropout rate, and it is better placed immediately after the last convolutional layer with higher dropout rates.

Funding

Ministry Education of Turkey

MIMIc: Multimodal Imitation Learning in MultI-Agent Environments

Engineering and Physical Sciences Research Council

Find out more...

History

School

  • Loughborough University London

Published in

Entropy

Volume

24

Issue

1

Publisher

MDPI AG

Version

VoR (Version of Record)

Rights holder

© The authors

Publisher statement

This is an Open Access Article. It is published by MDPI under the Creative Commons Attribution 4.0 Unported Licence (CC BY). Full details of this licence are available at: http://creativecommons.org/licenses/by/4.0/

Acceptance date

24/12/2021

Publication date

2021-12-30

Copyright date

2022

eISSN

1099-4300

Language

en

Depositor

Dr Xiyu Shi. Deposit date: 7 January 2022

Article number

67