sliced_cramer_synaptic_consolidation_for_preserving_deeply_learned_representations.pdf (3.42 MB)
0/0

Sliced Cramer synaptic consolidation for preserving deeply learned representations

Download (3.42 MB)
conference contribution
posted on 24.03.2020 by Soheil Kolouri, Nicholas A Ketz, Praveen K Pilly, Andrea Soltoggio
Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks. We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic Weight Consolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network’s output at an arbitrary layer for previous tasks while learning a new one. We propose the sliced Cramer distance as a suitable ´ choice for such preservation and evaluate our Sliced Cramer Preservation (SCP) ´ algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks.

History

School

  • Science

Department

  • Computer Science

Source

International Conference On Learning Representations (ICLR 2020)

Version

AM (Accepted Manuscript)

Rights holder

© the authors

Acceptance date

14/02/2020

Language

en

Location

Addis Abeba, Ethiopia

Event dates

26th April 2020 - 30th April 2020

Depositor

Dr Andrea Soltoggio. Deposit date: 23 March 2020

Exports

Logo branding

Categories

Exports