Loughborough University
Browse
2405.19047v1.pdf (3.38 MB)

Statistical context detection for deep lifelong reinforcement learning

Download (3.38 MB)
conference contribution
posted on 2024-06-26, 15:43 authored by Jeff DickJeff Dick, Saptarshi NathSaptarshi Nath, Christos PeridisChristos Peridis, Eseoghene Benjamin, Soheil Kolouri, Andrea SoltoggioAndrea Soltoggio

Context detection involves labeling segments of an online stream of data as belonging to different tasks. Task labels are used in lifelong learning algorithms to perform consolidation or other procedures that prevent catastrophic forgetting. Inferring task labels from online experiences remains a challenging problem. Most approaches assume finite and low-dimension observation spaces or a preliminary training phase during which task labels are learned. Moreover, changes in the transition or reward functions can be detected only in combination with a policy, and therefore are more difficult to detect than changes in the input distribution. This paper presents an approach to learning both policies and labels in an online deep reinforcement learning setting. The key idea is to use distance metrics, obtained via optimal transport methods, i.e., Wasserstein distance, on suitable latent action-reward spaces to measure distances between sets of data points from past and current streams. Such distances can then be used for statistical tests based on an adapted Kolmogorov-Smirnov calculation to assign labels to sequences of experiences. A rollback procedure is introduced to learn multiple policies by ensuring that only the appropriate data is used to train the corresponding policy. The combination of task detection and policy deployment allows for the optimization of lifelong reinforcement learning agents without an oracle that provides task labels. The approach is tested using two benchmarks and the results show promising performance when compared with related context detection algorithms. The results suggest that optimal transport statistical methods provide an explainable and justifiable procedure for online context detection and reward optimization in lifelong reinforcement learning. 

History

School

  • Science

Department

  • Computer Science

Source

Conference on Lifelong Learning Agents (CoLLAs) 2024

Publisher

CoLLAs

Version

  • AM (Accepted Manuscript)

Publisher statement

Shared with permission from the publisher.

Acceptance date

2024-04-26

Publisher version

Language

  • en

Location

Pisa, Italy

Event dates

29th July 2024 - 1st August 2024

Depositor

Dr Andrea Soltoggio. Deposit date: 30 May 2024

Usage metrics

    Loughborough Publications

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC