Loughborough University
Browse
803_lifelong_reinforcement_learnin (3).pdf (4.58 MB)

Lifelong reinforcement learning with modulating masks

Download (4.58 MB)
journal contribution
posted on 2023-08-18, 14:40 authored by Eseoghene Ben-Iwhiwhu, Saptarshi NathSaptarshi Nath, Praveen Pilly, Soheil Kolouri, Andrea SoltoggioAndrea Soltoggio

Lifelong learning aims to create AI systems that continuously and incrementally learn during a lifetime, similar to biological learning. Attempts so far have met problems, including catastrophic forgetting, interference among tasks, and the inability to exploit previous knowledge. While considerable research has focused on learning multiple supervised classification tasks that involve changes in the input distribution, lifelong reinforcement learning (LRL) must deal with variations in the state and transition distributions, and in the reward functions. Modulating masks with a fixed backbone network, recently developed for classification, are particularly suitable to deal with such a large spectrum of task variations. In this paper, we adapted modulating masks to work with deep LRL, specifically PPO and IMPALA agents. The comparison with LRL baselines in both discrete and continuous RL tasks shows superior performance. We further investigated the use of a linear combination of previously learned masks to exploit previous knowledge when learning new tasks: not only is learning faster, the algorithm solves tasks that we could not otherwise solve from scratch due to extremely sparse rewards. The results suggest that RL with modulating masks is a promising approach to lifelong learning, to the composition of knowledge to learn increasingly complex tasks, and to knowledge reuse for efficient and faster learning.

Funding

United States Air Force Research Laboratory (AFRL) and Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8750-18-C-0103 (Lifelong Learning Machines) and Contract No. HR00112190132 (Shared Experience Lifelong Learning).

History

School

  • Science

Department

  • Computer Science

Published in

Transactions on Machine Learning Research

Pages

1 - 23

Publisher

Transactions on Machine Learning Research (TMLR)

Version

  • VoR (Version of Record)

Publisher statement

This is an Open Access Article. It is published by Transactions on Machine Learning Research (TMLR) under the Creative Commons Attribution 4.0 International Licence (CC BY). Full details of this licence are available at: https://creativecommons.org/licenses/by/4.0/

Acceptance date

2023-06-23

Publication date

2023-07-20

Copyright date

2023

eISSN

2835-8856

Publisher version

Language

  • en

Depositor

Dr Andrea Soltoggio. Deposit date: 1 July 2023

Usage metrics

    Loughborough Publications

    Categories

    No categories selected

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC