Perspective view of autonomous control in unknown environment: dual control for exploitation and exploration vs reinforcement learning
This paper overviews and discusses the relationship between Reinforcement Learning (RL) and the recently developed Dual Control for Exploitation and Exploration (DCEE). It is argued that there are two related but quite distinctive approaches, namely, control and machine learning, in tackling intractability arising in optimal decision making/control problems. In the control approach, the original problems (of an infinite horizon) are approximated by finite horizon problems and solved online by taking advantage of the availability of computing power. In the machine learning approach, the optimal solutions are approximated through iterations, or (offline) training through trials when models are not available. When dealing with unknown environments, DCEE as a technique developed from the control approach could potentially solve similar problems as RL while offering a number of advantages, most notably, coping with uncertainty in environment/tasks, high efficiency in learning through balancing exploitation and exploration, and potential in establishing its formal properties like stability. The links between DCEE and other relevant methods like dual control, Model Predictive Control and particularly Active Inference in neuroscience are discussed. The latter provides a strong biological endorsement for DCEE. The methods and discussions are illustrated by autonomous source search using a robot. It is concluded that DCEE provides a promising, complementary approach to RL, and more research is required to develop it as a generic theory and fully realise its potential. The relationships revealed in this paper provide insights into these relevant methods and facilitate cross fertilisation between control, machine learning and neuroscience for developing autonomous control under uncertain environments.
Funding
Goal-Oriented Control Systems (GOCS): Disturbance, Uncertainty and Constraints
Engineering and Physical Sciences Research Council
Find out more...History
School
- Aeronautical, Automotive, Chemical and Materials Engineering
Department
- Aeronautical and Automotive Engineering
Published in
NeurocomputingVolume
497Pages
50 - 63Publisher
ElsevierVersion
- VoR (Version of Record)
Rights holder
© The AuthorPublisher statement
This is an Open Access Article. It is published by Elsevier under the Creative Commons Attribution 4.0 International Licence (CC BY). Full details of this licence are available at: https://creativecommons.org/licenses/by/4.0/Acceptance date
2022-04-30Publication date
2022-05-05Copyright date
2022ISSN
0925-2312Publisher version
Language
- en