A Multimodal Perception Driven Self Evolving Autonomous Ground Vehicle (1).pdf (9.7 MB)
Download file

A multimodal perception-driven self evolving autonomous ground vehicle

Download (9.7 MB)
journal contribution
posted on 05.11.2021, 13:46 by Jamie RocheJamie Roche, Varuna De-SilvaVaruna De-Silva, Ahmet KondozAhmet Kondoz
Increasingly complex automated driving functions, specifically those associated with Free Space Detection (FSD), are delegated to Convolutional Neural Networks (CNN). If the dataset used to train the network lacks diversity, modality or sufficient quantities, the driver policy that controls the vehicle may induce safety risks. Although most autonomous ground vehicles (AGV) perform well in structured surroundings, the need for human intervention significantly rises when presented with unstructured niche environments. To this end, we developed an AGV for seamless indoor and outdoor navigation to collect realistic multimodal data streams. We demonstrate one application of the AGV when applied to a self-evolving FSD framework that leverages online active machine learning (ML) paradigms and sensor data fusion. In essence, the self-evolving AGV queries image data against a reliable data stream, ultrasound, before fusing the sensor data to improve robustness. We compare the proposed framework to one of the most prominent free space segmentation methods, DeepLabV3+ [1]. DeepLabV3+ [1] is a state-of-the-art semantic segmentation model composed of a CNN and an auto-decoder. In consonance with the results, the proposed framework out preforms DeepLabV3+ [1]. The performance of the proposed framework is attributed to its ability to self-learn free space. This combination of online and active ML removes the need for large datasets typically required by a CNN. Moreover, this technique provides case-specific free space classifications based on information gathered from the scenario at hand.


MIMIc: Multimodal Imitation Learning in MultI-Agent Environments

Engineering and Physical Sciences Research Council

Find out more...



  • Loughborough University London

Published in

IEEE Transactions on Cybernetics


1 - 11


Institute of Electrical and Electronics Engineers (IEEE)


AM (Accepted Manuscript)

Rights holder


Publisher statement

Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Acceptance date


Publication date









Dr Varuna De Silva. Deposit date: 4 November 2021