Developing a novel process for producing optimised real-world and synthetic training datasets designed for application driven machine learning

Computer vision systems are implemented in many applications and require high tests performance, especially in applications linked to safety. A key component in developing any system is the training dataset, which is the set of images used by the system to learn how to detect the desired objects. When images are collected for training datasets it is often seen that little to no thought is put into how the conditions under which the images are taken can affect the appearance of features, and therefore affect training and test performance. This can result in many training datasets causing sub-optimal test performance because they include images taken under undesirable conditions. Images can be affected by many factors, for example the environmental conditions related to lighting. Datasets using synthetic images can be affected by simulation factors, for example the rendering engine used. As little research exists into the effects of different factors, for both real-world and synthetic datasets, there is little understanding into how factors affect test performance, how they may interact, and how the effects and interactions can be identified through a robust process.

The thesis explores and demonstrates the importance of considering the effects on test performance, whilst establishing a process which can be used to identify the effects and interactions between various factors. This is taken further by investigating how the information can be used to make/identify an optimised training dataset. The thesis shows this through three hypothetical application based case studies which use the convolutional neural networks.

Case study 1 concentrates on detecting construction machines to investigate the initial process used for identifying the effects and interactions of environmental factors in a real-world dataset, and simulation factors in a synthetic dataset.

Case study 2 focuses on general litter items to investigate any commonalities between different applications. Furthermore, the synthetic dataset uses more factors to establish how the process needs to adapt as the number of factors being investigated increases.

Case study 3 uses LEGO® to create objects of varying complexity, which enables investigations into how the effect of factors change as the complexity of the objects in a scene also changes.

Each of the case studies provides numerous insights into how the factors influence the test performance and defines an overarching process on how to gather images and investigate the effects and interactions for use in other applications.