Plant classification combining colour and spectral cameras for weed control purposes

This study was conducted to design and evaluate a novel dual camera sensor for use in an accurate single leaf level plant detection and classification system for weed control purposes. The system was to utilise and combine the benefits of colour and spectral imaging technologies together with novel data processing techniques. Such combination of colour and spectral imaging devices has not been previously used in precision agriculture. Environmental consciousness and requirements for production volumes of organic produce are constantly increasing. Reductions or total elimination of chemical spraying is needed, and a technological solution of automating the weed control has been seen as one solution to solve the limitations in current crop production methods. Recent studies have shown automatic plant detection and classification to be the only economically viable solution for the problem of automatic weed control. Previous detection systems have shown adequate capabilities to detect and classify weeds and crop plants with certain limitations. Depending on the system, these limitations have been in spatial accuracy, operation in certain lighting conditions or selection of plants to be classified. A flexible system capable of robust plant classification under any circumstances and plant combinations has not yet been realised. It would be desirable to introduce a system capable of detecting any plant species and plants separately, thus allowing targeted and optimal weed control methods for each plant species. The proposed system addressed the problem of automatic plant detection and classification by providing sub-centimetre level information on plant part locations separately for each plant species. This information could then be directly used to guide mechanical weeding tools or precision sprayers. The detection system was based on a novel combination of a sub-millimetre level colour camera and an accurate hyperspectral line scanning camera (spectrometer) in the spectral range of 400 – 1000 nm. The spatial accuracy of the spectrometer was approximately five times lower than that of the colour camera. The system operated under controlled lighting conditions. The colour camera allowed precise segmentation of plant borders, while the spectral camera produced detailed reflectance information to discriminate between plant types. The system was able to collect data for classification from an area on a plant of approximately 6.5 mm by 6.5 mm, although typically areas as small as 3.5 mm by 3.5 mm were detected. These were also the spatial resolution boundaries of the system with the used test settings. The system was first designed and evaluated in laboratory conditions using controlled lighting and a selection of leaves from 6 plants. Data collection and analysis methods were designed for a scanning system with simultaneous image acquisition from both cameras. Shape, colour and spectral reflectance information were used to correctly classify these individual leaves with a probability of up to 98% using linear stepwise discriminant analysis. A method of classifying separate leaves is not robust in a real field environment where plant leaves are often overlapping. This makes the use of shape calculations difficult. A novel method of extracting data from the small windows was proposed. Colour and spectral data within these windows was classified separately and the windows formed a grid like structure with approximately 3.5 mm spacing between them. This allowed spatial filtering of the classification data and noise reduction for the results by utilising information in a 3 by 3 window neighbourhood around each data window. During laboratory tests the windows for 6 plant leaves were correctly classified at 97.8% when the leaves were separated, and at 85.2% with overlapping leaves. The system operation was also evaluated in real field conditions. Four crop plants and 16 weed plant types were imaged on a field over a period of 11 to 25 days after sowing. Total average classification performance in field conditions with linear discriminant analysis was up to 85.1%, while classification results investigated as a two-class case of crop vs. weed plants was up to 99.5% and 83.8%, respectively. The spatial filtering method was shown to improve results on average by 7.5%. Plant reflectance measurements on different days allowed a novel analysis of short term temporal changes in the plant spectra due to growing conditions and growth stages. Analysis on short term spectral changes and their effects on classification accuracies have not been found in the literature. The temporal analysis showed that the average spectra of any plant type changes considerably over a period of just few days, and has a trend like behaviour when investigated at individual wavelengths. Classification models with training set data from previous days did not perform well. This indicates the need to have an up to date training set available at all times explaining the subtleties in local conditions. The proposed detection and classification system with intelligent data processing methods has been shown to perform at a comparable level with previous systems. The novel system does not suffer from the typical limitations of previous systems, and is flexible to be used with any plant types in their early growth stages. There is also potential to include plant height, shape or any other relevant feature to the classification for increased robustness. The presented data processing method allows considerable processing and data reductions within the camera hardware. Only small fractions of the processed image data would need to be transferred via the camera interface. This would compensate for the increased data flow created by using two cameras. Therefore, the real-time implementation of the system is thought possible with the right hardware choices and optimised data processing algorithms.