Automated sewer pipe defect tracking in CCTV videos based on defect detection and metric learning

© 2020 Elsevier B.V. Computer vision techniques are widely studied for automating the interpretation of sewer pipe inspection videos, yet previous studies mainly focus on defect detection and segmentation of individual images, which cannot identify if the defect is the same one across consecutive video frames (i.e. track the defect). Nevertheless, the number of unique defects in the video is required for evaluating the pipe condition. This paper proposes a framework for tracking multiple sewer defects in CCTV videos based on defect detection and metric learning. First, a deep learning -based defect detection model and a metric learning model is developed and trained respectively using with our sewer datasets. Then, using the detections and their features from the trained models as inputs, the tracking module predicts tracks by Kalman filter and associates tracks based on defect motion, appearance features, and defect types. Our experiments demonstrate the framework is able to track sewer defects in CCTV videos with a decent IDF1 score of 57.4%. We notice that tracking performance can be influenced by the detection accuracy and configurations of the metric learning module. By analyzing the tracking results based on different weights of the distance metrics, we find that assigning larger weights to appearance and defect class distance metrics tends to increase IDF1 score, while larger motion distance weight may degrade tracking accuracy. The proposed framework contributes by tracking multiple sewer defects, which can assist with counting unique defects in inspection videos.