Visual constituent analysis: a novel approach to annotating, constructing, and analysing large-scale visual and text-visual corpora

This thesis outlines the rationale, development, testing, and subsequent deployment of visual constituent analysis, a novel dualistic approach to annotating, constructing, and analysing large-scale multimodal text-visual corpora. Multimodal imagery is a feature that has been largely ignored in corpus linguistic research, primarily because of the difficulties inherent to annotating and analysing visual content at scale. In this thesis, I present two methods that attempt to solve these issues. The first, constituent tagging, serves as a novel, metadata-based approach to annotating images in a manner that ensures homogeneity and accessibility while retaining scale. The second, constituent analysis provides the first truly integrated large-scale approach to text-visual analysis through the concatenation of constituent tags and co-textual elements. Together, these two novel corpus linguistic tools allow researchers to transfer seamlessly between horizontal and vertical readings of text-visual corpora. Constituent tagging provides researchers immediate access to image filtering and searching, enabling a data driven rather than hypothesis-driven approach to multimodal corpus construction and analysis. Constituent analysis retains the quality of close in-context reading of imagery and text while adding the additional element of identifying quantitative trends. The effectiveness of constituent tagging and analysis as tools in corpus linguistic analysis of multimodal discourse is particularly evident in the analysis of over 60,000 tweets of multimodal hate speech, where the approach highlights trends and biases in the annotation process that are not otherwise apparent when using text-focused corpus linguistic tools or vision-based neural networks.