Computer Vision for Protest Analysis

Abstract

How can computer vision help us to understand protests better? Every day, people take to the streets to protest, and images of these events are shared thousands of times on social media. While qualitative studies have effectively demonstrated that protests are diverse and highly dynamic, quantitative research faces the challenge of capturing this nuanced information. However, protest images offer a unique opportunity to do so, as each image provides detailed documentation of what is happening at a particular time and place. Since these images are shared thousands of times on protest days, they can be used to reconstruct the events as they unfold. Researchers have rarely analyzed these images due to the difficulty of extracting protest-related information from them. Fortunately, recent advances in computer vision are changing this landscape. Computers are now capable of performing many visual tasks, including extracting high-level insights from images and videos. Dedicated models have already been trained to recognize protest images and assess the level of violence depicted in them. Additionally, many generic models can be adapted from computer science to applications in social sciences. For instance, segmentation models can identify a wide range of objects in images, such as people and faces. Although these tasks could theoretically be performed manually, the large scale of images on social media renders this infeasible. Therefore, researchers increasingly rely on computer vision methods to efficiently extract information from these images. This dissertation explores different applications in which computer vision enhances our understanding of protests. To achieve this, readily available computer vision methods are adopted, trained, and optimized specifically for analyzing protest images. These methods facilitate the extraction of various characteristics from these images, enabling a deeper analysis of the protests themselves. A distinct image dataset complements each method. The first dataset comprises more than 140,000 images collected from social media, with annotations indicating whether each image depicts a protest or not. This dataset aims to provide a comprehensive overview across ten different countries. The second dataset focuses on capturing protest periods in specific cities, covering 13 protest episodes and incorporating approximately 22,000 images. The findings reveal that persons, flags, and signboards are important objects in protest images. But particular features of protests vary across different countries and protest episodes. The results also indicate that the escalation of protest events can be tracked through images shared on social media, allowing for predictions of protest dynamics on the same day. However, predictions for the following day show only marginal improvements. Experimental results highlight how individuals perceive protests through sequences of images. If generative computer vision models manipulate crowds in these protest images, it threatens public perception, as estimates of crowd sizes become distorted. Overall, these findings expand our understanding of protests in a world saturated with visual information, opening exciting avenues for future research in protest studies and other fields of social science.