TROLL PATROL FINDINGS
Using Crowdsourcing, Data Science & Machine Learning to
Measure Violence and Abuse against Women on Twitter
Welcome to the findings of our Troll Patrol project: a joint effort by human rights researchers, technical experts and thousands of online volunteers to build the world’s largest crowd-sourced dataset of online abuse against women.
Our findings reveal the sheer scale and nature of online abuse faced by women and provides a resource to researchers and engineers interested in exploring the potential of machine learning in content moderation.
These findings are the result of a collaboration between Amnesty International and Element AI, a global artificial intelligence software product company. Together, we surveyed millions of tweets received by 778 journalists and politicians from the UK and US throughout 2017 representing a variety of political views, and media spanning the ideological spectrum. Using cutting-edge data science and machine learning techniques, we were able to provide a quantitative analysis of unprecedented scale of online abuse against women in the UK and USA.
Our findings reinforce Amnesty International’s previous research into online abuse and violence against women online. Specifically, we found that:
- 7.1% of tweets sent to the women in the study were “problematic” or “abusive”. This amounts to 1.1 million tweets mentioning 778 women across the year, or one every 30 seconds.
- Women of colour, (black, Asian, Latinx and mixed-race women) were 34% more likely to be mentioned in abusive or problematic tweets than white women.
- Black women were disproportionately targeted, being 84% more likely than white women to be mentioned in abusive or problematic tweets.
- Online abuse targets women from across the political spectrum - politicians and journalists faced similar levels of online abuse and we observed both liberals and conservatives alike, as well as left and right leaning media organisations, were targeted.
Note: This analysis is specific to the group of women in the study and would likely differ if applied to other professions, countries or the wider population.
Online abuse against women on this scale should not and does not have to exist on social media platforms. Companies like Twitter have a responsibility to respect human rights, which means ensuring that women using the platform are able to express themselves freely and without fear.
Amnesty International has repeatedly asked Twitter to make available meaningful and comprehensive data regarding the scale and nature of abuse on their platform, as well as how they are addressing it. Such data will be invaluable for anyone seeking to understand and combat this barrier to women’s human rights online. In light of Twitter’s refusal to do so, it is our hope that this project will shed some insight into the scale of the problem.
Twitter - a toxic place for women
For the past two years, Amnesty International has documented the scale and nature of abuse and violence against women online. In 2017, we commissioned an online polling of women in 8 countries about their experiences of abuse on social media platforms and used data science to analyse the abuse faced by female Members of Parliament (MPs) on Twitter prior to the UK’s 2017 snap election.
In March 2018, we released our report Toxic Twitter: Violence and abuse against women online which outlines the human rights harms facing women on Twitter, and proposed concrete steps that the company can take to address these harms. The report found that as a company, Twitter is failing in its responsibility to respect women’s rights online by failing to adequately investigate and respond to reports of violence and abuse in a transparent manner which leads many women to silence or censor themselves on the platform.
We are experiencing a watershed moment with women around the world using their collective power to speak out about the abuse they face and amplify their voices through social media platforms. However, Twitter’s failure to effectively tackle violence and abuse on the platform has a chilling effect on freedom of expression online and undermines women’s mobilisation for equality and justice – particularly groups of women who already face discrimination and marginalisation.
Amnesty International has repeatedly urged Twitter to publicly share comprehensive and meaningful information about reports of violence and abuse against women, as well as other groups, on the platform, and how they respond to it.
On 12 December 2018 Twitter released an updated Transparency Report in which it included for the first time a section on 'Twitter Rules Enforcement'. This was one of Amnesty International’s key recommendations to Twitter and we see the inclusion of this data as an encouraging step. The disclosure of this data sheds some light on levels of abuse reported on the platform, and how the company responds to it.
We are disappointed, however, that the information provided in the transparency report does not go far enough. For instance, the data Twitter includes on enforcement is not disaggregated by the category of abuse reported, but is simply aggregated in two categories of ‘abuse’ and ‘hateful conduct’. Meaningful data on how Twitter responds to the different types of reports of abuse available in its reporting mechanism will increase understanding of how different categories of abuse, in particular tweets that direct hate against a protected category, are responded to. The data in Twitter’s Transparency report also fails to include any information about how long it takes moderators (on average) to respond to reports of abuse, how many decisions were overturned in an appeals process or any information about the number of content moderators employed per region and language to respond to reports of abuse on the platform.
In addition, Twitter's data states that 2.8 million unique accounts were reported for abuse of which Twitter actioned 248,000 – approximately 9%. However, the data published only reflects unique accounts that were reported for abuse and actioned. Twitter should also publish the total number of tweets reported for abuse and hateful conduct - disaggregated by category - in order to avoid potentially underplaying the true scale of abuse on the platform. Knowing more complete and robust figures on reports of abuse and hateful conduct on Twitter will ultimately allow policy makers, civil society and technologists to develop more concrete and comprehensive solutions to tackle the problem.
This study is an attempt to - in part - remedy this lack of data.
Crowd-sourcing the world’s biggest dataset of online abuse targeting women
Following the launch of our Toxic Twitter report in March 2018, Amnesty International launched Troll Patrol, a global crowdsourcing effort to demonstrate the scale and nature of abuse that women continue to experience on Twitter.
Working with Element AI, we first designed a large, unbiased dataset of thousands of tweets mentioning 778 women politicians and journalists from the UK and US. The sample of women selected included all parliamentarians from the UK Parliament and US Congress and Senate as well as women journalists from publications like the Daily Mail, Gal Dem, the Guardian, Pink News, the Sun in the UK and Breitbart and the New York Times in the USA.
More than 6,500 digital volunteers from around the world then took part in Troll Patrol, analysing 288,000 unique tweets to create a labelled dataset of abusive or problematic content. The volunteers were shown an anonymized tweet mentioning one of the women in our study, then were asked simple questions about whether the tweets were abusive or problematic, and if so, whether they revealed misogynistic, homophobic or racist abuse, or other types of violence. Each tweet was analysed by multiple people. The volunteers were given a tutorial and definitions and examples of abusive and problematic content, as well as an online forum where they could discuss the tweets with each other and with Amnesty International’s researchers.
Troll Patrol Project
This video has a caption
There’s a degree of racism everywhere, and this is true in Algeria too, but I didn’t let it worry me too much. When the prime minister announced that all migrants living in Algeria needed to be registered, I dutifully went along to the police stations both in Maghnya, where I was working, and in my weekend home, Oran.
However attitudes changed when the current prime minister, Ahmed Ouyahia, came to power in 2017. Oil prices had fallen and the authorities started blaming immigrants for the country’s economic woes. This really encouraged the growth of racist attitudes, and even in Oran black people like me started hiding at home and not going out. I didn’t realise though just how bad things would get.
A showcase of the Amnesty Decoders’ Troll Patrol project interface that ran from March to August 2018.
The volunteers collectively dedicated an incredible 2,500 hours analysing tweets - that’s the equivalent of someone working full-time for 1.5 years. They were a very diverse group of people- aged between 18 to 70 years old and from over 150 countries.
Three experts on violence and abuse against women also categorized a sample of 1,000 tweets, to ensure we were able to assess the quality of the tweets labelled by our digital volunteers.
Using a subset of the Decoders and experts’ categorization of the tweets, Element AI used data science to extrapolate the abuse analysis to the full 14.5 million tweets that mentioned the women journalists and politicians selected for our study. The results we are publishing are based on this.
The tweets in this study dated from January-December 2017, and were collected for analysis in March 2018. Crucially, because we could only download historical Twitter data, our sample did not include tweets that had already been deleted or tweets from accounts that were suspended or disabled during 2017, but only tweets that were still available on the platform in March 2018. Therefore, we can only assume the true scale of abuse was even higher than our results show. Some of the tweets in our sample have been deleted since March 2018.
What did we find?
Our study found that 7.1% of tweets sent to the women in the study were problematic or abusive. This amounts to 1.1 million problematic or abusive mentions of these 778 women across the year, or one every 30 seconds on average. Women of colour were more likely to be impacted - with black women disproportionately targeted with problematic or abusive tweets. The study also reiterated that online abuse against women cuts across the political spectrum and does not pay heed to political party divisions.
All estimates (including percentages here) are subject to margins of error. For details, see methodology note.
Abusive tweets
Abusive tweets violate Twitter’s own rules and include content that promote violence against or threats of people based on their race, ethnicity, national origin, sexual orientation, gender, gender identity, religious affiliation, age, disability, or serious disease. Examples include physical or sexual threats, wishes for the physical harm or death, reference to violent events, behaviour that incites fear or repeated slurs, epithets, racist and sexist tropes, or other content that degrades someone.
Examples of a abusive tweets
Problematic tweets
Problematic tweets contain hurtful or hostile content, especially if repeated to an individual on multiple or occasions, but do not necessarily meet the threshold of abuse. Problematic tweets can reinforce negative or harmful stereotypes against a group of individuals (e.g. negative stereotypes about a race or people who follow a certain religion).
Such tweets may still have the effect of silencing an individual or groups of individuals. It’s important to acknowledge that problematic tweets may qualify as legitimate speech and would not necessarily be subject to removal from the platform. We included problematic tweets because it is important to highlight the breadth and depth of toxicity on Twitter in its various forms and to recognize the cumulative effect that problematic content may have on the ability of women to freely expressing themselves on the platform.
Twitter, itself, has committed to increase the 'health' of public conversations on the platform and the problematic content labelled in our study will be useful for Twitter to consider.
Examples of a problematic tweets
WARNING: The examples below contain explicit and threatening messages that show violence against women.
Black women were disproportionately targeted with online abuse
Our study shows that as a group, women of colour, (black, Asian, Latinx and mixed-race women) were 34% more likely to be mentioned in abusive tweets than white women. Black women were particularly affected, being 84% more likely than white women to be mentioned in abusive tweets. This included all kinds of abuse - sexual and physical threats and misogyny, as well as racial slurs.
To better understand if and how women of colour were targeted as compared to white women on Twitter, we classified our sample of 778 journalists and politicians by race/ ethnic background. To determine race/ethnic background, we used official sources such as ethnic diversity studies but also information in the public domain such as public profiles, articles written by or about the women, Wikipedia pages, etc. It is important to note that this is a crude classification for the purpose of this analysis and is not necessarily a reflection of how each of the 778 women self-identify.
% of mentions
Problematic Abusive Confidence intervals
The percentage ratios are subject to statistical margins of error.
Twitter recently updated their hateful conduct policy recognising that some groups are disproportionately targeted with abuse online - this includes “women of colour, lesbian, gay, bisexual, transgender, queer, intersex, asexual individuals, marginalized and historically underrepresented communities”. We did not replicate this analysis for other aspects of women's identity such as religion, sexual orientation or gender identity. However, our previous research shows that women of colour, religious or ethnic minority women, lesbian, bisexual, transgender or intersex (LBTI) women, and other women with different identities will often experience abuse that targets them in unique or compounded way.
Online abuse against women cuts across the political spectrum
Journalists
Our analysis found that 7% of Twitter mentions of women journalists were problematic (5.8%) or abusive (1.2%) - that is 1 in every 14 mentions. In total, an estimate of 225,766 mentions sent to 454 journalists in our study were found to be abusive or problematic.
Politicians
In terms of percentage, politicians received a similar rate of abuse to journalists - 5.85% of mentions were problematic and 1.27% abusive (7.12% in total). The overall estimated volumes for 324 politicians studied was 867,136 (on average, the politicians got many more mentions than journalists).
Online abuse against women cuts across all political parties and is a bi-partisan issue that does not pay heed to political boundaries. Women politicians and journalists faced similar levels of online abuse and we observed that both liberals and conservatives alike, as well as left and right leaning media organisations, were targeted (source for classification of media bias).
Women working for left leaning parties and media organisations received slightly more problematic and abusive mentions than women working for right leaning causes.
