It was in the course of another project that we initially came across a social media moderator’s perception that ‘bad’ content online keeps ‘coming back’ – that moderation often feels like a dystopian digital version of Whac-a-mole. The grimmest videos or images keep popping back up on another platform, or posted by a different user, or tagged with a different set of words. The job of the moderator was to clean up digital pollution after the fact, but it was a near-impossible task.

Moderators not only see the content users create, post and share, they also know what platforms allow, promote and prevent

So we set out to do a small, self-funded research project in which we would find out if other moderators had the same experiences, and whether they’d reached the same conclusions.

Most of us know that what we see in our own social media feeds isn’t the same as what other people see in theirs, but algorithms and echo chambers make it hard to accurately imagine. Moderators have a unique perspective, because they not only see the content that users create, post and share, they also get first-hand experience of what platforms allow, promote and prevent.

For this project we interviewed five moderators from one video-sharing platform, which we are not naming to protect the respondents’ anonymity. The participants were self-selecting and are not a representative sample, so it’s not possible to extrapolate specific findings. However, we can use the research evidence alongside our work on a range of projects exploring people’s behaviours and experiences online to draw inferences about the wider system. This system – the ‘attention economy’, in which businesses buy, sell and compete for our attention – is not new, but this research provides additional evidence that

1. The motivations and incentives built into the attention economy shape people’s behaviour

In the attention economy, many digital businesses’ financial success relies on advertising revenue. This, in turn is predicated on capturing – and then essentially selling – as much of people’s attention as possible. This means platforms seek to maximise users’ time and engagement.

These drivers shape the way the features and functions of social media platforms are designed. And the features and functions inevitably influence users’ behaviour – incentivising people to create, post and re-share content that might capture other users’ attention. Such content might be entertaining, informative or inspiring, but equally it might be shocking, sexual, violent. The users are – perhaps unknowingly – supporting the platform’s business model.

2. The attention economy has externalities – indirect but large harmful costs borne by people who aren’t in control of that system

In this research we can see that the negative experiences of the people whose job it is to moderate the content are one externality of this economy. Even if we assume that not all moderators feel as unhappy about their experience of moderating as the sample we have interviewed, we understand that at least some moderators are experiencing distress and sometimes mental ill-health.

But it’s not only those who are trying to do the ‘clean up’ operation who may be harmed. Because moderators on most social media platforms are required to check the content only after it is already live, any number of tens, hundreds or thousands of people who view the content before it is moderated may have negative experiences as well. This is another externality of the system.

Then there is the question of what effect the attention economy – in particular the behaviours it incentivises – has on norms and culture more widely. What are the consequences for society if some people’s first instinct when they see something horrific is to reach for their phones to film it and share it? The compulsion to film, share and reshare graphic acts of violence or horrific accidents to get views, likes, shares or followers, to unthinkingly or uncaringly ‘pollute’ the environment with content which seems often to override previously held collective notions of decency and sensitivity, is arguably an externality too.

We hope that by providing evidence that shines a light on these externalities, this research will contribute to debates about whether as a society we are happy with the attention economy as it operates at present and, if not, how we might seek to change it.