Facebook moderation is working with a bandageMehdi Habab

Last month, more than a hundred of Facebook’s internal moderation documents were leaked to the Guardian, revealing for the first time the social media giant’s secret policies regarding how it moderates content posted to the site. Covering topics such as bullying, revenge porn, terrorist propaganda and even cannibalism, the extraordinary scope of the ‘Facebook Files’ demonstrates the enormity of the challenges faced by the company in striking the right balance between free expression and the protection of its users. The leaked files raise serious questions as to whether Facebook has the means, or indeed the will, to effectively achieve this balance.

“Videos of extreme animal abuse, for example, must be marked as ‘disturbing’ while stills taken from the same videos do not have to be”

So what do the files actually tell us about Facebook’s approach to moderation? What stands out in particular is the level of detail contained in the documents. On the subject of terrorism, moderators are provided with a handbook detailing the insignia, key leadership figures and area of operation of more than 600 terrorist organisations, all of which they are expected to become familiar with. ‘Sexual activity’ alone is divided into ‘arousal’, ‘handy-work’ (yes, really), ‘mouth work’, ‘penetration’, ‘fetish’ and ‘groping’ – training slides show that content is split into a complex multitude of categories and sub-categories of permitted material. Specific examples, taken from actual user content, illustrate each point.

It's not okay to hate (except in certain cases)Sam Michel

The guidelines show that Facebook is clearly attempting to systematise and standardise the moderation process. On some level, this makes perfect sense – their small army of 4,500 moderators must be given clear, internationally, and cross-culturally applicable guidelines if they are to work effectively and consistently. Yet such an approach, focused on examples rather than principles, has produced surprising and sometimes seemingly arbitrary distinctions. Videos of extreme animal abuse, for example, must be marked as ‘disturbing’ while stills taken from the same videos do not have to be. Suicide threats referring to an event more than five days into the future or using emoticons may be ignored, even though an expressed intention to commit suicide within the next four days would be actionable. ‘Handmade’ art showing nudity and sexual activity is permissible, while digitally made art showing sexual activity is not.

“it seems that there is an inherent conflict of interest at the heart of Facebook’s moderation processes”

Facebook’s moderation documents raise two key issues. The first is operational: Facebook’s current moderation infrastructure is arguably insufficient to deal with the volume of material being generated. Currently, moderators have on average less than 10 seconds to make a decision on a piece of information. Training programmes and psychological support provisions are lacking and high numbers of moderators report experiencing high levels of stress and burnout. Although Facebook is looking into developing AI that could assist the work of moderators, the current plan is to recruit a further 3,000 staff, a move which even their own Head of Policy, Simon Milner, has conceded may not provide a satisfactory solution.

Claire Lilley, Head of Child Safety Online at the NSPCC, has noted that “Facebook are not the arbiter of social norms and expectations”, but this is not strictly accurate. Considering the current state of things, Facebook is already arguably a barometer, if not an actual influencer, of social norms. The critical point that demands public attention and discussion is whether this should be the case. The status quo lacks accountability and transparency – to bastardise the old phrase: who moderates the moderators? Without regulation or at least some kind of accountability mechanism, we risk entrusting Facebook and other tech behemoths with vast power over the content that we are actively (but more often passively) exposed to and consume. One possible solution, proposed by Facebook itself, has been to create a global voting system allowing users to set their own levels of comfort with content, a system of “personal controls and democratic referenda”. This is no better – allowing users to choose the challenging content they are happy to see would gut Facebook of much of its potential benefit as a platform for free speech and debate.

What Facebook should do now is focus on articulating the principles underlying its moderation rather than further refining to absurdity its already incredibly detailed mass of regulations. Plausibly, this could provide a simpler system that could accommodate both the subjective judgement required to make sense of the infinite complexity and indeed stupidity (!) of user-generated content while also mapping out non-arbitrary boundaries regarding what is acceptable content in the societies that we live in. Easily forgotten in a profit-minded modern world, what Facebook really needs to do is to go back to philosophy and start building from there