The Social Impact of AI-Based Content Moderation (Part I.)
Modern social media platforms are not just technological businesses but have also become the basic communication infrastructure over the past decade. With this role, however, comes the ability to shape and constrain user interactions, discourse, and, in some cases, political participation or social engagement. Partly because of the sheer volume of content published, and partly because of their legal obligations, these platforms can no longer rely solely on the moderators to supervise all content. Automated (algorithmic) solutions are therefore becoming increasingly important. In this post, we explore the potential impact of such automated solutions on society.
Freedom of expression
Platforms such as Facebook or Google undeniably have a significant power over public communication. This power manifests itself in the fact that these platforms can decide what content to display, what algorithms to use to sort content, and what is acceptable or unacceptable on a given platform. But in doing so, they also necessarily influence what information is available to society, which actors can make their voices heard in public, and when and to what extent freedom of expression is restricted.
Of course, this kind of influence is not just technical or commercial but also has important political consequences. The power of social media platforms over communication means that these companies can regulate political discourse, facilitate or hinder political movements, and even directly influence electoral outcomes by favoring certain information (i.e. getting it to more users) and marginalizing others.
It is important to note that decisions on such platforms may be based on commercial considerations, which often do not coincide with the public interest or the interests of users. This raises the question of the extent to which these platforms should be subject to stricter regulation, especially if the power they wield is such that they have the capacity to fundamentally change the dynamics of social communication. Before the digital world, only states had this type of power over their own citizens. Today, however, this more effective and comprehensive ability to shape opinion than ever before is held by a handful of large technology companies whose user base extends beyond national borders to virtually the entire modern world.
This phenomenon has of course not escaped the attention of policymakers. Among other things, it led to the introduction of the Digital Markets Act (DMA) and the Digital Services Act (DSA) in the European Union. Perhaps one of the most important definitions of these two regulations was the designation of companies as gatekeepers. The term “gatekeeper” in the context of the regulations refers to large technology companies that have a significant impact on the functioning and competitive conditions in digital markets. As already mentioned, such companies (such as Apple, Google, or Meta) play a central role in determining which services and products, or even information, reach users.
The main aim of the DMA is to ensure fair competition in digital markets, especially for large technology companies, which may indirectly affect freedom of expression. If large platforms have fewer opportunities to exploit their monopoly position, smaller, alternative platforms may be able to enter the market, which could also make public discourse more diverse and democratic. DSA aims to regulate digital services, with a particular focus on the safety and protection of users’ rights in the online space. The irony is that the introduction of stricter regulation also risks technology companies becoming more protective of their own interests and even adopting stricter moderation policies to comply with the legislation. This, in turn, could further restrict the manifestations of free speech on the platforms they operate, as stricter (even automated) moderation could make them even more sensitive to user content, creating a form of online censorship in the interests of the platform they serve.
Of course, freedom of expression is also seriously affected by algorithmic errors. Machine learning algorithms learn from large amounts of data and aim to recognize patterns that can help them decide whether a piece of content is compliant with the rules of a platform. Essentially, the task of such algorithms is to make decisions based on the cases they know (training data) about cases they have never seen before. In such cases, there will inevitably be false positive and false negative decisions.
False positives can be particularly damaging in terms of the impact on freedom of expression. When an algorithm mistakenly removes harmless content, it can restrict users’ right to freedom of expression. These types of errors can be particularly worrying in political or social discourse, where stricter moderation can lead to the silencing of critical opinions. For example, if an algorithm mistakenly removes a post about a political protest, the platform may be contributing to censorship and limiting expression.
On the contrary, false negatives pose a different kind of risk. When the algorithm fails to recognize offending content, it contributes to the free flow of harmful information such as disinformation, hate speech, or extremist, destructive ideologies. This can be a direct threat to users and can seriously distort public opinion. In addition, such mistakes can undermine the credibility of social media platforms and the trust that users place in them.
Opinion bubbles
There are two ways to moderate the content that is presented to users. On the one hand, the moderation process decides whether to keep or delete content. On the other hand, the content that is retained is also ranked, in many cases, in a personalized way (e.g. recommendation systems). The first of these determines which content will be available on the platform in the first place. The second case controls exactly which content a person is likely to encounter among the available options.
The combination of algorithmic content moderation and recommender systems can have a significant impact on social divisions, by facilitating the creation of so-called “echo chambers” or “filter bubbles”. This phenomenon occurs when users mainly encounter content that reinforces their existing views, while essentially not encountering opposing views. The simplest way this can happen is that content recommendation systems on social media platforms tend to present content that matches the user’s known interests and preferences. This content ranking then also interferes with confirmation bias, whereby people tend to seek, prefer, and rely on information that confirms their existing beliefs while ignoring contrary views. Together, the two influence decision-making and opinion-forming, as people often only accept information that supports their own views as true. Of course, false certainty also contributes directly to drawing wrong conclusions and increasing social divisions.
The existence of opinion bubbles and political polarization are closely linked, as the former fosters the latter, i.e. the extremism of political views and the growth of social divisions. As discussed, opinion bubbles are formed when the algorithms of digital platforms recommend content that reinforces existing beliefs, based on users’ past activities, while excluding information that represents opposing views. This then (often falsely) reinforces users’ own opinions and contributes to people’s decreasing exposure to perspectives different from their own. As a result, political discourse becomes increasingly polarized as different political groups become more isolated from each other and less open to hearing or understanding opposing views. In addition, within opinion bubbles, views tend to shift towards the extremes, which also means that the search for compromise is increasingly being phased out and replaced by the propagation of increasingly radical views.
Underlying this, of course, is that divisive (even extreme) content on social media tends to generate more interaction than consolidated content. Interactions also lead to more attention, which in turn makes it paradoxically in the interest of companies seeking to maximize attention time to promote such content. This is because, in the attention economy, service providers aim to maximize the time users spend on their platforms. The easiest way to achieve this is to prioritize content that increases the time spent on the platform. And sharing discussion-starting content is a prime means of doing this.
István ÜVEGES is a researcher in Computer Linguistics at MONTANA Knowledge Management Ltd. and a researcher at the HUN-REN Centre for Social Sciences, Political and Legal Text Mining and Artificial Intelligence Laboratory (poltextLAB). His main interests include practical applications of Automation, Artificial Intelligence (Machine Learning), Legal Language (legalese) studies and the Plain Language Movement.