OFCOM's register of risks of Part 3 services
Protecting people from illegal harms online
Register of Risks¶
V2.0¶
Published 25 June 2026
Published 25 June 2026
Contents¶
Section¶
Introduction to the causes and impacts of illegal harm online (Register of Risks) .................. 5
1. Terrorism ....................................................................................................................... 33
2. Child Sexual Exploitation and Abuse (CSEA) .................................................................. 53
2A. Grooming ....................................................................................................................... 65
2B. Child Sexual Abuse Material (CSAM) ............................................................................. 84
3. Hate ............................................................................................................................. 104
4. Harassment, stalking, threats and abuse ..................................................................... 122
5. Controlling or coercive behaviour (CCB) ...................................................................... 144
6. Intimate image abuse .................................................................................................. 164
7. Extreme pornography offence ..................................................................................... 186
8. Sexual exploitation of adults ....................................................................................... 197
9. Human trafficking ........................................................................................................ 208
10. Unlawful immigration .................................................................................................. 226
11. Fraud and financial services offences .......................................................................... 235
12. Proceeds of Crime ....................................................................................................... 256
13. Drugs and psychoactive substances ............................................................................ 266
14. Firearms, knives and other weapons ........................................................................... 282
15. Encouraging or assisting suicide (or attempted suicide), and serious self-harm ......... 292
16. Foreign interference offence ....................................................................................... 322
17. Animal cruelty.............................................................................................................. 346
18. Cyberflashing ............................................................................................................... 361
19. Epilepsy trolling offence .............................................................................................. 369
20. False communications ................................................................................................. 377
21. Obscene content showing torture of humans and animals (the s.127(1) offence) ...... 385
22. Threatening communications ...................................................................................... 394
23. Search services ............................................................................................................ 396
24. Governance, systems and processes ........................................................................... 409
Annex¶
A1. Glossary of terms ......................................................................................................... 427
A2. Updating the Register of Risks ..................................................................................... 439
A3. Updating the Risk Profiles ............................................................................................ 451
Introduction to the causes and impacts of illegal harm online (Register of Risks)¶
The Illegal Harms Register of Risks (‘Register of Risks’) is our assessment of the causes and impacts of illegal online harms based on the evidence that we have gathered over the past four years. The Register of Risks presents our full risk assessment of where and how illegal harms manifest online and the characteristics of services that are relevant to the risks of harm. It forms part of our duty under the Act to assess the factors that can cause a risk of harm to individuals on a service. Ensuring service providers undertake a high-quality risk assessment is one of our strategic objectives and the Register of Risks is an important resource for achieving this. It is intended to act as a central resource for service providers when they are conducting their risk assessments, providing a clear understanding of how harms manifest online and how specific characteristics of services and users play a role. The risk factors identified in the Register of Risks also inform the Risk Profiles that sit within the Risk Assessment Guidance, helping service providers identify the areas of greatest likely risk due to particular characteristics in the design, functionality and user base of their services. In their risk assessment, they will be expected to assess the likelihood and impact of those risks. Our assessment focuses on the over 140 priority offences defined in the Act. For ease of navigation and due to similarities in how some kinds of illegal content manifest, we have grouped these into 18 broad kinds of illegal harm. These include illegal harms such as child sexual exploitation and abuse (CSEA), terrorism, suicide and self-harm, fraud, hate speech, weapons and drugs offences, modern slavery and human trafficking, foreign interference and most recently, animal cruelty. We summarise the findings of this assessment below and set out the detailed analysis in the subsequent Register of Risks chapters – one for each kind of illegal harm. We also have four additional chapters, each covering a relevant non-priority offence, such as false communications and ‘epilepsy trolling’. Based on the evidence contained in the illegal harms Register of Risks, we’ve identified risk factors for each kind of harm – the characteristics of online services and users that we believe increase the likelihood of illegal content being encountered, and the associated risk of harm. Offenders often rely on different types of services to commit or facilitate the offences covered by the Act. For instance, both fraudsters and perpetrators of grooming will often contact potential victims on public forums and then seek to move them onto private, sometimes encrypted, messaging services. This means that action to tackle online harms cannot focus exclusively on a small subset of services and cannot be targeted exclusively at the largest services. Rather, it needs to address a broad range of service types including both large services and the many smaller services in scope of the Act. The role of the new online safety regulations is to get services to put in place safeguards which allow users to enjoy the benefits they bring while managing the risks appropriately.
Online harms and the factors which can increase the risk of harm occurring are changing all the time as technology develops. The recent emergence of generative AI provides a particularly clear example of this. As well as bringing important benefits, generative AI creates new risks. Image-generation models, for example, can be used in some cases to create child sexual abuse material (CSAM). Studies have also highlighted the use of generative AI to create ‘deepfakes’ in support of foreign interference campaigns. They have also been used to generate instructions for how to access unlicensed firearms and to create authentic sounding audio or textual messages to deceive victims in cases of fraud. The constant emergence of new risks makes it important that services conduct regular risk assessments. It also makes robust corporate governance particularly important. Where services have good governance arrangements in place with clear accountability for managing risks, they are more likely to detect and appropriately manage emerging risks. In addition to recommending measures to address specific harms, a key focus for us is ensuring service providers conduct robust risk assessments and have appropriate governance arrangements in place. The Register of Risks was originally published 16th December 2024. This updated version was published on 25th June 2026.
About this document¶
This introduction to the Register of Risks is structured as follows: a) Section 1: Introduction to the causes and impacts of illegal harm online (this section): provides a summary of our findings and an introduction to how illegal harm manifests online. We set out some concepts and context useful for interpreting the huge range of issues covered in the Register of Risks itself (see Section 3) b) Section 2: Ofcom’s Register of Risks for illegal harms: i) Methodology for conducting our risk assessment: this chapter provides an overview of the methodology used to conduct our sector-wider risk assessment, and introduces concepts presented in the Register of Risks
one for each kind of illegal harm.
Summary of Findings¶
Over the past three years we have conducted an extensive analysis of the causes and impacts of illegal online harms. As part of our analysis, we reviewed thousands of sources from hundreds of research organisations, academic institutions, online service providers, government, law enforcement and civil society organisations. In the November 2023 Consultation we published an initial draft of this analysis. This drew on research that we have commissioned, a comprehensive review of the existing published evidence and engagement with a wide variety of stakeholders. The 199 responses to the November 2023 consultation contained a large body of additional evidence on and insights into the illegal harms in scope of the Act. Over the past year, we have analysed these responses in detail and conducted follow up research, analysis and stakeholder engagement to deepen our understanding of the harms. Taken together, the work we have done over the past three years shows that illegal online content is widespread and, in many cases, growing in prevalence. For example, Ofcom’s own research found that 87% of adult internet users report having encountered a scam or fraud online and 25% of these people have lost money as a result.1 Almost a fifth of children experienced sexual solicitation from adults they have chatted with online. In a recent report the NSPCC stated more than 7,000 Sexual Communication with a Child offences were recorded by Police in 2023/24, an 89% increase since this offence came into force in 2017/18.2 Given the breadth of the risks online, anyone can experience harm in some capacity, just like anyone can be a victim of crime offline – 68% of UK internet users reported having encountered potentially harmful content online in the past four weeks.3 But in most instances, the risks people face online are not equal, with children and people with certain protected characteristics most likely to be affected and certain kinds of harm being significantly more prevalent among certain groups. Anyone can fall prey to the right kind of online fraud; but it is not a surprise that victims of various forms of online harassment are so often women, mirroring and potentially even amplifying wider challenges in society. Studies have shown that women are five times more likely to be victims of intimate image abuse. 30% of minority ethnic internet users report having encountered ‘hateful, offensive or discriminatory content’, compared to 25% of all internet users.4 Generally, the more protected characteristics or vulnerabilities someone has, the greater the risk of harm they face from priority illegal harms in the Act. Therefore, since the impact of the harms we have looked at can be extremely severe, our work has never been more critical than it is today. Harm is not limited to the online world and profoundly affects people’s lives. Our updated Register of Risks demonstrates that harms, such as online grooming, can cause lifelong negative psychological impacts for victims. Additionally, the harm experienced may not be limited to the individuals who directly encounter risks or illegal content online. It also impacts those who are the victims of the subsequent actions of those people who have encountered illegal content or participated in illegal activity online. In some cases – in relation to terrorism, for example, or the erosion of trust in democratic processes caused by state-sponsored disinformation campaigns – it can be argued that these harms have a wider societal impact. We have seen how all types of services can pose a risk of harm from the priority illegal content addressed by the Act, individually and when used together to facilitate criminal activity. While many
1 Ofcom, 2023. Online Scams & Fraud Research. [accessed 21 November 2024]. 2 NSPCC, 2024. Online grooming crimes against children increase by 89% in six years. [accessed 14 November 2024]. 3 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 28 November 2024]. 4 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 28 November 2024].
service providers have made significant investments in tackling online harms in recent years, our evidence shows these have not yet been sufficient. For instance, encrypted cloud-based file-sharing services are still used by offenders to store and share CSAM; social media services with large numbers of targetable users and with highly effective content recommender systems remain effective places for state-sponsored disinformation campaigns to take place. Our analysis and engagement with stakeholders also shows how the risks to the UK public evolve in step with the online landscape and are arguably more significant now than they ever have been. Our work in the past year shows that the kinds of illegal harm we have looked at occur on services of all types. Services as diverse as social media services, dating services, marketplaces and listings services, search services, user-to-user pornography services, and file-storage and file-sharing services are all used to disseminate some of the types of illegal content and facilitate the offences we have looked at in the Register of Risks. While certain characteristics of a service might make a certain type of harm more likely to occur on or via a particular service, no service in-scope of the Act is entirely immune from risk simply due to the nature of its design, user base or the way the service provider conducts business and runs their organisation. For instance, the size of a service does not dictate the level of risk but influences how and what kinds of illegal harm are more likely to manifest. Bad actors use both large and small services to spread illegal content, though the way in which they use these services is likely to differ. For example, terrorists often use large services to disseminate propaganda to large audiences, but often use small services for more covert activities such as recruitment, planning and fundraising. Although a very wide range of service types pose risks of the priority illegal harms in the Act, there are certain service types that appear to play a particularly prominent role in the spread of priority illegal content. For example, our updated analysis suggests that video-sharing services and dating services pose a particular high risk with regards to hate offences and drugs and psychoactive substances offences respectively. Similarly, certain ‘functionalities’ stand out as posing particular risks because of the prominent role they appear to play in the spread of illegal content and the commission and facilitation of offences: • End-to-end encryption: Offenders often use end-to-end encrypted services to evade detection. For example, end-to-end encryption can enable perpetrators to circulate CSAM, engage in fraud, and spread terrorist content with a reduced risk of detection. • Pseudonymity and anonymity: In most cases where offenders are using online services to engage in illegal activity, hiding their identity is incredibly important as it supports their ability to evade detection. There is also some evidence that pseudonymity (where a person’s identity is hidden from others through the use of aliases) and anonymity can embolden offenders to engage in a number of harmful behaviours with reduced fear of consequences who otherwise might not have. For example, while the evidence is contested, some studies suggest that pseudonymity and anonymity can embolden people to commit hate speech. At the same time, cases of harassment and stalking often involve perpetrators creating multiple fake user profiles to contact individuals against their will and to circumvent blocking and moderation. • Livestreaming: There are many examples of terrorists livestreaming attacks, this can in turn incite further violence. The use of livestreaming remains a persistent feature of far-right lone attackers, many of whom directly reference and copy aspects of previous attacks. Similarly, perpetrators can exploit livestreaming functionality when abusing children online.
• Content recommender systems: Content recommender systems are commonly designed to optimise for user engagement and learn about users’ preferences. Where a user is engaging with harmful content such as hate speech or content which promotes suicide, there is a risk that this might result in ever more of this content being served up to them.
Online harms and the risk factors which cause them are changing all the time as technology develops and society evolves. The recent emergence of generative AI provides a particularly clear example of this. As well as bringing important benefits, generative AI creates new risks across a variety of kinds of illegal harm including CSAM, terrorism, fraud and foreign interference. The functionalities we describe above are not inherently harmful and can have important benefits for users. End-to-end encryption plays an important role in safeguarding privacy online. Pseudonymity and anonymity can allow people to express themselves and engage freely online. In particular, anonymity can be important for historically marginalised groups such as members of the LGBTQ+ community who wish to talk openly about their sexuality or explore gender identity without fear of discrimination or harassment. Recommender systems benefit internet users by helping them find content which is interesting and relevant to them. The constant emergence of new risks makes it vital that services conduct regular risk assessments. It also makes robust corporate governance particularly important. Where services have good governance arrangements in place with clear accountability for managing risks, they are more likely to detect and appropriately manage emerging risks. In addition to recommending measures to address specific harms, a key focus for us as the Online Safety regime comes into force will, therefore, be ensuring that services conduct robust risk assessments and have appropriate governance arrangements in place. As we explain in ‘Our approach to developing Codes measures’, we have designed measures in our Codes of Practice to target high-risk service types and functionalities. The role of the new online safety regulations is not to restrict or prohibit the use of such functionalities, but rather to get service providers to put in place safeguards which allow users to enjoy the benefits they bring while managing the risks appropriately.
Understanding illegal harms online¶
Most of our analysis of the risks of harm is focused on specifics: every chapter of the Register of Risks is focused on one kind of illegal harm, in some cases individual offences, and seeks to present a non-exhaustive set of risk factors that we have identified from the evidence. We recognise that every instance of harm experienced or facilitated online is also unique, and risk is a multi-faceted, often highly complex and, at times, personal issue. We have identified some broad trends that underpin how harm can manifest online that can be seen in many, if not all, instances where illegal activity occurs or is facilitated by online services. Understanding these trends is helpful when interpreting the huge variety of risks and harms explored in the Register of Risks. • Perpetrator behaviour is diverse and adapts to the online environment: In most instances, it is intentional misuse of online services that leads to illegal harm, whether directly or indirectly. As a term “perpetrators” – those who commit offences online – describes a very broad range of people and organisations. For example, perpetrators can be state-sponsored agencies or form part of organised crime groups or networks. In other cases, a perpetrator may be an individual operating alone to commit an offence. The circumstances that lead to the use of online services to commit illegal acts are also multi-faceted and complex. The mitigations service providers put in place must therefore be able to disrupt a huge variety of intentional criminal activity – from using messaging services to advertise illegal knives and drugs for sale to offenders sharing ‘how to’ guides to groom children online – while also preventing other users from encountering illegal content. • Functionalities, features and technologies: Any functionality, feature or technology which facilitates the spread of information and makes it easier for people to interact with one another necessarily facilitates the spread of both good and bad content and enables positive and harmful interactions. The functions that deliver value to users in some contexts can and are misused by perpetrators to achieve their own aims. For instance, predators contact children using the same direct messaging functionality designed for children to keep in touch with their friends and family, hiding their real identity with fake user profiles. In some cases, functionalities simply do what they are designed to do, but with limited regard to the negative consequences resulting from working as intended – a system designed to maximise engagement or provide accurate responses to queries has no reason not use potentially illegal or harmful content to do so unless designed not to. • Services: Online services are where these functionalities and the people who use and misuse them come together. They provide the specific combinations of functionality and access to people and content, along with the systems, processes, governance and even culture, that make them uniquely suited to delivering huge value for users, but also the wide variety of harm. For example, encrypted cloud-based file-sharing services are used by offenders to store and share CSAM; social media services with large numbers of targetable users and with highly effective content recommender systems are effective places for state-sponsored disinformation campaigns to take place. • Victim’s experiences: Given the breadth of illegal harm that manifests online everyone can be at risk of some form of online harm. But in most instances, there are certain people who are more vulnerable, more susceptible, and more likely to experience severe harm as a result of their experience. Anyone can fall prey to the right kind of
online fraud; but the victims of various forms of online harassment are disproportionately women.
Every instance of online harm requires some combination of these components. By better understanding the specific instances in which they come together to cause harm – as we have set out in our Register of Risks – we can better minimise the risks and prevent harm from occurring.
Perpetrator behaviours¶
As a term “perpetrators” – those who commit offences online – describes a very broad range of people and organisations, acting alone or as part of collective efforts. For example, perpetrators can be state-sponsored agencies or form part of organised crime groups or networks. In other cases, the perpetrator may be an individual operating alone to commit the offence. There are also cases of preparators supporting one another to provide advice or share content with each other among an informal network of offenders. For example, child sexual abuse material (CSAM) is spread through offender-to-offender networks, as well as perpetrators sharing advice and tips with each other in user groups. Service providers should be mindful that perpetrators also ‘service-hop’, moving between different services in the course of carrying out certain offences. Perpetrators may meet their victims and survivors, or other perpetrators, on one service and then move their interactions to another service. Moving between services is often driven by the unique benefits each service affords to this activity, based on the characteristics of the service. For example, in a case of grooming, a perpetrator may choose to identify a child and initiate contact with them on a social media service that is well-suited to finding and initiating contact with other (child) users, and then move their communication to a private, likely encrypted messaging service where less moderation is expected. The journey of an offence, and the point at which a service is most likely to be used by a potential perpetrator will be relevant to a service provider’s risk assessment. As the accuracy and consistency of moderation efforts continue to improve and other measures restrict the opportunities afforded to perpetrators, particularly on the largest most commonly used services, we expect to see more of this activity displaced to online spaces perpetrators perceive to be more privacy-preserving. The channel of communication used to share content and interact with other users is particularly important and perpetrators use both open and closed channels to commit offences. Open channels which allow for more visible one-to-many communication, such as posting on social media groups, are more likely to be used when perpetrators want to maximise the number of people they are disseminating content to, or want to fish for potential victims in the widest possible pool.5 Conversely, perpetrators are more likely to use closed channels, such as end-to-end encrypted messaging services, for communication they do not want to be visible to others or to the service providers. This might be discussing illegal activity or planning offences covertly or having private interactions with victims and survivors.6
5 This could be the case for fraud, hateful content or promotion of suicide content, for example. 6 For example, perpetrators often message children on private messaging services in grooming offences. Similarly, perpetrators have been known to share CSAM with one another on private messaging services.
Service functionalities, features and technologies¶
Any functionalities which facilitate communication or dissemination of content create opportunities and risks¶
The functionalities of online services, in general, are not inherently positive nor negative. They facilitate communication at scale and reduce friction in user-to-user interactions, making it possible to disseminate both positive and harmful content. For example, users can engage with one another through direct messaging and livestreaming, develop relationships and reduce social isolation. However, these same functionalities can also enable the sharing of illegal material such as livestreams of terrorist atrocities or messages sent with the intent of grooming children. Many functionalities are common across a wide range of services, and therefore a very large number of services can potentially pose some risks of harm to individuals. Whether functionalities ultimately create opportunities for positive engagement or lead to risks of harm will depend on the service’s governance, systems and processes put in place to mitigate risk.
Generative artificial intelligence (GenAI)¶
GenAI delivers a range of benefits. For example, it powers a range of features and functionalities online, from summarising search results to creating avatars or new gaming environments. The underlying models and tools also can be utilised by service providers to mitigate risk online, improving the efficiency, accuracy and scale of tools that improve user safety. However, there is emerging evidence that GenAI technologies can be used to facilitate or commit harm on user-to-user (U2U) and search services, although given the rapid developments in the technology, it can be difficult to quantify the risk of harm to individuals and new evidence of risk of harm is likely to emerge in the future. In particular, AI-generated child sexual abuse material (CSAM) has been identified as a significant and growing problem7, with the National Crime Agency (NCA) warning of the ease and availability of AI-generated CSAM contributing to a “normalisation of offending behaviour” and creating “a more permissive environment for perpetrators” which will put children at increased risk.8 Reports of deepfake intimate image abuse in the past year have also increased significantly9, alongside a dramatic increase in the volume of services that ‘nudify’ targets, enabling users to create non-consensual intimate images of anyone they have images of. The use of GenAI to generate realistic content intended to deceive, and to do this with ease or at scale is also particularly important. Deepfake content is being used increasingly in cases of fraud, with highly targeted deepfakes used to defraud individuals or designed to deceive a mass audience of thousands or even millions.10 UK Finance has highlighted specific risks GenAI technologies pose in regard to fraud, including image generation to produce images of real or fictional people for the purpose of fraud; deepfake audio which could be used to impersonate a victim’s contact to persuade
7 See the Register of Risks section Child Sexual Abuse and Exploitation for further information 8 National Crime Agency, 2024. Technological Tipping Point Reached in Fight Against Child Sexual Abuse. [accessed 14 October 2024]
9 In 2023, more deepfake abuse videos were posted online than in every previous year combined 10 For example, a well-known fake advert featuring a likeness of Martin Lewis was shared on Facebook, in which he appeared to be asking users to sign up for a non-existent Elon Musk investment. Deepfakes can also be used in romance scams, with fraudsters using GenAI and related tools to create inauthentic profiles with images of non-existent people. More elaborate romance scams have seen fraudsters take part in live deepfake video calls with their victims. Sources: BBC, 2023. Martin Lewis felt 'sick' seeing deepfake scam ad on Facebook. [accessed 14 October 2024]; Burgess, M. 2024. The Real-Time Deepfake Romance Scams Have Arrived, Wired, 14 April. [accessed 14 October 2024]
them to make a payment, or potentially to pass voice identity verification; and AI text generation to make it easier to produce fraudulent messages at greater speed and volume.11 GenAI technology can be used in foreign interference campaigns. Across the world, there has been evidence of the use of AI-generated audio, image, video and text-based content, by both state-linked and non-state actors, to influence elections and public opinion more broadly.12 This can include creating false or misleading videos of state figures. Recent reports also suggest that GenAI chatbots13 have been shown to offer advice that promotes eating disorders14 and encourages self-harm.15 There are also reports of early GenAI chatbots generating instructions for how to access unlicensed firearms and how to make explosive materials as well as dangerous chemicals16 which could be used to cause harm.17 The Online Safety Act is technology neutral and AI-generated content which is illegal and shared on a U2U service or present in search results needs to be treated the same way as all other illegal content.18
Recommender systems can deliver benefits but can also increase the risks of harm¶
Many services use recommender systems, which deliver a range of benefits to users. They personalise each user’s experience by helping them find content they are likely to want to engage with and should be considered in the context of each specific service, the kind of content accessible via that service and the way the recommender system is designed. However, they can also increase risk. Our evidence has found that two types of recommender systems could potentially increase risk: content recommender systems and network recommender systems.
Content recommenders¶
Content recommenders are algorithmic systems which determine the relative ranking of an identified pool of content, including user-generated content from multiple users, on content feeds. These systems may amplify risks of harm in a number of ways. Firstly, they can push users into ‘filter bubbles’, where users will only see content similar to other content they engage with and there are limited, or no, opportunities to be exposed to opposing views. In regard to illegal harms, risk stems from the chance that once a user has been exposed to illegal content or engaged with it, they will
11 UK Finance, 2024. The impact of AI in financial services: opportunities, risks and policy considerations. [accessed 18 November 2024]. 12 See the Register of Risks chapter Foreign interference for a detailed discussion of the implications of GenAI in this area. 13 An AI chatbot is an automated software program that uses artificial intelligence and natural language processing to simulate a conversation. 14 The US-based National Eating Disorder Association had taken down its chatbot, “Tessa”, after reports that it had been providing inappropriate content, including advice on how to lose weight. Source: Aratani, L, 2023. US eating disorder helpline takes down AI chatbot over harmful advice, The Guardian, 31 May. [accessed 5 October 2023]. 15 During OpenAI’s ’red-team‘ exercises, a term describing the systematic use of adversarial testing methods to probe and address the risk of language models producing harmful outputs, researchers found that early versions of the company’s new GPT-4 generative AI model was capable of generating advice or encouragement for self-harm behaviours, for example self-mutilation. Source: OpenAI, 2023. gpt-4-system-card. [accessed 5 October 2023].
16 Rose, J, 2022. OpenAI’s New Chatbot Will Tell You How to Shoplift And Make Explosives, Vice, 1 December. [accessed 5 October 2023]; OpenAI, 2023. gpt-4-system-card. [accessed 5 October 2023]. 17 Given the limited evidence of this emerging technology, we have not included some of the examples as part of the detailed risk analysis presented in the Register. We will continue to monitor the landscape with the expectation that more evidence showing a risk of harm associated with GenAI will emerge. 18 For more information on GenAI in the Online Safety Act (2023) see Ofcom’s Open letter to UK online service providers regarding Generative AI and chatbots
continue to be pushed more and more similar content.19 Users may also be pushed into ‘rabbit holes’. This describes a situation where the user is encouraged (by algorithms) to consume more and more content on the same theme, and which can become more extreme or harmful over time, potentially leading to illegal content.20 Potential perpetrators can also ‘game’ recommender systems, to cause an outcome favourable to them as a result of abusing the algorithm by, for example, continuously posting or pushing extreme content.21 22 23 By design, content recommender systems also serve content to users they have not actively chosen to see, increasing the risk that any one individual encounters content that is harmful to them. The relationship between specific kinds of illegal or otherwise harmful content and recommender systems is set out in the harms-specific chapters in this volume. For several kinds of illegal harm, content recommender systems emerge as an important risk factor in how relevant content spreads and is accessed online. Recommender systems are also often an integral part of a service provider’s business model as they form part of a strategy for maximising engagement. Through lengthening the time spent or level of active engagement with content, service providers can increase revenue generated through advertising, for example, by serving more, or more targeted, adverts to users. The risks associated with advertising-based business models originate primarily from recommender system design. See ‘A service’s business model and commercial profile can increase risk’ further in this chapter for more detail.
Product recommenders¶
Product recommenders are technically distinct from content recommender systems, but the two technologies are not legally distinguishable due to the OSA definition of ‘content’ being inclusive of product listings. We distinguish between ‘product’ and ‘content’ recommender systems for two reasons. First, there is a lack of evidence showing product recommender systems contributing to the dissemination of illegal content. Second, they are a distinct technology which would require a separate policy approach from content recommender systems. We also do not have evidence suggesting product recommender systems directly recommend user-generated attributes of product listings, or affect their visibility, unlike the way relevant content recommender systems do. Available research leads us to believe that although product listings may include user-generated attributes (for example, images, text, and videos), users primarily interact with (click, view, purchase) non-user-generated attributes such as product categories, price range, and brands. The evidence we have suggests product recommender systems use algorithms to
19 The evidence available suggests this could be a risk for CSAM, hateful content and suicide and self-harm. 20 Ofcom-commissioned research has also shown that design choices can influence the extent to which users are led on ‘pathways’ from benign to increasingly harmful content (also known as ‘rabbit holes’). Studies have shown that these effects can increase user exposure to a number of harmful content types, including self-harm content, eating disorder content and extreme. However, design choices are not the only factor shaping user exposure to illegal content. Some researchers, for example, have argued that rabbit-hole effects occur more frequently in cases where users are already inclined to seek out harmful content.
21 Gaming of the algorithms can be achieved by using keyword and tagging features or by coordinating a large number of queries or posts to inflate the popularity of a query or post. Because recommender systems can deliver personalised content in an engaging manner, they can be an attractive target for extremist movements and might also be used to draw users towards more niche, ephemeral, or anonymised social media services which are harder to moderate. 22 Design choices can also determine the likelihood of recommender systems being gamed by bad actors. This can happen if the design of a system is too simplistic (e.g. with only a small number of information signals feeding into its ranking decisions) or if granular details of the design are made publicly available. 23 Evidence suggests that this is relevant for the foreign interference offence, and harassment, stalking and threat offences.
analyse only user interactions with the non-user-generated attributes of content to make product recommendations.24
Network recommenders¶
Network recommenders are a type of recommender system that suggests users or groups of users to connect with. Network recommenders may consider a variety of user interactions, mutual connections, and group memberships to determine which network recommendations might be relevant and useful. These systems may amplify risks of harm by helping offenders to find each other by connecting like-minded individuals, or by helping offenders to find potential victims. This is particularly relevant in grooming, as offenders who attempt to connect with multiple child users can then be recommended to other child users.
Online services¶
A range of services can attract perpetrators¶
Our analysis of U2U services found that some service types are used to facilitate and commit a wide range of offences. Social media services and private messaging services, in particular, were found to pose risks of several different kinds of illegal harm, including fraud, terrorism, CSAM and the foreign interference offence. This may be due to their popularity and the functionalities that are typically found on them. Other service types can be used in more targeted ways to facilitate and commit specific offences. For instance, online marketplaces and listings services can be used by individuals to sell illegal goods or to sexually exploit adults; discussion forums and chat room services can act as spaces where suicide and self-harm is assisted or encouraged; users on online dating sites are subjected to cyberflashing; and hateful content can be shared and encouraged on online gaming services. We note that the impact of any risk factor will depend on the combination of risk factors present on the service and how they interact. For example, a social media service offering both livestreaming and screen capture may increase the risk of CSAM on a service. Further, even when a risk factor is relevant to a number of kinds of illegal harms, the way in which it can increase risk can vary between these kinds.25 A service’s risk assessment should consider all risk factors in the round and have a good understanding of how they may affect different kinds of illegal harms, the context in which they are present, and how they interact (for further information on a service’s risk assessment requirements, see the Risk Assessment Guidance and Risk Profiles).
Services with large and small user bases can increase the risks of harm to individuals¶
Services with large and small user bases pose risks to individuals, but often for different reasons. Large services can pose a particular risk of harm because harmful content or conduct on them can reach a large volume of people and because they sometimes attract perpetrators looking to target large volumes of users. Smaller services can pose a particular risk of harm because they may be more focused on particular interests or topics and can therefore be exploited by perpetrators
24 Raza, S., Rahman, M., Kamawal, S., Toroghi, A., Raval, A., Navah, F. and Kazemeini, A. 2024. A Comprehensive Review of Recommender Systems: Transitioning from Theory to Practice. ArXiv:2407.13699v1 [cs.IR]. [accessed 12 November 2024].; Salumke, T. and Nichite, U. 2022. Recommender Systems in E-commerce. ArXiv: 2212.13910. [accessed 12 November 2024]. 25 For example, user groups can be used by perpetrators to share CSAM or advice/tips on illegal practices OR user groups can be used by perpetrators to identify and target individuals for fraudulent activities.
looking for specific communities to target. Smaller services may also have fewer resources available to moderate content, and therefore offer more protection to a perpetrator. For instance, a terrorist organisation may target a service with a large user base to propagate its message and increase the virality of its illegal content. The same terrorist organisation may use a smaller service to store illegal content or organise an attack due to lack of content moderation that may exist (see the Terrorism chapter for further information).
A service provider’s business model and commercial profile can increase risk¶
In this section we provide our consideration of how a service provider’s business model (revenue model and growth strategy) and commercial profile can give rise to an increased risk of illegal harms on services. In assessing the risk of harm, we are required to consider the role of the “business model” 26 of a service. Within this, we consider there are three components worth exploring: a) Revenue model, i.e. how a service provider generates income or revenue (for instance, through advertising or subscriptions). b) Growth strategy, i.e. how a service provider plans to expand its business (for instance, through increasing revenue and number of users). c) Commercial profile, i.e. the size of the service provider in terms of capacity, the stage of service maturity and rate of growth in relation to users or revenue. In and of themselves, these characteristics of a service alone do not necessarily determine the risk of harm, but they can contribute to environments on services in which a variety of illegal harms are more likely to occur. Therefore, the considerations and conclusions outlined here are relevant to every kind of illegal harm explored in the Register of Risks. Where we have further evidence that links business model characteristics very specifically with the kind of illegal harm, content or offences being described we have added it to the relevant chapter of the Register of Risks.
Revenue model¶
Revenue models, which ultimately dictate how online services will generate profit, can create financial incentives that – intentionally or unintentionally – lead to business decisions which prioritise revenue and profit over user safety, exposing users to an increased risk of harms. Service providers that generate revenue in proportion to the number of service users or volume of user engagement – such as through an advertising revenue model27 or subscription revenue model28 – can be incentivised to design systems and features that influence user experience in a way that maximises time spent on service and engagement with content. A system that prioritises engagement will prioritise serving users a larger volume of content or content that generates more engagement. If potentially illegal or harmful content exists on a service
26 As part of our risk assessment duty, we must identify characteristics that are relevant to these risks of harm and assess their impact. This entails considering other aspects of a service, beyond the content presented, such as the business model. We have considered revenue models, growth strategy and commercial profile as parts of business models and the risks they pose to individuals in the United Kingdom, including how user-to-user services can be used to commit or facilitate priority offences.
27 Service providers for which advertising is a key income stream are incentivised to report to advertisers a high user base and high user time spent, as these are key to attracting advertisers to the service. Therefore, service providers which rely on advertising revenue models have a financial incentive to promote content that drives user engagement. 28 Subscription revenue models generate revenue in proportion to the number of paying subscribers and can create financial incentives to promote engaging content that helps attract more paying subscribers and minimise user churn.
then, unless prevented, automated systems can use this to fulfil these engagement-based goals. In this scenario, a focus on quantity of content and engagement increases the risk that a user will encounter harmful content at some point within the total volume of content they are served; that they will be served further harmful content should they engage with it once encountered; and that they will be able to find harmful content if they are searching for it. Where users themselves are also rewarded – for example, monetarily or in terms of status – as a result of the engagement their content generates, it can encourage the creation and sharing of content designed primarily with engagement as the goal. This can fuel this cycle and, where harmful or borderline content correlates with high engagement, as it has been shown to, further increases the risk of other users encountering it. We have discussed the role of specific features and functionalities that play a role in this above, in the sub-section ‘Service functionalities, features and technologies’. As well as advertising- and subscription-based revenue models, other forms of funding are possible. Some online service providers are funded partly or wholly by donations. These may operate commercially or on a not-for-profit basis. Service providers that are funded by donations may still be incentivised to maximise user numbers and/or engagement to drive the number and size of donations, which may not necessarily incentivise services to prioritise user safety. Where a service provider is reliant on large donors, it may be influenced by those donors’ priorities, which may or may not align with user safety. In the same way in which commercial incentives may favour service design choices that increase the risk of exposure of illegal content to users, these incentives may not sufficiently support the development of systems and processes that better protect users, because investing in such measures may lead to a reduction in revenue (or profitability). One example is content moderation. It may be resource-intensive for services to accurately detect illegal or harmful content and to distinguish it from other kinds of content. Service providers may not therefore have sufficient incentive to moderate content, especially if the risk of over-blocking content (e.g. where legal content is unnecessarily taken down) could reduce user engagement or numbers, and therefore revenue. In addition, specific features or functionalities that relate directly to how a service generates revenue can increase the risk that users encounter content that is harmful to them. One example is the ability for users to pay service providers for greater prominence of their user-generated content (e.g. boosting posts). There is a risk that bad actors could abuse this functionality by paying for harmful content (including potentially illegal content) to be boosted. Being promoted, such content reaches a wider audience, so could more easily be encountered by users and pose risks to them, if harmful. On the other hand, the reputational risk of exposing users to illegal content could negatively affect a service’s revenue in the long run, potentially giving rise to some countervailing incentives. Some users may unsubscribe from, or disengage with, services where they encounter illegal content, and business customers (for example, advertisers in advertising models29) or the wider industry (for example, payment providers or investors30) may put commercial pressure on providers to clamp
29 For example, it was reported that recent changes to Twitter’s content policies have led to a surge in harmful content on the site, and in turn, a drop in advertising revenue. Source: New York Times, 2023. 30 Investors are one of the actors who may consider the risk of online harms and potentially influence the approach of services they may invest in. To help inform our understanding of risks, and how investors may influence the risk of online harms, we commissioned a report, ‘Investors Attitudes to Online Harms – Risks, Opportunities and Emerging Trends’. This report was published alongside the Illegal Harms Consultation. Investors Attitudes to Online Harms - Risks, Opportunities, and Emerging Trends.
down on such content. This may sustain some incentives to have effective measures in place to protect users from harmful content. However, to date, market incentives do not appear to have been strong enough to lead the entire online sector to put in place safety systems and processes to mitigate risk and harm on the scale required. The link between revenue models and the risk of harm may be complex, and the extent of any trade-off between financial optimisation and user safety is likely to vary depending on the specific circumstances of each service.
Growth strategy¶
Growth strategies can also be associated with incentives that are in tension with user safety. Service providers may be incentivised to prioritise the use of their limited financial resources for activities and strategies aimed at growing their business (for example, marketing campaigns, research and development (R&D) activities, acquiring new assets and technologies) rather than for the development or improvement of systems and processes that protect its users from harms. This is true especially if such systems and processes could negatively affect their growth. For instance, services whose growth strategy is aimed at increasing the user base can have a disincentive to moderate content that is harmful to UK citizens if it attracts a large number of new users quickly.
Commercial profile¶
The commercial profile of an organisation may influence the sophistication and relative importance of the risk management processes for a service. This, in turn, may affect the risks faced by users if it translates into a reduced capacity to protect users from encountering illegal content. For example, services that are low-capacity31, at an early-stage32 or have a fast-growing user base may face an increased risk of harm. For instance, all else being equal: • Low capacity and early-stage services are less likely to have technical skills and financial resources to introduce effective risk management, compared to more mainstream services. For instance, they may have insufficient resources to adopt technically advanced automated content moderation processes (for example, automated content classifiers), or to employ a large number of paid moderators, and may rely significantly on community moderators instead. In addition, they are likely to seek growth, which may affect their incentives to have effective risk management in place. • Service providers with a fast-growing user base may face difficulties in effectively moderating content, given the increased scale and sophistication of the moderation technologies and processes required to keep track of the user base (since the sources of risk, and kinds of harms on the service, can change quickly as the user base develops). On the other hand, businesses with a more mature profile are likely to have larger user bases and can hence be targeted by bad actors looking to reach large populations of users with illegal content. Such services can therefore present high risks, even when they have significant resources devoted to risk management, unless appropriate systems and processes are in place to protect users from exposure to risk of harms. As with other factors discussed in this section, the link between commercial profiles and risk of harm may be complex, and the extent of any trade-off between financial optimisation and user safety may vary depending on the specific circumstances of a service. For example, it is possible for an early-stage service to have access to substantial resources, depending on its funding context.
31 Services with a small number of employees and/or limited revenue. 32 A service in the initial phases of its lifecycle (for example, start-up and early growth stages).
We consider business models and commercial profiles a ‘general risk factor’ – relevant to all services in-scope of the Act – recognising their importance, but also acknowledging that they may affect risk of harm in an indirect way. The financial incentives and commercial context associated with a service can influence the approach to governance, service design (for example, functionalities) and systems and processes (for example, content moderation). Our Codes focus primarily on these aspects – governance, systems and processes, service design and functionalities – which can affect risk of harm more directly.
Individuals at risk of harm¶
Personal characteristics¶
Experiences of harm are incredibly personal, as is the combination and frequency of risks of illegal harm any one person encounters online. As has been noted, personal characteristics can play an important role in the extent to which people, or groups of people with shared characteristics, experience risk and harm. One of the ‘characteristics’ referred to in the Online Safety Act is ‘user base’. In our assessment of the causes and impacts of illegal harm online we have explored ‘user base’ in terms of both size and composition. As part of this, where possible we have considered various demographic and other personal characteristics of users and how this relates to the risk of harm. Where the evidence allows, we highlight personal characteristics that appear relevant to each of the kinds of illegal harm explored in the Register of Risks. Given the huge variety of personal characteristics and circumstances that have an influence on how someone experiences risk and harm online we have focused on the personal characteristics that are most often referred to in the evidence base or which are inherently related to the kind of harm or offence being explored (for example, gender differences in experience of intimate image abuse).
Vulnerability¶
People can be considered vulnerable to online risk and harm for a huge range of reasons. People with specific personal characteristics as described above may be considered more vulnerable than users who don’t have those characteristics. But someone’s personal circumstances at any one time can also play a huge role in the extent to which they are exposed to risks and an increased risk of harm as a result. Where relevant in each Register of Risks chapter, we have drawn out specific issues around potentially vulnerable users from the evidence, including considerations such as user’s mental health or digital literacy.
Intersectionality¶
The coming together of an individual’s personal characteristics may make them more vulnerable to risk and harm. For instance, a black Muslim woman may be more susceptible to harassment and hate due to the triple discrimination they face as a result of their racial, religious, and gender characteristics. Therefore, we consider intersectionality where relevant in our risk assessments to aim to understand people’s experiences and risk of harm.
Media literacy’s influence on risks of harm¶
Media literacy33, the ability to use, understand and create media and communications in a variety of ways34 can both contribute to and limit the risk of harm. We broadly consider users with a strong knowledge of services and online systems, the confidence to use them adeptly, and those with a good level of critical understanding of online media to have high levels of media literacy. Those with lower levels of media literacy may struggle to navigate the online space, tend not to have good critical understanding online and find it hard to comprehend online services. A low level of media literacy may make users more vulnerable to some forms of online harm. This could be due to a lack of awareness that means they may not recognise the harm being perpetrated until it is too late; or due to a lack of knowledge about how to raise any concerns about what is happening. Nevertheless, this does not mean or imply that the victim of harm is at fault. Some of the illegal offences in the Act rely on a deliberate and sophisticated use of services to avoid detection. Therefore, some chapters note high levels of media literacy as a relevant risk factor for services with functionalities that may be exploited for harmful purposes. But high levels of media literacy can also be an empowering quality as it enables users to better avoid certain harms.
Ofcom’s Register of Risks for illegal content¶
This section explains how we have conducted the analysis for our sector-wide risk assessment, the findings of which are presented in the Illegal Harms’ Register of Risks. The information presented below is to help interested parties understand how we conducted our analysis, and the considerations involved in assessing the risks of harm arising from illegal content and by the use of U2U services for the commission or facilitation of priority offences. This section covers different aspects of our approach to conducting this risk assessment. It is structured as follows: a) Aims and scope: including the definition of harms and kinds of illegal harms considered b) Methodology: including risk factors considered c) Evidence: including considerations regarding our evidence base
Aims and scope¶
The Online Safety Act (the Act) requires Ofcom to carry out sector-wide risk assessments to identify and assess the risk of physical and psychological harm to individuals in the UK presented by regulated user-to-user (U2U) and search services, and to identify characteristics relevant to such risks of harm.35 We must publish the findings of our risk assessments in a ‘Register of Risks’, and then prepare ‘Risk Profiles’. The Risk Profiles are a list of characteristics of online services (which we refer to as risk factors), such as user base size, functionalities, and business model, that are likely to increase risk and indicate
33 Ofcom is mandated to promote media literacy. Section 11 of the Communications Act 2003 [accessed 28 June 2023]. 34 Ofcom, 2023. Making Sense of Media Homepage. [accessed 14 October 2024]. 35 ‘Risks of harm’ refers to the harm to individuals presented by (a) content on U2U or search services that may amount to the offences listed in the Act, and (b) the use of U2U services for the commission and/or facilitation of these offences (collectively, the ‘risks of harm’). ‘Harm’ means physical or psychological harm; we discuss physical or psychological harm as part of our assessment of the risks of harm.
which kinds of illegal harm may be more likely to occur. U2U and Search service providers are required to take account of the Risk Profiles when they carry out their illegal content risk assessment duties in the Act. Ofcom must keep both the Register of Risks and Risk Profiles up to date. We will monitor harms and regulated services trends and will revise our Register of Risks as appropriate. In future we may expand the scope of our risk assessment if necessary. For example, as new technologies develop, and risks to online safety emerge due to the rapid innovation of the sector. This may include technologies such as immersive online virtual worlds, augmented realities, and generative artificial intelligence (‘generative AI’). We will update the Register of Risks over time as new risks, risk factors, harm and evidence that underpins our understanding of these things emerges. We will monitor future case law and associated precedents and consider updating our guidance as appropriate.
‘Harm’¶
In the Illegal Harms Register of Risks, we consider ‘harm’ according to how it is defined in the Act: Harm means physical or psychological harm.36 As set out in the Act, harm can arise from isolated incidents, or from cumulative incidents where illegal content is repeatedly encountered by an individual, whether this is the same kind of illegal content or multiple different kinds.37 Harm can include circumstances of indirect harm, in which a group or individual are harmed, or the likelihood of harm is increased, as a consequence of another person encountering illegal content, which then affects their behaviours towards others.38
Wider societal harm¶
As well as exploring the harm caused to individuals, where possible in the Register of Risks we also reference the concept of wider societal harm – where there are negative collective impacts experienced by the UK public at large, associated with the specific illegal acts we cover. We have not explored the societal harm caused by crime in its totality in the Register of Risks. But have provided information related to the scale of some kinds of illegal harm, and discussed instances of the potential societal harm where it is less inherently obvious. For example, where the prevalence and accessibility of (illegal) non-consensual intimate imagery online can normalise intimate image abuse39; or where society as a whole might be impacted by a foreign state seeking to manipulate how UK citizens participates in an electoral event.40
36 Section 234(2) of the Act. 37 See section 234(4) of the Act, which states: “References to harm presented by content, and any other references to harm in relation to content, include references to cumulative harm arising or that may arise in the following circumstances— (a) where content, or content of a particular kind, is repeatedly encountered by an individual (including, but not limited to, where content, or a kind of content, is sent to an individual by one user or by different users or encountered as a result of algorithms used by, or functionalities of, a service); (b) where content of a particular kind is encountered by an individual in combination with content of a different kind (including, but not limited to, where a kind of content is sent to an individual by one user or by different users or encountered as a result of algorithms used by, or functionalities of, a service).”
38 As set out in Section 234(5) of the Act. 39 See the Intimate Image Abuse section. 40 See the Foreign Interference Offence section.
Kinds of illegal harm considered¶
The relevant offences considered in our risk assessment are: • Priority offences or priority illegal content, which include terrorism offences, offences related to CSEA and other priority offences. These are detailed in the Risk Assessment Guidance and Risk Profiles. So-called ‘inchoate offences’41 are also treated as priority offences. • Relevant non-priority offences42 including the Communications offences (Part 10): false communications offence, threatening communications offence, and offences of sending or showing flashing images electronically (‘epilepsy trolling’). To make our assessment as accessible as possible, we have grouped these offences in the Register of Risks into kinds of illegal harms. This helps our analysis bring out risks that are similar in nature across the group of offences. However, within each grouping we sometimes refer to individual offences where appropriate, for example where a particular observation or evidence is relevant only to specific offences.
41 As explained in Overview of illegal harms, ‘inchoate offences’ include assisting someone else to commit a priority offence, encouraging someone else to commit a priority offence, attempting to commit a priority offence or conspiring to commit a priority offence. 42 Referred to in the Act as ‘other offences’, they are all offences under UK law that are not priority offences, where (a) the victim or intended victim of the offence is an individual (or individuals); (b) the offence is created as a result of the Act, another Act, an order of Council or other relevant instruments; (c) the offence does not concern the infringement of intellectual property rights, the safety or quality of goods, or the performance of a service by a person not qualified to perform it; and (d) the offence is not an offence under the Consumer Protection from Unfair Trading Regulations 2008.
22. Non-priority offence - False communications 23. Non-priority offence - Obscene content showing torture of humans and animals (the s.127(1) offence) 24. Non-priority offence - Threatening communications
Relationship with Protection of Children¶
Our Illegal Harms Register of Risks focuses on offences set out in the Online Safety Act. Content that amounts to one or more of these offences is ‘illegal content’, and a full breakdown of what content could amount to each offence can be found in the Illegal Content Judgements Guidance (ICJG). Both adults and children are at risk of harm from illegal content. As well as this illegal content, the Act also defines different kinds of content that is not illegal but is harmful to children. Our Children’s Register of Risks43 is focused only on this content, which is split into three categories: Primary priority content; Priority content; and ‘non-designated content’ which is neither of the previous two, but which still presents a material risk of significant harm to an appreciable number of children in the United Kingdom. There are some similarities between kinds of illegal content and content that is harmful to children. For example, content that amounts to the offence of encouraging or assisting suicide or attempted suicide is illegal. Meanwhile, content that encourages, promotes, or provides instructions for suicide is classed as content harmful to children, regardless of whether it meets the threshold for being classified as illegal. Because we cannot always definitively draw the line between illegal content and content that is harmful to children – such as within our Register of Risks chapters on encouraging or assisting suicide, encouraging or assisting serious self-harm, hate, harassment, stalking threats and abuse – we have sometimes drawn on evidence that could be describing both kinds of content as we feel it still plays an important role in demonstrating how offences manifest online and the risks of harm. Similarly, the Children’s Register of Risks draws on some evidence that could be referring to illegal content. Where relevant, we have highlighted in the Illegal Harms Register of Risks chapters where the content described by evidence may be broader than the strict definitions of illegal content.
43 The Children’s Register of Risks was published in our April 2025 Protection of Children Statement.
Methodology¶
Understanding service characteristics as risk factors¶
The Act requires Ofcom to take into account how the characteristics of a service may give rise to risk. The Act defines ‘characteristics’ broadly as including a service’s functionalities, user base, business model, governance and other systems and processes. We consider these characteristics both individually and, where relevant, in combination. We explain here how we have analysed characteristic risks to help individuals navigate the Register of Risks and Risk Profiles. These characteristics form the basis of the analysis within our Register of Risks and the Risk Profiles. Most of the characteristics referenced in the Act are not specifically defined. We recognise that given the diversity and range of services in scope of the regime, many services are likely to define some of these concepts differently. We have set out the definitions we have used to conduct our sector-wide risk assessment in the Glossary (Annex). Where possible, we have also used these terms consistently across the other regulatory documents. The list of characteristics in the Act is not exhaustive, so it is open to Ofcom to identify other relevant characteristics. We consider our evidence justified including three additional service characteristics that can give rise to risk: service type, recommender systems and commercial profiles. a) There is some evidence to suggest that certain service types with common features and functionalities, are more likely to be used to commit and facilitate some offences. We have therefore identified some service types as a driver of risk, although we recognise it has limitations in offering a comprehensive and robust picture of what drives risk across all services. b) We have also identified recommender systems as a relevant characteristic because of the key role they play in determining what content users see and engage with, therefore contributing significantly to a user’s experience of a service. Recommender systems can be used in many ways which can influence how a user might experience risk of harm on a service. Most commonly this includes content recommender systems designed for the curation of content feeds, and network recommender systems that are used to recommend other users to follow/befriend. c) We have also included commercial profiles as our evidence showed that services with certain commercial profiles are likely to have weaker risk management, which can make them targets for perpetrators. We recognise that not all characteristics are inherently harmful; we therefore use the term ‘risk factor’ to describe a characteristic for which there is evidence of a risk of harm to individuals. For example, a functionality like livestreaming can be and is used by many people for benign purposes (for example, it is a very popular means by which people watch streamers playing videogames) but evidence has shown that it can be abused by perpetrators; when considering specific offences such as terrorism or CSEA, a functionality like livestreaming can give rise to risk of harm or the commission or facilitation of an offence.
Important distinctions in the Register of Risks¶
‘Advertising’ vs. ‘posting goods and services for sale’¶
We have made a distinction between two ways in which goods or services may be promoted on a service. This is because in some cases the service provider is generating revenue from this activity: where a fee is paid to the service provider in order for goods or services to be promoted or
‘advertised’ on the service. In other cases, users of a service may be advertising goods or services for sale, but the service provider generates no revenue from this activity. Therefore, we distinguish between: a) Advertising refers to paid-for advertising that generates direct advertising revenue for the service. This includes display advertising44, classified advertising45, and search advertising.46 We cover this under ‘business model’.47 b) Posting goods and services for sale refers to the ability for users to upload and share content that is dedicated to offering goods and services for sale on open channels of communication. Users may promote goods and services in this way, but it is distinct from ‘classified’ advertising because users do not pay for the content to be shared – although they, or buyers, may pay a fee in a different way, such as a percentage of the cost or a flat platform fee paid per sale. Therefore, it is not designed to generate direct advertising revenue for the service as classified advertising does. We cover this under ‘functionality’.48 The reason this distinction is important is because the risks associated with how a service provider generates revenue are separate to the risks posed by the functionalities provided to users and the different ways they might be used.
Size of a service¶
We also use two different ways to measure service size. Although they can sometimes be correlated, it is important to distinguish them in the risk assessment because of how they might increase risks of harm to individuals.49 a) Services with a large user base: refers to services which has an average user base of 7 million or more per month in the UK. b) Services with a small user base: refers to services with a small number of monthly UK users c) High-capacity services: refers to services with a large number of employees and/or revenue d) Low-capacity services: refers to services with a small number of employees and/or revenue
44 ‘Display advertising’ is where advertisers pay to display their advertising on an online service. It can appear in a variety of formats such as banner-style adverts (e.g. a banner advert at the top of a page in the Guardian), video advertising (e.g. a video ad appearing on Mumsnet or within a YouTube video), ‘native’ advertising (e.g. an ad for a sponsored product appearing on a Facebook feed) and sponsored content (e.g. a sponsored article on holidays in Italy in The Sunday Times). 45 ‘Classified advertising’ is where advertisers (who can be service users) pay to list specific products or services on an online service serving a market. The ad is listed under various headings and is grouped entirely in a distinct section away from display advertising. For example, an ad to sell a car in a dedicated section for car listings or advertising job opportunities under a dedicated category for job offers.
46 ‘Search advertising’ is where an advertiser pays for its advert to appear within a user’s search results on a search engine (e.g. on Bing, Yahoo or Google); the paid for ad will appear alongside the search engine results. 47 This is covered under advertising revenue models. ‘Boosted posts’, where users pay to amplify their content, will be captured under the analysis of business model as ‘transaction fees’ within our consideration of revenue models. 48 This is also sometimes considered under the umbrella of ‘organic advertising’. 49 For example, risks of harm from services with a large user base are related to higher reach, while risks from low number of employees/revenue (low-capacity service) are related to limited financial and technical ability to manage risk.
Ofcom’s approach to our risk assessment¶
Service characteristics¶
In our Register of Risks we have assessed the risks of harm associated with the specific characteristics of a service. The characteristics of a service as set out in the Act include any aspect of a service, including its functionalities, user base, business model, governance, and other systems and processes.50 • Functionalities is an umbrella term for the front-end features of a service that are visible to users. For U2U services, functionalities are defined as features that enable interaction between users. Functionalities for search services are defined as features that enable users to search websites or databases, as well as features that make suggestions relating to users’ search requests. The Act includes a non-exhaustive list of functionalities51, and the Ofcom risk assessment has also considered several other relevant functionalities in addition. • User base refers to the users of a service. The Ofcom risk assessment has considered the size of a service’s user base and user base demographics. It includes consideration of both registered and non-registered users of a service.52 • Business models, in a broad sense, refers to the ways in which a business operates to achieve its goals. For the purposes of the analysis in this Register of Risks, we adopt a narrow definition that includes revenue model and growth strategy.53 ‘Revenue model’ refers to how the service generates income or revenue (for instance, through advertising or subscriptions). ‘Growth strategy’ refers to how the service plans to expand its business. For instance, through increasing revenue and number of users. Also see ‘commercial profile’ below which was not specified in the Act but which we have added. • Governance, systems and processes (GSP):
50 These characteristics are specified in section 98(11). 51 Section 233: For U2U: (a) creating a user profile, including an anonymous or pseudonymous profile; (b) searching within the service for user-generated content or other users of the service; (c) forwarding content to, or sharing content with, other users of the service; (d) sharing content on other internet services; (e) sending direct messages to or speaking to other users of the service, or interacting with them in another way (for example by playing a game); (f) expressing a view on content, including, for example, by— (i) applying a ‘like’ or ‘dislike’ button or other button of that nature, (ii) applying an emoji or symbol of any kind, (iii) engaging in yes/no voting, or (iv) rating or scoring content in any way (including giving star or numerical ratings); (g) sharing current or historic location information with other users of the service, recording a user’s movements, or identifying which other users of the service are nearby; (h) following or subscribing to particular kinds of content or particular users of the service; (i) creating lists, collections, archives or directories of content or users of the service; (j) tagging or labelling content present on the service; (k) uploading content relating to goods or services; (l) applying or changing settings on the service which affect the presentation of user-generated content on the service; (m) accessing other internet services through content present on the service (for example through hyperlinks). For Search: (a) a feature that enables users to search websites or databases; (b) a feature that makes suggestions relating to users’ search requests (predictive search functionality). 52 The Act makes clear that ‘it does not matter whether a person is registered to use a service’ for them to be considered a ‘user’ (section 227 of the Online Safety Act). The Act is only concerned with the number of ‘United Kingdom users’ of the service, so where the user is an individual, they count as a user only where they are in the United Kingdom; similarly, where the user is an entity, they count only where they have been formed or incorporated in the United Kingdom (section 227(1) of the Online Safety Act 2023).
53 ‘Business model’ can be defined more widely to describe the way in which a service creates value to its users (value proposition), how it delivers this value to users, and how it captures value for itself. However, we adopt a narrow definition in the risk assessment to avoid overlap with the other risk characteristics. This does not affect the overall risk assessment as risk factors that would have been identified under the broader definition are captured elsewhere.
> Governance refers to the structures that ensure the adequate oversight, accountability, and transparency of decisions within a service which affect user safety. This is in relation to organisational structure as well as product and content governance. > Systems and processes refer to the actions taken by a service, including procedures to mitigate the risk of harm arising from illegal content being encountered, such as human moderators and automated systems or processes.
We also consider other characteristics that are not specified in the non-exhaustive list of characteristics in the Act, but for which there is evidence showing a relationship with the risk of harm to individuals. These include: • Service type. In general, this refers to the nature of the service,54 and includes, for example, social media services and private messaging services. • Recommender systems. Refers to information retrieval systems that determine the relative ranking of suggestions made to users on a U2U service. These include systems that recommend either content (content recommender systems) or other users (network recommender systems). • Commercial profile. Refers to the size of the service in terms of capacity (i.e. revenue and/or number of employees), the stage of service maturity and rate of growth in relation to users or revenue. Within the analysis for the Register of Risks, where we find evidence of a relationship between a characteristic of a service and a harm, we consider the characteristic to be a ‘risk factor’. As such, risk factors are specific characteristics of a service which Ofcom has identified as being associated with a risk of one or more kinds of illegal harm.55 For instance, direct messaging is a functionality that has been identified as a risk factor for some offences. These characteristics and the associated risk factors are broad and complex in scope. To make our assessment as accessible as possible, we sometimes group risk factors that are similar in nature or increase the risks of harm in a similar way. For example, functionalities such as direct messaging and video calling have been grouped under ‘user communication’ because they allow users to communicate with one another in a similar way. However, they are still considered to be separate risk factors and we have assessed them accordingly. Further information on this, including the full list of the most prominent, and potentially harmful, risk factors, is included in the Risk Profiles in the Service Risk Assessment Guidance and Risk Profiles. More information and definitions of terms used throughout this Register of Risks can be found in the Glossary (in the Annex of this document).
54 Certain kinds of services or ‘service types’ have been selected because our evidence suggests that they can be used to facilitate or commit relevant offences. 55 Terrorism offences, Child Sexual Exploitation and Abuse (CSEA) offences (Grooming; Child Sexual Abuse Material (CSAM)), Encouraging or assisting suicide (or attempted suicide) or serious self-harm offences, Harassment, stalking threats and abuse offences, Hate offences, Controlling or Coercive Behaviour (CCB) offence, Drugs and psychoactive substances offences, Firearms and other weapons offences, Unlawful immigration and human trafficking offences, Sexual exploitation of adults offences, Extreme pornography offence, Intimate image abuse offences, Proceeds of crime offences, Fraud and Financial services offences, Foreign Interference Offence, Communication offence - False communications offence, Threatening communications offence, Offences of sending or showing flashing images electronically (‘epilepsy trolling’), Offence of sending etc photograph or film of genitals (‘Cyberflashing’). For further information, refer to Section 5: Evidence and methodology for conducting our risk assessment.
How risk factors associated to characteristics have been identified¶
We used the following questions as a guide to identify the risk factors relevant to each group of characteristics: • Service type: What type of service (or aspects of it) can lead to a higher risk of harms to individuals from different offences? • User base: Who is using the service? How can user demographics influence which groups of users may experience or perpetrate harm and the ways in which this happens? How does the size of a user base affect risk? • Functionalities and recommender systems: How can the way in which the service enables users to interact or search lead to higher risks of harm to individuals? • Business model and commercial profile: How can the way in which the service achieves the goals of its business model and growth strategy lead to higher risks of harm to individuals? How can its commercial profile (capacity and maturity) affect its ability to manage risks? We acknowledge that some of the risk factors, which the evidence has demonstrated are linked to a particular kind of illegal harm, can also be beneficial to users. This can be in terms of the communication that they facilitate, or in some cases fulfilling other objectives, such as protecting user privacy. For instance, end-to-end encryption guarantees a user’s privacy and security of messages but makes it harder for service providers to moderate for illegal content. Similarly, the creation of an anonymous user profile appears to embolden user behaviour by providing users with a sense of protection, and confidence that they will not be held accountable for their actions online. This encourages some users to engage in behaviour, or post content, which they would not do if their real identity was recognisable. For example, there is evidence that hateful content targeting race or sexual orientation is more likely to be posted anonymously on some services. But at the same time, anonymous profiles allow users to question and criticise those in power without fear of repercussion, allows freedom of expression and protects a user’s right to privacy. While livestreaming can be a risk factor for several kinds of illegal harm as it can allow the real-time sharing of illegal content, it also allows for real-time updates in news, providing crucial information to a wide range of individuals. These considerations are a key part of the analysis underpinning our Codes measures.
Evidence¶
Our risk assessment process has consisted in identifying and analysing a repository of quality assured evidence of over 1000 individual sources. We have considered responses from our July 2022 call for evidence, our November 2023 Illegal Harms Consultation and August 2024 Illegal Harms Further Consultation, as well as relevant Ofcom research, academic papers from a range of disciplines, government bodies including law enforcement, third-party sources and information from charities and other non-government organisations. Given the wide range of third-party evidence that we are relying on, we have taken steps to ensure that our evidence sources are reliable. In particular, we
have considered the evidence by reference to the following criteria: method, robustness, ethics, independence and narrative.56 57 For the purpose of our risk assessment, we have identified a list of specific characteristics of services that we considered may be relevant to the risks of different kinds of illegal harms. We have then assessed any relevant evidence of whether and how particular kinds of illegal harm are impacted by the presence or absence of those characteristics, either individually or in combination. We have also engaged with external stakeholders including law enforcement and specialist agencies dealing with online threats to ensure we represent harms accurately. Most of the evidence reflects the experiences of UK users. However, for some offences, we have used research from other parts of the world where we felt it helps understand online experiences, either by complementing any UK evidence available, or providing additional insights in cases where there was no UK evidence.
Evidence base¶
Despite the extensive review of evidence, there remain gaps for some offences; in particular, there are gaps in the evidence that link the characteristics the Act requires us to assess against the kinds of illegal harms.58 At present, we hold less evidence about risk on search services compared to U2U services – there is less publicly available information about how they operate, and about the presence of illegal content in search results that can cause harm to individuals on these services. The amount of available evidence for specific kinds of illegal harm and offences is also varied. We have found it to be limited for some kinds of illegal harm (for example, extreme pornographic content offence), and for how services were used for the commission or facilitation of certain priority offences (for example, unlawful immigration, human trafficking, and firearms, knives and other weapons offences). We do not take this as an indication that the content or offences do not cause harm online, but as a reflection of the lack of reliable evidence at this time. We are also aware of the ethical and legal limitations to conducting research into certain kinds of illegal harm; research in those cases often focused on qualitative information instead. In some cases, we have been able to support our understanding of these harms by engaging with law enforcement and other specialist agencies. Where evidence is limited, we have used our judgment and expertise about specific harms to draw conclusions where we think this can help services to identify potential risks. We have also relied on some evidence that is about content or conduct that is broader than the specific offences, where we consider that this is nevertheless likely to be relevant to those offences. We have signposted this in the relevant parts of the Register of Risks. Due to the fast pace of technological changes and the
56 ‘Method’ examined the strengths and weaknesses of the methodology for that particular topic, such as whether appropriate data collection methods were used. ‘Robustness’ considered both the size and coverage of the sample, and quality of analysis – for example, how missing data values were accounted for. ‘Ethics’ refers to how well ethical considerations were addressed in the study, such as how personal data was handled. ‘Independence’ examined the origins of the research and whether any stakeholder interests might have influenced findings. ‘Narrative’ refers to the commentary within the report and whether conclusions are sufficiently backed by the research, and whether there is a clear distinction between the findings and the interpretation.
57 Some of the evidence used in this Register was published in a response to the development of the Act and other relevant legislation. These sources may have had aims or ambitions associated with the development of legislation. Moreover, some of the evidence used in this risk assessment comes from experts in their field, who may have developed their expertise while in the former employment of online services. 58 For example, our evidence base assessing governance, systems and processes and illegal harms is under-researched in some areas. We have therefore used different types of research and supporting evidence in this analysis.
speed at which risks of harm can manifest online, some of the evidence used within our risk assessment has come from non-traditional research sources; this timely evidence may not have the traditional levels of methodological and sampling rigour and peer reviewing that more traditional research sources have. This includes the use of videos and podcasts, as well as the use of investigative journalism. Lastly, we would highlight two important observations regarding the evidence of ‘types’ of service. First, some of the research-based evidence we refer to relates to specific services. We have included this evidence because it provides insights about particular risks that we consider have more general application. Its inclusion should not be seen as a judgement about the online safety practices of those specific service providers. Second, we do not have specific evidence relating to all types of U2U services. There is more research available - including on risks of harm to individuals - about large social media sites, gaming sites, and services that publish public information that can be analysed. Where appropriate, we have made reasonable inferences about the risks that may arise on other services where we do not have specific evidence about that service type.
Using the Register of Risks¶
The detailed research and analysis set out in our Register of Risks is intended to help services comply with their obligations under the Act. Over the subsequent 25 chapters, we set out our full risk assessment for illegal content on U2U and search services and consider the use of regulated U2U services to commit or facilitate priority offences. The Register of Risks is split into three parts: a) The first part user-to-user services (chapters 1 to 23) identifies characteristics of U2U services that may lead to increased risks of harm to individuals in relation to each of the kinds of illegal harm covered by the Act. This includes functionalities and recommender systems, user base, business models and commercial profiles. We consider both the risk of harm presented by the dissemination of illegal content on a U2U service, as well as the use of these services for the commission and/or facilitation of each kind of illegal harm. The risks from each kind of illegal harm on U2U services are explored in their own chapters. b) The second part search services (chapter 23) identifies the characteristics which can increase the risk of harm to individuals on search services. For search services, we only consider illegal content and not the use of a search service for the commission or facilitation of an offence (as per the Act’s requirements). We consider all of the kinds of illegal harm together. c) The third and final part (chapter 24) explores how the governance, systems and processes of a U2U or search service may lead to an increased risk of harm. We have identified two general scenarios where risk can arise from these areas themselves: (a) inadequate governance and/or other systems and processes currently in place within regulated services; and/or subsequently (b) an absence of such governance and other systems and processes. We also have an Annex which includes a glossary of terms used throughout the Register of Risks.
User-to-user services¶
This part of the Register of Risks presents a detailed analysis of the kinds of illegal harm, and their associated risks, on user-to-user (U2U) services.
This part includes 23 chapters. 18 cover the ‘kinds of illegal harms’ into which we have grouped all the priority offences in the Act, and a further 5, each covering a relevant non-priority offence. The chapters on kinds of illegal harm distinguish each priority offence where the evidence permits, and consider risks of harm to users more widely, where justified. For example, the Terrorism chapter includes a number of priority offences relating to terrorism. Within it, we have evidence pointing to the risks of harm relating to several of the terrorism offences listed, while other evidence may point to the particular harm of one specific terrorism offence. We have analysed evidence for the other relevant non-priority offences in the same way, using the same structure as the other chapters. In each chapter we have considered evidence from a variety of sources, including information provided by services, academic literature, third-party research, civil society in general and Ofcom’s own research. Each chapter is structured as follows: a) Summary of the chapter, including important risk factors identified. b) Introduction to the harm and the relevant offences covered. c) How the offences manifest online. This reviews the presence of the harm online and the risks of harm that users may experience. This will help a service understand the context for the harms and the particular risks a service should be aware of. d) Evidence of risk factors. The evidence to form the basis of our analysis is presented for each characteristic: service type, user base, functionalities and recommender systems, and business models and commercial profiles. This final section will allow services to develop a better understanding of how specific characteristics relate to, and impact, the risks of harm.
U2U service types¶
We refer to U2U service types that we expect to be recognisable to both users and businesses, to illustrate how harms can manifest online and how the characteristics of a service can affect the risks of harm to individuals. The U2U service types below should not be taken to be a definitive view of the services (or parts of services) that may be in scope of the Act or a classification that sets expectations about a service provider’s risk assessment. It is for services to assess themselves and seek their own independent advice to enable them to understand and comply with the Act. For more, please refer to the ‘Overview of regulated services’: • Social media services: Social media services connect users and enable them to build communities around common interests or connections. • Video-sharing services: Video-sharing services allow users to upload and share videos with the public. • User-to-user pornography services: Service type whose principal purpose is to disseminate user-generated pornography. • Discussion forums and chat room services: Discussion forums and chat rooms generally allow users to send or post messages that can be read by the public or by an open group of people. • Marketplaces and listings services: Marketplaces and listings services allow users to buy and sell their goods or services.
• Dating services: Dating services enable users to find and communicate with romantic or sexual partners. • Gaming services: Gaming services allow users to interact within partially or fully simulated virtual environments. • Messaging services: Messaging services are typically centred around the sending and receiving of messages that can only be viewed or read by a specific recipient or group of people. • File-storage and file-sharing services: File-storage and file-sharing services are services whose primary functionalities involve enabling users to store digital content and share access to that content through links. • Information-sharing services: Information sharing services are primarily focused on providing user-generated informational resources to other users. • Fundraising services: Fundraising services typically enable users to create fundraising campaigns and collect donations from users. • Payment services: Financial payment providers often have websites or applications that enable users to send and receive money.59
Recent developments such as GenAI can also be relevant when considering service types. We will continue to monitor the U2U landscape with the expectation that new types of services and research showing a risk of harm associated with them will emerge.
59 These services can sometimes allow users to share user-generated content such as messages.
Section 1 Terrorism¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for terrorism offences: how harm manifests online, and risk factors Terrorism is considered a violent action or threat of action, designed to influence a government or intimidate the public and advance a cause. Online terrorism content is any content made available to others online, which can encourage or promote terrorism. Although online terrorism content is not widespread on user-to-user (U2U) services, the impact on individuals and communities can be substantial, both physically and mentally. Service type risk factors: Terrorist content encountered by UK users does not rely on a single service but on many services and their associated functionalities. We have found that there are often cross platform sharing of terrorism content; from being first posted on smaller U2U services and then linked to larger, higher-reach services and vice versa. A wide range of types of U2U services are known to be used by terrorist actors. Social media services are particularly relevant to the perpetration of this harm because of their reach and popularity. Terrorist content is also often identified on file-storage and file-sharing services. Similarly, file-storage and file-sharing services have been identified specifically as a risk factor in facilitating the creation of 3D-printed firearms. Gaming services have also been used by terrorists as recruitment and training tools, while marketplaces and listing services can be used to raise and collect funds. Other services are also used to organise, recruit, fundraise and disseminate terrorism content. These include video-sharing services, discussion forums and chat rooms, messaging services, fundraising services, and payment services. Our evidence suggests that services which facilitate the creation of online communities of like-minded individuals, such as in discussion forums or chat rooms, may increase the risks of harm related to terrorism. They can enable potential perpetrators and organised communities to encourage each other to share terrorism content, which may lead to an increase in the risks of harm from terrorism. User base risk factors: User base size can increase the risks of harm from terrorism offences. U2U services with a large user base and high reach are a risk factor because services with a large user base can enable the dissemination of terrorism content to many users, often quickly or virally. However, services with a small user base can also be used by
perpetrators to undertake more sensitive activities, such as recruitment, planning and fundraising. Research also indicates that men are more likely to encounter radicalisation and terrorist content than women. And children are more vulnerable to becoming radicalised. Functionalities and recommender systems risk factors: Many of the functionalities listed are common across U2U services; in principle, a wide range of U2U services could be used for disseminating terrorism content, and this makes it harder to identify which services are especially risky, based solely on a general analysis of functionalities. Perpetrators often use functionalities such as posting content, commenting on content and hyperlinking to share and direct users to content such as memes, and content which provides instructions related to terrorist activities. Our evidence points to these functionalities increasing the likelihood of terrorism content being shared. This, in turn, increases the risk of harm to individuals exposed to this content and could lead to incitement to commit terrorist acts, the dissemination of material such as weapons training, and the recruitment of people. User connections and user tagging also allow terrorism content to be disseminated through users’ networks, especially when official pages or channels are removed. The ability to livestream is a risk factor that has been used to broadcast terrorist attacks and to target groups with protected characteristics. In the past, this functionality has been abused or exploited on many occasions to incite and encourage terrorism, particularly by far-right terrorists who often seek to emulate the tactics of previous terrorists. Any service offering group messaging can allow terrorists to share content in a low-friction way with like-minded people. Encrypted messaging is particularly attractive to terrorist groups as this can reduce the chance of detection. Several other functionalities are relevant to terrorism offences. Screen capturing or recording increases the risks of harm by enabling users to store and disseminate extremist content. User-generated content searching allows individuals to easily seek out terrorist content while content recommender systems can increase the risk of exposure to it. Our evidence finds that direct, encrypted, and ephemeral messaging are also used by terrorist actors for organisation and security purposes. Anonymous user profiles can also heighten the risks of harm, with users less fearful of sharing such content when they are anonymous. Users have also been shown to create fake user profiles by altering their usernames to avoid having their accounts blocked. Business model risk factors: The capacity and maturity of a service can contribute to risk and be exploited by perpetrators, with varying effects on users. Low capacity and early-stage services may be more vulnerably to risks of hosting terrorist content because of their
limited knowledge, resources or technical capability to moderate such content. This reflects the opportunistic nature of perpetrators to use online services to heighten the risks of harm from terrorism, for varying purposes.
Introduction¶
Relevant offences¶
60 The offences listed at points (a) to (p) refer to the offences under the following provisions of the Terrorism Act 2000: sections 11; 12(1), 12(1A); 12(2), 13(1A), 15; 16(1), 16(2); 17; 18; 54(1); 54(3); 56; 58; 58A; sections 59 to 61f.
• Publishing information about members of the armed forces etc • Inciting terrorism outside the United Kingdom • Use of noxious substances or things61 • Encouragement of terrorism62 • Dissemination of terrorist publications • Preparation of terrorist acts • Training for terrorism • Terrorist threats relating to radioactive devices
61 The offence listed at point (q) refers to the offence under section 113 of the Anti-terrorism, Crime and Security Act 2001. 62 The offences listed at point (r) to (v) cover the offences under the following provisions of the Terrorism Act 2006: sections 1; 2; 5; 6; 11. 63 The Crown Prosecution Service, 2022. Terrorism. [accessed 29 June 2023]. 64 The Home Office, 2021. Proscribed terrorist groups or organisations. [accessed 29 June 2023]. 65 Note that in the event of a livestreamed attack, service providers are unlikely to be in a position to carry out an assessment of the motives of the attacker in sufficient time, so it is much more likely the content would be ‘illegal content’ under a public order offence. See the ICJG chapter ‘Terrorism’ for more information.
How terrorism offences manifest online¶
66 Ofcom, 2024. Online Experiences Tracker - Wave 6. [accessed 11 November 2024]. 67 In a survey with UK adults, 23% were identified as displaying information use behaviour that reflected a moderate or high likelihood of actively seeking information that incited/facilitated terrorism, and that 31% of participants exhibited information use behaviours that suggested a moderate of high likelihood of being incidentally exposed to such information. Note that this is a much broader definition of potentially terrorist content and exposure risk than the Online Experiences Tracker. Source: Schuman, S., Clemmow, C., Rottweiler, B., Gill, P. 2024. Distinct patterns of incidental exposure to and active selection of radicalizing information indicate varying levels of support for violent extremism. [accessed 30 August 2024]. 68 For example, a study conducted by the Tony Blair Institute states that ‘to access extremist content via social media, you need to know where to look and, in most cases, individuals will already have been exposed to terrorist thinking through social circles offline, or they will be aware of accounts disseminating content and will actively follow them on other platforms.’ Source: Tony Blair Institute, 2016. A War of Keywords: How extremists are exploiting the internet and what to do about it. [accessed 19 September 2023].
69 Tech Against Terrorism is an initiative launched and supported by the United Nations Counter Terrorism Executive Directorate (UN CTED) working with the global tech industry to tackle terrorist use of the internet whilst respecting human rights. 70 Tech Against Terrorism, 2021. Trends in Terrorist and Violent Extremist Use of the Internet. [accessed 29 June 2023]. 71 Under the Act, Ofcom is required to identify and assess risks connected with regulated U2U and search services. If a standalone website does not offer either of these services, it is likely to be out of scope of regulation. 72 Tech Against Terrorism, 2021. 73 Ofcom, 2022. The Buffalo attack: Implications for online safety. [accessed 29 June 2023].
potential perpetrators makes it harder for a service to track down terrorism content, which can lead to it reaching a higher number of individuals.
74 The provision of blueprints or instructions which allow another person to 3D print (‘make’) a firearm would be considered illegal content in relation to offences relating to the provision of instructions or training in the making or use of firearms. Refer to the ICJG (Terrorism offences chapter) for further details. 75 The 2024 National Strategic Assessment from the National Crime Agency (NCA) stated “there were no confirmed criminal discharges using 3D-printed firearms in 2023, although it is highly likely that criminals have a growing interest in hybrid 3D-printed firearms.” Source: NCA, 2024. Criminals still want to acquire and use original lethal purpose weapons but they are finding them more difficult to obtain. [accessed 22 October 2024]; Global Network on Extremism and Technology (GNET) (Basra, R.), 2022. The Future is Now: The Use of 3D-Printed Guns by Extremists and Terrorists. [accessed 22 October 2024]. 76 Potential new laws to criminalise the making, supply and possession of items strongly suspected to facilitate serious crime – such as digital templates for 3D-printing firearms components. Source: Home Office, 2023. Consultation document (accessible). [accessed 20 September 2023]. 77 For example, in 2023 and 2024, several men have been convicted for possession of semi-automatic firearms, printed from files available online. For example: Laversuch, C., 2023. Pair jailed for 3D-printed sub-machine gun plot, BBC News, 16 May. [accessed 22 October 2024]; Staffordshire Police, 2024. Man who made guns using 3D printer jailed for more than 10 years. [accessed 22 October 2024].
78 For example, several British men have been convicted of terrorism offences in recent years in cases where 3D printed firearms were linked to the offenders engagement with extreme right-wing ideology. Sources: Counter Terrorism Policing, 2024. Portsmouth man sentenced for nine years and six months for terrorism offences. [accessed 18 November 2024]; The Independent (Dearden, L.), 2022. ‘Fascist cell’ convicted of terror and firearm offences after trying to make 3D-printed gun, 29 March. [accessed 18 November 2024]. 79 National Crime Agency response to the Illegal Harm November 2023 Consultation. 80 NCA, 2024. 81 Tech Against Terrorism, 2023. Early Terrorist Adoption of Generative AI [accessed 12 June 2024].
Risks of harm to individuals presented by online terrorism offences¶
82 Defined as "any offence committed in association with a group, cause, and/or ideology that propagates extremist views or actions and justifies the use of violence or illegal conduct in pursuit of its objective” Source: HM Prison and Probation Service, 2019. Exploring the role of the Internet in radicalisation and offending of convicted extremists. [accessed 3 July 2023]. 83 The research states that information was coded relating to internet activities and behaviours commonly associated with online radicalisation. The following variables were coded: 1. Learnt from online sources. 2. Interact with co-ideologues online. 3. Generate their own extremist propaganda online. 4. Provision of material support online. 5. Access to specific extremist websites. 6 Use of open social media platforms. 7. Use of email/standard chat applications. 8. Use of encrypted applications. It is therefore possible that activity conducted on out-of-scope services has been captured as part of this research. Source: HM Prison and Probation Service (Kenyon. J, Binder. J, and Baker-Beall, C.), 2021. Exploring the role of the Internet in radicalisation and offending of convicted extremists. [accessed 3 July 2023]. 84 It is worth noting that the scope of the research extended beyond regulated services and included websites that are unlikely to offer any in-scope user-to-user or search services, and email service providers. However, many services included in the report are likely to be within scope of regulation and hence the insights highly relevant for an assessment of the risks of harm from terrorism offences. Furthermore, other evidence in this chapter also points to websites with terrorism content being part of the wider online ecosystem and at times being found or accessed through the sharing of links on in-scope service. Radicalisation was found to take place primarily online, particularly between 2019 and 2021. However, it is currently unclear to what extent the Covid-19 pandemic and associated restrictions accounted for this. The primary method of radicalisation for individuals who were convicted between 2015 and 2017 was as follows: internet: 17 individuals (27%); face-to-face: 11 individuals (17%) and hybrid: 36 individuals (56%). The report states that despite evidence suggesting the increasing prominence of the internet in radicalisation processes, it cannot be concluded that the online domain is simply replacing the offline domain, as offline influences such as previous involvement or conviction for non-terrorism offences featured at least to some extent for most convicted extremists in the dataset. Source: HM Prison and Probation Service report, 2021. [accessed 19 September 2023].
85 United Nations Office of Drugs and Crime, 2012. The Use of the Internet for Terrorist Purposes. [accessed 29 June 2023]. 86 United Nations Office of Drugs and Crime, 2012.
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services and messaging services¶
87 The Institute for Strategic Dialogue (ISD) is an organisation dedicated to reversing rising extremism worldwide. Source: Institute for Strategic Dialogue, Northern Ireland Related Terrorism (Manzi, Z.), 2024. [accessed 22 October 2024]. 88 Further information on the specific ways in which these services may be used can also be found under Risk factors: Functionalities and recommender systems. 89 Observer Research Foundation (Saltman, E.), 2022. Identifying and Removing Terrorist Content Online: Cross-Platform Solutions. [accessed 4 July 2023]. 90 Internal ‘social media strategy’ documents from the proscribed terrorist organisation ISIS provide a simple example of how important the group felt it was to maintain a social media presence on large mainstream services, as far back as 2014 and earlier. Source: Berger, J.M., & Morgan, J. 2015. The ISIS Twitter census: Defining and describing the population of ISIS supporters on Twitter. p.55. [accessed 12 September 2024].
91 Tech Against Terrorism, 2021. Terrorist use of E2EE: State of play, misconceptions and mitigation strategies. [accessed 3 July 2023]. 92 Texas National Security Review (Fishman, B.), 2019. Crossroads: Counter-terrorism and the Internet. [accessed 3 July 2023]. 93 Tech Against Terrorism, 2021.
Discussion forums and chat room services¶
File-storage and file-sharing services¶
94 HM Prison and Probation Service (Kenyon, J., Binder, J. and Baker-Beall, C.), 2022. The Internet and radicalisation pathways: technological advances, relevance of mental health and role of attackers. [accessed 3 July 2023]. 95 Tech Against Terrorism, 2022. Terrorist content analytics platform: year one: 1 December 2020 – 30 November 2023. [accessed 3 July 2023]. 96 Tech Against Terrorism, 2023. Patterns of Online Terrorist Exploitation. [accessed 3 July 2023]. 97 In 2022 Police raided a gun factory and found 3D printed components understood to play a role in the firearms. Source: Davis, M., 2022. Haul of 3D-printed gun parts and bullets one of largest in UK, The Independent, 12 October. [accessed 24 September 2024]; Dass, R. and Mok, B., 2023. Assessing the Impact of 3D-Printed Weapons on the Violent Extremist Milieu. [accessed 24 September 2024]. 98 Global Network on Extremism and Technology (Lewis, J.), 2021. 3D-Printed Guns, Untraceable Firearms, and Domestic Violent Extremist Actors. [accessed 17 October 2024].
99 One type of 3D-printed firearm was identified as being used in at least 18 open source reports of conflict between 2020 and 2023. Sources: Basra, R., 2023. Behind the Mask: Uncovering the Extremist Messages of a 3D-Printed Gun Designer. [accessed 11 September 2024]; Dass, R., 2023. 3D-Printed Weapons and the Far-Right: The Finnish Accelerationist Cell. [accessed 11 September 2024]. 100 Miotto, N., 2021. The Role of Online Communities in Supporting 3D-Printed Firearms. [accessed 17 October 2024]. 101 Advancements in technology have led to 3D printers becoming much more advanced. A CAD or STL file can now be sent directly to the printer via a LAN connection or Bluetooth. 102 Counter-terror Policing response to the Illegal Harm November 2023 Consultation.
Gaming services¶
Marketplaces and listings services, fundraising services, and payment services¶
Risk factors: User base¶
User base size¶
User base demographics¶
103 United Nations Office of Drugs and Crime, 2012. The Use of the Internet for Terrorist Purposes. [accessed 29 June 2023]. 104 United Nations Office on Drugs & Crime report, 2012. 105 United Nations Office on Drugs & Crime report, 2012. 106 National Crime Agency, 2023. 107 Dass, R., 2023. 3D-Printed Weapons and the Far-Right: The Finnish Accelerationist Cell. [accessed 11 September 2024]. 108 United Nations Office of Drugs and Crime, 2012. The Use of the Internet for Terrorist Purposes. [accessed 29 June 2023].
109 Foreign Policy Magazine (Ware, J. and Clarke, C.P.), 2022. How Far-Right Terrorists Choose Their Enemies [accessed 3 July 2023]. 110 Ofcom, 2022. The Buffalo attack: Implications for online safety. [accessed 29 June 2023]. 111 Ofcom, 2024. Online Experiences Tracker - Wave 6. [accessed 11 November 2024]. 112 Europol, 2022. European Union Situation and Trend Report 2022 [accessed 12 June 2024]. 113 Home Office, 2023. Individuals referred to and supported through the Prevent Programme, April 2021 to March 2022. [accessed 12 June 2024].
114 Rose, H. and Vale, G., International Centre for the Study of Radicalisation, 2013. Childhood Innocence?: Mapping Trends in Teenage Terrorism Offenders [accessed 27 June 2024]. 115 Schumann, S., Clemmow, C., Rottweiler, B. and Gill, P., 2024. Distinct patterns of incidental exposure to and active selection of radicalizing information indicate varying levels of support for violent extremism, PLoS ONE 19(2). [accessed 5 June 2024]. 116 We recommend readers refer to the Protection of Children’s Register of Risks to understand the role that legal content can play in radicalising children online.
with mental health conditions or in ethnic minority groups, and manipulate or threaten them into performing various acts which can include mass shootings and spreading far-right terrorist ideology.117
Risk factors: Functionalities and recommender systems¶
User identification¶
Anonymous user profiles and fake user profiles¶
User networking¶
User connections and user tagging¶
117 Global Network on Extremism and Technology (Argentino, M., Barrett, G. and Tyler, M. B.), 2024. 764: The Intersection of Terrorism, Violent Extremism and Child Sexual Exploitation [accessed 02 October 2024]. 118 Koehler, D., 2014. The Radical Online: Individual radicalisation processes and the role of the internet, Journal for Deradicalisation, Winter 2014/15 (1). [accessed 4 July 2023]. 119 Institute for Strategic Dialogue (O’Connor, C.) 2021. Hatescape: An In-Depth Analysis of Extremism and Hate Speech on TikTok. [accessed 4 July 2023].
120 ISIS (Islamic State of Iraq and Syria), also known as ISIL (Islamic State of Iraq and the Levant), is a proscribed terrorist organisation based in Syria. Source: RAND Corporation, n.d. The Islamic State (Terrorist Organisation). [accessed 23 August 2023]. 121 Global Network on Extremism and Technology (McDonald, B.), 2022. Extremists are Seeping Back into the Mainstream: Algorithmic Detection and Evasion Tactics on Social Media Platforms. [accessed 4 July 2023]. 122 Global Network on Extremism and Technology (GNET) (Basra, R.), 2022. The Future is Now: The Use of 3D-Printed Guns by Extremists and Terrorists. [accessed 22 October 2024].
says that although official pages or channels spreading pro-ISIS content may be removed, a large network of individual users can quickly amass thousands of connections. GNET gives the example of Facebook, where it found that many pro-ISIS accounts had “maxed out the 5,000 ‘friend connections’ allowed by the platform”. According to the GNET, the thousands of ‘friend connections’ by “pro-ISIS accounts on this platform are large enough to match or even exceed the audiences of many official ISIS channels found on encrypted alt-tech123 platforms”.124
User communication¶
Livestreaming¶
Live audio¶
124 Global Network on Extremism and Technology (GNET) (Basra, R.), 2022. 125 “Al-Shabaab, the al-Qaeda affiliate based in East Africa, live-tweeted an attack on the Westgate mall in Nairobi, Kenya, explaining and justifying its actions as it killed 67 people”. Source: Cronin, A. K., 2020. Power to the People – How Open Technological Innovation is Arming Tomorrow’s Terrorists. Oxford University Press, p.189. 126 Ofcom, 2022. The Buffalo attack: Implications for online safety. [accessed 29 June 2023]. 127 “A gunman went live on a social media service before he shot and killed 51 people at local mosques. In the same year, a gunman in Germany also livestreamed his attack on a social media service”. Source: Brooks, A and Matromarino, J.P., 2022. Extremists exploit gaming networks and social media to recruit and radicalize, Wbur, 19 May. [accessed 19 September 2023]. 128 In 2016, a man in France used a social media live feature to broadcast his justification for killing two police officers whilst holding a child hostage and pledging his allegiance to the Islamic State. In 2019 a gunman reportedly livestreamed himself through his channel on a social media service, attacking a synagogue and a kebab shop in Halle, Germany. In 2020, an attacker livestreamed himself carrying out an attack in a mall in Glendale, Arizona. Source: Ofcom, 2022. The Buffalo attack: Implications for online safety. [accessed 29 June 2023].
129 Ofcom, 2022. The Buffalo attack: Implications for online safety. [accessed 29 June 2023]. 130 Andrews, S. 2023. The ‘First Person Shooter’ Perspective: A Different View on First Person Shooters, Gamification, and First Person Terrorist Propaganda, Games and Culture 19(1). [accessed 4 July 2023]. 131 Bryden, M., Bahra, P., Cruickshank, P., Macklin, G., Cook, J., Vale, G and Simcox, R., 2019. CTCSENTINEL. [accessed 26 July 2023]; Kupper, J., Christensen, T. K., Wing, D., Hurt, M., Schumacher, M., Meloy, R., 2022. The_Contagion_and_Copycat_Effect_in_Transnational_Far-right_Terrorism_An_Analysis_of_Language_Evidence. Perspectives on Terrorism 16(4), 4-26. [accessed 4 July 2023].
behaviours leading to these gaming-related online “spaces providing extremists with the opportunity to broadcast their messages widely, and relatively undisturbed”.132
Direct messaging, group messaging, encrypted messaging, and ephemeral messaging¶
Posting content (text, images, videos)¶
132 United Nations Office of Counter-Terrorism, 2022. Examining the Intersection Between Gaming and Violent Extremism. [accessed 4 July 2023]. 133 Fishman, B. 2019. Crossroads: Counter-terrorism and the Internet, Texas National Security Review, 2(2), 82-100. [accessed 4 July 2023]. 134 Tech Against Terrorism, 2021. Terrorist use of E2EE: State of play, misconceptions, and mitigation strategies. [accessed 4 July 2023]. 135 Tech Against Terrorism report, 2021. 136 “Alt-jihadists draw on the narratives of the alt-right and far right in Western culture wars while staying on brand with support for staple extremist groups such as Hezbollah, the Houthis, Hamas, the Taliban, Hayat Tahrir al-Sham, al-Qaeda, and the Islamic State.” Source: Ayad, M., 2021. An ‘Alt-Jihad’ is Rising on Social Media, Wired, 8 December. [accessed 4 July 2023].
137 “A/B testing is a way to compare two versions of something to figure out which performs better. While it’s most often associated with websites and apps, the method is almost 100 years old and it’s one of the simplest forms of a randomized controlled experiment”. Source: Harvard Business Review, 2017. A refresher on A/B testing. [accessed 26 July 2023]. 138 Ayad, M., 2021. 139 Ayad, M., 2021.
hazardous materials; and how to plan and execute terrorist attacks”.140 The services make it easy for material to be shared among a large group of people, and can also help build a sense of community among individuals in different locations and with different backgrounds, “encouraging the creation of networks for the exchange of instructional and tactical material”.141
Commenting on content¶
140 United Nations Office on Drugs & Crime, 2012. The use of the Internet for terrorist purposes. [accessed 4 July 2023]. 141 United Nations Office on Drugs & Crime, 2012. 142 Institute for Strategic Dialogue (Ayad, M.), 2023. CaliphateTok: TikTok continues to host Islamic state propaganda [accessed 14 June 2024]. 143 Atomwaffen Division (AWD), an extreme right-wing group. 144 Institute for Strategic Dialogue (Gallagher, A., O’Connor, C., Vaux, P., Thomas, E., Davey, J.), 2021. Gaming and Extremism, The Extreme Right on Discord. [accessed 4 July 2023]. 145 Global Network on Extremism and Technology (GNET) (Basra, R.), 2022. The Future is Now: The Use of 3D-Printed Guns by Extremists and Terrorists. [accessed 22 October 2024].
146 Bots is an umbrella term that refers to a software application or automated tool that has been programmed by a person to carry out a specific or predefined task without any human intervention. 147 Silva, S., 2020. Islamic State: Giant library of group's online propaganda discovered, BBC, 4 September. [accessed 19 September 2023]; Institute for Strategic Dialogue, 2020. Click reveals ISD discovery of huge pro-ISIS online cache. 8 September. [accessed 19 September 2023]. 148 Global Network on Extremism and Technology (GNET) (Basra, R.), 2022. 149 Bellingcat, (R. Evans), 2021. White Boy Summer, Nazi Memes and the Mainstreaming of White Supremacist Violence. [accessed 4 July 2023].
Transactions and offers¶
Online payments and crowdfunding¶
Content storage and capture¶
Screen capturing or recording¶
Content editing¶
Editing visual media¶
154 For instance, research exploring the impact of exposure to potentially radicalizing information suggests that individuals who actively seek out terrorism content are at a higher risk of radicalisation. Source: Schuman, S., Clemmow, C., Rottweiler, B., Gill, P. 2024. Distinct patterns of incidental exposure to and active selection of radicalizing information indicate varying levels of support for violent extremism. [accessed 30 August 2024]. 155 Ofcom, 2022. The Buffalo attack: Implications for online safety. [accessed 29 June 2023]. 156 Ofcom, 2022. The Buffalo attack: Implications for online safety. [accessed 29 June 2023].
157 Fishman, B. 2019. Crossroads: Counter-terrorism and the Internet Texas National Security Review, 2 (2). [accessed 4 July 2023]. 158 Tech against Terrorism response to Ofcom’s Call for Evidence dated 2022. 159 “ISD has identified a networked community of ‘alternative’ support for groups like Al Qaeda and the Islamic State, blending the aesthetics of ‘chan culture’ the alt-right, and extremist groups. This community of supporters, whom ISD researchers are referring to as ‘alt-jihadists’, specialises in producing and disseminating Salafi-jihadist content using familiar ‘chan culture’ and alt-right meme characters such as Pepe the Frog.” Source: Institute of Strategic Dialogue, 2022. Looking Beyond the Traditional threat: Alt-Jihadism. [Accessed August 23 2023].
militant groups, including Hamas and jihadist organisations.160 There is also evidence that steganography, the hiding of messages in images, is in widespread use by terrorist organisations.161
Recommender systems¶
Content recommender systems¶
Risk factors: Business models and commercial profiles¶
Revenue model¶
160 Ayad, M., 2021. An ‘Alt-Jihad’ is Rising on Social Media, Wired, 8 December. [accessed 4 July 2023]. 161 United Nations Office of Drugs and Crime, 2012. The Use of the Internet for Terrorist Purposes. [accessed 29 June 2023]. 162 The term ‘extreme’ is based on the Holbrook’s Extremist Media Index definition. Source: Holbrook, D., 2015. Designing and Applying an ‘Extremist Media Index‘. Perspectives on Terrorism, 9(5). [accessed 20 September 2023]. 163 RUSI (Reed, A., Whittaker, J., Votta, F. and Looney, S.), 2019. Radical Filter Bubbles: Social Media Personalisation Algorithms and Extremist Content. [accessed 4 July 2023].
ensuring that bad actors have no access to advertiser funding”164 (Global Alliance for Responsible Media) show how advertisers have a role in protecting individuals against harm such as terrorism.
Commercial profile¶
Capacity and maturity¶
164 World Federation of Advertisers, 2020. Marketing leaders take action on harmful online content. [accessed 4 July 2023]. 165 Section 2.3.1 of the European Commission’s Proposal for a Regulation of the European Parliament and of the Council on preventing the dissemination of terrorist content online: “Hosting service providers are abused for the dissemination of terrorist content online affecting the business models and users’ trust in the digital single market” Source: European Commission, 2018. Proposal for a Regulation of the European Parliament and of the Council on preventing the dissemination of terrorist content online. [accessed 4 July 2023]. 166 This is different from our analysis of user base size set out earlier. Low capacity is related to size in terms of number of employees and/or revenue, which may increase risk due to ability to moderate, while risks from large services (i.e. with large user base) are related to reach. 167 Tech Against Terrorism, 2021. Trends in Terrorist and Violent Extremist Use of the Internet. [accessed 29 June 2023]. 168 Tech Against Terrorism, 2019. Analysis: ISIS Use of smaller platforms and the DWeb to share terrorist content – April 2019. [accessed 4 July 2023].
169 Service size as it relates to the size of its user base is discussed under user base size. However, our research shows that sometimes services with a small user base can have fewer technical and financial resources. 170 During a panel event hosted for its UK launch, Tech Against Terrorism explained that “terrorists exploit an overlapping ecosystem of services, not just the big platforms like Facebook and Twitter but also the smaller services.” The initiative expressed the concern that smaller technology companies are at risk of being exploited by terrorist groups when disseminating propaganda but often do not have the scale or resources to tackle terrorism content or to comply with legal requirements. Source: Tech Against Terrorism, 2017. UK Launch of Tech Against Terrorism at Chatham House. [accessed 4 July 2023].
171 Europol, 2018. European Union Terrorism situation and trend report. [accessed 4 July 2023].
Section 2 Child Sexual Exploitation and Abuse (CSEA)¶
Warning: This chapter contains content that may be upsetting or distressing in relation to CSEA.
Introduction¶
172 We will set out and describe the evidence we have available about Child Sexual Exploitation and Abuse, which will require us to refer regularly to terms that are long and which, when replaced with acronyms, make reading the document a swifter and neater experience. As is common practice, each term is written out in full upon its first usage, followed by its acronym in parenthesis, which will be used on each occasion afterwards. Our use of acronyms does not detract from the significance and gravity of these terms and in no way reflects any intent to diminish their seriousness.
173 HM Government, 2023. Working Together to Safeguard Children. [accessed 24 September 2024]. 174 CSEA offences vary in relation to the age, or reasonably perceived age, of the victim; some offences may relate to children under the age of 16 while others apply to all children under the age of 18. In addition, other offences consider factors such as the power imbalance between the perpetrator and the child, namely those in a position of trust. For a full overview of these offences please see the Illegal Content Judgements Guidance (ICJG).
structures and overlapping content. However, the evidence in each section focuses on these harms individually. The subsections explore in greater detail: a) How these specific harms manifest online b) Evidence of risk factors in user-to-user (U2U) services c) Specific risk factors related to service types d) Risk factors associated with user bases, such as size and demographics e) Risk factors related to platform functionalities and recommender systems f) Risk factors linked to business models and commercial strategies
Relevant offences¶
176 Child sexual exploitation is a form of child sexual abuse that occurs where an individual or group takes advantage of an imbalance of power to coerce, manipulate or deceive a child or young person under the age of 18 into sexual activity (a) in exchange for something the victim needs or wants, and/or (b) for the financial advantage or increased status of the perpetrator or facilitator. The victim may have been sexually exploited, even if the sexual activity appears consensual. Child sexual exploitation does not always involve physical contact; it can also occur through the use of technology. This is explained in greater detail in the Human trafficking chapter. See also, Department for Education, 2017. Child sexual exploitation. [accessed 22 September 2023]. 177 An offence under the following provisions of the Sexual Offences Act 2003: section 47 (paying for sexual services of a child); section 48 (causing or inciting sexual exploitation of a child); section 49 (controlling a child in relation to sexual exploitation); section 50 (arranging or facilitating sexual exploitation of a child). An offence under the following provisions of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)): article 21 (arranging or facilitating commission of a child sex offence); article 37 (paying for the sexual services of a child); article 38 (causing or inciting child prostitution or pornography); article 39 (controlling a child prostitute or a child involved in pornography); article 40 (arranging or facilitating child prostitution or pornography). An offence under the following provisions of the Protection of Children and Prevention of Sexual Offences (Scotland) Act 2005: section 9 (paying for the sexual services of a child); section 10 (causing or inciting provision by child of sexual services or child pornography); section 11 (controlling a child providing sexual services or involved in pornography); section 12 (arranging or facilitating provision by child of sexual services or child pornography).
2.9 relation to offences in Scotland, being involved art and part in the commission of those offences).
178 For children under the age of 13: section 8 of the Sexual Offences Act 2003, article 15 of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)), and section 21 of the Sexual Offences (Scotland) Act 2009. For children over the age of 13: Section 10 of the Sexual Offences Act 2003, Article 17 of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)), section 31 of the Sexual Offences (Scotland) Act 2009, and section 54 of the Sexual Offences (Scotland) Act 2009. 179 Section 11 of the Sexual Offences Act 2003 and Article 18 of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)). 180 Section 12 of the Sexual Offences Act 2003 and Article 19 of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)). 181 section 23 and 33 of the Sexual Offences (Scotland) Act 2009. 182 Section 14 of the Sexual Offences Act 2003 and Article 21 of the Sexual Offences (Northern Ireland) Order 2008 (S.I. 2008/1769 (N.I.2)). 183 Section 15 of the Sexual Offences Act 2003 and Article 22 of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)). 184 Section 1 of the Protection of Children and Prevention of Sexual Offences (Scotland) Act 2005. 185 Section 15A of the Sexual Offences Act 2003, article 22A of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)).
186 sections 24 and 34 of the Sexual Offences (Scotland) Act 2009. 187 Section 2 of the Obscene Publications Act 1959. For present purposes, the relevant offences are those listed under paragraphs 2, 4, 5, 7, and 8 of Schedule 6 to the OSA. 188 Section 62 of the Coroners and Justice Act 2009. 189 Section 69 of the Serious Crime Act 2015. 190 Section 1 of the Protection of Children Act 1978 and section 160 of the Criminal Justice Act 1988. 191 Sections 52(1)(b); section 52(1)(d); section 52A of the Civic Government (Scotland) Act 1982. 192 Article 3 of the Protection of Children (NI) Order 1978 (S.I. 1978/1047 (N.I. 17)).
How CSEA offences manifest online¶
Scale of offending and scale of reporting¶
193 ONS, 2020. Child sexual abuse in England and Wales: year ending March 2019. [accessed 31 July 2023]. The IICSA also reported that research indicates that one in six girls and one in twenty boys are sexually abused before the age of 16Source: IICSA, 2022. : IICSA, 2022. The Report of the Independent Inquiry into Child Sexual Abuse. [accessed 2 October 2023]. 194 CSA Centre (Karsna, K. and Bromley, P.), 2023. Child sexual abuse in 2022/23: Trends in official data. [accessed 11 June 2024]. 195National Strategic Assessment of Serious and Organised Crime 2024. Child Sexual Abuse - National Crime Agency [accessed 20 September 2024] 196 The IICSA found that 79% of the participants in the Truth Project were aged 11 or under when they were first sexually abused: IICSA, 2022. The Report of the Independent Inquiry into Child Sexual Abuse. [accessed 31 July 2023]; ONS, 2020. Child sexual abuse in England and Wales: year ending March 2019. [accessed 31 July 2023].
197 The IICSA found that 79% of the participants in the Truth Project were aged 11 or under when they were first sexually abused: IICSA, 2022. The Report of the Independent Inquiry into Child Sexual Abuse. [accessed 31 July 2023]; ONS, 2020. Child sexual abuse in England and Wales: year ending March 2019. [accessed 31 July 2023]. 198 5,862 victims and survivors participated in the Truth Project: The IICSA, 2022. The Report of the Independent Inquiry into Child Sexual Abuse [accessed: 2 October 2023]. 199 ONS, 2020. Child sexual abuse in England and Wales: year ending March 2019. [accessed 31 July 2023]
200 EUROPOL, 2020. Exploiting Isolation: Offenders and victims of online child sexual abuse during COVID-19 pandemic. [accessed 31 August 2023]. 201 CSA Centre response to the November 2023 Illegal Harms Consultation. 202 NatCen (DeMarco, J., Sharrock, S., Crowther, T., and Barnard, M.), 2017. Behaviour and characteristics of perpetrators of online-facilitated child sexual abuse and exploitation: A Rapid Evidence Assessment Final Report. [accessed 22 September 2023]; CSA Centre response to the November 2023 Illegal Harms Consultation. 203 Babchishin, K., Hanson, R. & VanZuylen, H., 2014. Online Child Pornography Offenders are Different: A Meta-analysis of the Characteristics of Online and Offline Sex Offenders Against Children. Archives of sexual behaviour. 44. 204 For example, through sexually coerced financial gains or through so-called ‘invite child abuse pyramid’ sites which encourage the sharing of CSA sites to increase traffic to their site: IWF, 2023. The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023]. 205 VKPP 2022 National Analysis of Police-Recorded Child Sexual Abuse and Exploitation Crimes Report 2022? [accessed 25th September 2024].
206 We recognise that it is preferable in certain situations to refer instead to the behaviour rather than the individual (for example, 'person who sexually abused') but to account for the variety of evidence referring specifically to 'perpetrators' and the variety of different offences an individual may have committed, we have decided to use 'perpetrator' for simplicity. 207 An offence under section 13 of the Sexual Offences Act 2003 (child sex offences committed by children or young persons) or an offence under article 20 of the Sexual Offences (NI) Order 2008 (S.I. 2008/1769 (N.I. 2)) (child sex offences committed by children or young persons).
Cross cutting harms¶
2.22
How CSAM and grooming offending overlap¶
208 HM Government, 2021. Tackling Child Sexual Abuse Strategy 2021, page 86. [accessed 10 October 2024]. 209 Crown Prosecution Service, 2020. Indecent and Prohibited Images of Children. [accessed 18 August 2023]. 210 CEOP, year of publication unknown. What is sexual grooming? [accessed 24 September 2024]. 211 Please note that we have opted to use the word 'perpetrator' to describe an individual who commits any of the relevant offences discussed in this chapter. We recognise that it is preferable in certain situations to refer instead to the behaviour rather than the individual (for example, 'person who sexually abused') but to account for the variety of evidence referring specifically to 'perpetrators' and the variety of different offences an individual may have committed, we have decided to use 'perpetrator' for simplicity." 212 Contact abuse refers to instances where an abuser physically interacts with a child, such as through inappropriate touching or forcing the child to undress. Conversely, non-contact abuse, involves actions that do not require physical contact, such as encouraging the child to view sexual acts. NSPCC, 2024. Protecting children from sexual abuse. [accessed 25 September 2024]. 213 ‘First-generation’ or ‘novel’ CSAM refers to material that is newly generated and which has not been previously shared or re-shared; this is explored further in the CSAM section below. The relative volume of first-generation CSAM online compared to known CSAM is hard to determine as measures for the volume of CSAM circulating online necessarily rely on the identification of known CSAM (for example, via hash-matching). However, the dramatic recent increase in cases of financially motivated sextortion (see further in the document for more information), which occur after the creation of new first-generation material, suggests there is a significant volume of ‘novel’ CSAM circulating online.
214 IICSA (Senker, S., Scott, M. and Wainwright, L.), 2020. An explorative study on perpetrators of child sexual exploitation convicted alongside others. [accessed 22 September 2023]; Cale, J., Holt, T., Leclerc, B., Singh, S., & Drew, J., 2021. Crime commission processes in child sexual abuse material production and distribution: A systematic review. Trends and issues in crime and criminal justice, 617. [accessed 22 September 2023].
third) of surveyed individuals who had viewed CSAM tried to seek direct contact with a child afterwards.215
Increasingly, advanced technology has enabled a dramatic rise in the creation of Self-Generated Indecent Imagery (SGII)¶
215 Suojellaan Lapsia, Protect Children (Insoll, T., Ovaska, A., and Vaaranen-Valkonen, N.), 2021. CSAM Users in the dark web: Protecting children through prevention. [accessed 25 August 2023]. 216 Ofcom is aware of other terms used to describe self-generated indecent imagery (SGII), such as ’youth produced sexual imagery’ and colloquial terms such as ‘sexting’ or ‘sharing nudes’. Ofcom has used the term SGII in this statement as the one currently most used within the online child protection system, acknowledging that there is work ongoing to consider alternative descriptors. 217 The IWF found that of the 275,652 webpages it acted on during 2023, 92% were assessed as containing SGII: IWF, 2023 'Self-generated' child sexual abuse [accessed 24 September 2024]. 218 It is crucial for services to note that consensual SGII is typically created by children, and sent to other children, and although the image itself must be treated as illegal and removed and reported to the appropriate designated body (See ICJG - Volume 5, Chapter 26), it is important that services are aware of the nuances associated with this type of behaviour. There are contextual factors in the creation and uploading of the image that require a wider societal response including adequate policing response and education.
219 Brook and CEOP (McGeeney, E. and Hanson, E.) 2017. Digital Romance: A research project exploring young people’s use of technology in their romantic relationships and love lives.; A UK national survey of 14,944 children and young people found that 17% of 15 to 17-year-olds had shared a sexual image: Internet Matters (Katz, A. and El Asam, A.), 2020. Look at Me: Teens, sexting and risks. [accessed 18 August 2023]; Thorn, 2022. Online Grooming: Examining risky encounters amid everyday digital socialization. Findings from 2021 qualitative and quantitative research among 9-17-year-olds. 220 Revealing Reality, 2023. Anti-social Media: The violent, sexual and illegal content children are viewing on one of their most popular apps. [accessed 31 August 2023].
images of themselves and/or others. Some perpetrators will coerce children into producing SGII and use the SGII to blackmail children into sharing further SGII, sometimes inciting the child to abuse friends and siblings by threatening to publish the SGII online or send the images to friends and family. Some perpetrators also use SGII obtained through deceit and coercion, using the same form of threats, to blackmail children for financial gain. This is known as financially motivated sexual extortion (FMSE, or ‘sextortion’).
Financially motivated sexual extortion¶
221 NCA, 2024. NCA issues urgent warning about ‘sextortion’ [accessed 24 October 2024] 222 A US-based non-for-profit organisation which works to help find missing children, reduce child sexual exploitation, and prevent child victimisation.
223 NSPCC, 2024. Young people’s experiences of online sexual extortion or ‘sextortion'. [accessed 26 September 2024] 224 BBC (Tidy, J.), 2024. Dead in 6 hours: How Nigerian sextortion scammers targeted my son, 09 June. [accessed 24 October 2024]. 225 Brooks, L. and Milmo, D. 2024. National Crime Agency threatens extraditions over rise in sextortion cases, The Guardian, 22 August. [accessed 24 October 2024] 226 NCA, 2024. NCA issues urgent warning about ‘sextortion’. [accessed 24 October 2024] 227 NCA, 2024.
Other egregious harms¶
228 As well as encouraging self-harm, suicide, and animal torture, these groups also encourage children to victimise others, and stab and ‘cut sign’ themselves, whereby a victim carves their abuser’s name into their body. We have summarised the activities of these groups in this chapter, but the activities they are engaged in cut across various kinds of illegal harm with separate chapters in the Register of Risks. See ‘Encouraging and assisting suicide’; ‘Encouraging and assisting serious self-harm’; ‘Animal cruelty’; ‘Human and animal torture’ and ‘Terrorism’ chapters for how these kinds of offences tend to manifest online. 229 FBI, 2023. Violent Online Groups Extort Minors to Self-Harm and Produce Child Sexual Abuse Material. [accessed 2 November 2024]. 230 RCMP, 2024. RCMP reminds Canadians about violent online groups targeting youth. [accessed 25 October 2024]. 231 The action or practice of making a prank call to police or emergency services in an attempt bring about the dispatch of armed police officers such as a SWAT team to a particular address. 232 The action of obtaining and publishing personally identifiable information (PII) on the internet, usually for malicious intent.
233 FBI, 2023. Violent Online Groups Extort Minors to Self-Harm and Produce Child Sexual Abuse Material. [accessed 2 November 2024]. 234 FBI, 2023. 235 GNET (Argentino, M., Barrett G., and Tyler, M. B.), 2024. 764: The Intersection of Terrorism, Violent Extremism, and Child Sexual Exploitation. [access 2 November 2024]. 236 RCMP, 2024. RCMP reminds Canadians about violent online groups targeting youth. [accessed 25 October 2024] 237 GNET (Argentino, M., Barrett G., and Tyler, M. B.), 2024. 764: The Intersection of Terrorism, Violent Extremism, and Child Sexual Exploitation. [access 2 November 2024].
Some technologies and functionalities are linked to particular kinds of CSEA¶
Livestreaming¶
Generative Artificial Intelligence (Gen AI)¶
238 The Royal Canadian Mounted Police reports that one such group targeting children is commonly known as the 764 network (or "the com") but goes by various monikers: RCMP, 2024. RCMP reminds Canadians about violent online groups targeting youth. [accessed 25 October 2024]. 239 WeProtect outlines that the livestreaming of CSEA exists in two main forms: (i) It can be “livestreaming an act of child sexual exploitation and abuse happening offline”, or (ii) it can be “one or more children being forced into ‘performing’ sexual acts in front of a webcam (or camera). This can often be in exchange for payment.”: WeProtect. Livestreaming child sexual exploitation and abuse.[accessed 25 August 2023].
240 Interpol, 2020. Threats and trends Child sexual exploitation and abuse: covid-19 impact. [accessed 22 September 2023]. 241 WeProtect, n.d. Live Streaming Child Exploitation and Abuse. [accessed 22 September 2023]. 242 International Justice Mission and University of Nottingham Rights Lab, 2023. Scale of Harm Research Method, Findings, and Recommendations: Estimating the Prevalence of Trafficking to Produce Child Sexual Exploitation Material in the Philippines. [accessed 12 June 2024].
Virtual Reality and Augmented Reality¶
e The IWF have reported that most AI CSAM is now realistic enough to be treated as ‘real’ CSAM, with the most convincing AI CSAM being visually indistinguishable from real CSAM, even for trained IWF analysts: IWF, 2023. How AI is being abused to create child sexual abuse imagery. [accessed 11 June 2024]; In a one-month period between September and October 2023, the IWF were able to scrape 20,254 AI generated images from one dark web forum. Of the 11,108 images assessed using human review, 2,978 images were illegal either under the Protection of Children Act (1978) or the Coroners and Justice Act (2009). While the dark web is not the focus of this analysis, it is reasonable to assume that some of this imagery may eventually be shared to the clear web: Internet Watch Foundation (IWF) response to November 2023 Illegal Harms Consultation. 244 NCMEC, 2023. What does generative AI mean for CSE? [accessed 12 June 2024]. 245 Hedgecoe, G., 2023. AI-generated naked child images shock Spanish town of Almendralejo. The BBC, 24 September. [accessed 12 June 2024]. 246 IWF, 2023. How AI is being abused to create child sexual abuse imagery. [accessed 11 June 2024]; Active Fence, 2023. How predators are abusing generative AI. [accessed 12 June 2024].
247 Thiel, D., 2023. Investigation Finds AI Image Generation Models Trained on Child Abuse. Stanford University, 20 December. [accessed 12 June 2024]. 248 Note that, at the time of writing, although this specific activity would not constitute an offence covered by Schedule 6 of the OSA, proposed amendments to the Criminal Justice Bill include a new Clause 26 to make it an offence to use, create or share online digital tools which simulate the offence of sexual communication with a child: House of Commons, 2024. Criminal Justice Bill, As Amended (Amendment Paper). 249 IWF, 2024. Briefing from the Internet Watch Foundation: Criminal Justice Bill Report Stage. [accessed 15 November 2024]
and simulated abuse.250 Integrated technologies like haptics, which simulate real world sensations such as movements, vibrations, and force, can make virtual abuse feel more realistic. Further, research by the NSPCC indicates that perpetrators often find new ways to exploit immersive technologies to groom and exploit children.251 The harm posed by VR and AR to these environments but can easily escalate to offline offences.
250 Risks include opportunities for offenders to access victim-survivors; to distribute CSAM; simulate abuse of virtual representations of children; and use integrated tech such as haptics, which simulate real world sensations such as movements, vibrations, and force. Source: WeProtect and the University of Manchester, 2023. Extended Reality technologies and child sexual exploitation and abuse. [accessed 12 June 2024]. 251 NSPCC (Allen, C.), 2023. Child Safeguarding and Immersive Technologies: An Outline of the Risks. [accessed 21 November 2024]. 252 NSPCC (Hamilton-Giachritsis, C., Hanson, E., Whittle, H. and Beech, A.), 2017. “Everyone deserves to be happy and safe”. A mixed methods study exploring how online and offline child sexual abuse impact young people and how professionals respond to it. [accessed 11 June 2024].
253 5,862 victims and survivors participated in the Truth Project. Source: IICSA, 2022. The Report of the Independent Inquiry into Child Sexual Abuse. [accessed 22 September 2023]. 254 Maniglio, R., 2009. The Impact of Child sexual abuse on health: a systematic review of reviews. Clinical Psychology Review. 29(7), pp.647 – 657; Hailes, H.P., Yu, R., Danese, A., Fazel, S., 2019. Long-term outcomes of childhood sexual abuse: an umbrella review. The Lancet Psychiatry. 6(10), p.830 – 839. 255 CSA Centre response to November 2023 Illegal Harms Consultation. 256 CSA Centre response to November 2023 Illegal Harms Consultation.
2A Grooming¶
Warning: This chapter contains content that may be upsetting or distressing in relation to grooming. Summary of analysis for grooming offence: how harms manifests online and risk factors This chapter explores the evidence on online grooming for the purpose of conducting child sexual abuse. Grooming is the process of building a relationship or emotional connection with a child or young person so they can manipulate, exploit, and abuse them.257 Grooming is a complicated process that does not follow a set pattern. Sometimes it can take weeks or months, or it can happen quickly after the first contact. Grooming can involve many stages, and is not always limited to sexual conversations, although it can involve coercing or manipulating children into multiple sexual acts. The scale of online grooming is extensive, with the NSPCC reporting over 34,000 cases of online grooming crimes against children between 2017 and 2023.258Grooming will affect each victim and survivor differently, but the effects are serious and can often last a lifetime. These include negative psychological effects such as self-harm, loss of confidence, increased aggression and feelings of self-blame, and difficulties trusting others. Service type risk factors: Perpetrators can groom children in a many different ways, and through a variety of different services. Social media, video sharing services, gaming services, and messaging services are commonly used to carry out this offence. Other service types that children use, such as discussion forums and chat rooms are also involved in grooming cases. Groomers can ‘platform-hop’ between services to exploit different functionalities, enabling them to meet children, engage in grooming conversations and move interactions with children to more private spaces to avoid detection. User base risk factors: For grooming offences, a high-risk factor is the age of users. Groomers seeking to target children are more likely to be drawn to services that children use. While grooming can occur on services of any size, those with a larger number or higher proportion of child users may offer more incentive for perpetrators during the early stages of grooming, such as identifying and making contact with children. This factor may become less important once contact is made. In the later stages of grooming, a service’s perceived lack of detection technology may be a more257 HM Government,2021. Tackling Child Sexual Abuse Strategy 2021. [accessed 14 November 2024]. 258 IICSA, 2020. The Internet: Investigation Report. [accessed 31 August 2023].
important risk factor. However, a child’s ability to access a service will remain a constant factor, given the nature of the offence. Perpetrators assess a child’s personal circumstances to identify traits that may make them more vulnerable to online grooming, such as low self-esteem or lack of supervision. Other factors that can increase vulnerability include identifying as LGBTQIA +, having a disability, mental/physical health and/or socio-economic status. Gender also appears to be a risk factor, with evidence showing that girls are significantly more likely to be groomed than boys, although male victims are believed to be under-reported. Importantly, none of these factors are the reason a child is sexually exploited. Rather, it is the way these circumstances can leave children more isolated or dependent on others, that makes them more susceptible to online grooming. Functionalities and recommender systems risk factors: Functionalities that allow abusers to identify and contact children are risk factors in enabling grooming offences. Perpetrators can use information on user profiles to find and target victims and survivors, initiating the grooming process. The ability to create fake user profiles allows them to misrepresent themselves by displaying a false age, name, or location. Studies also show perpetrators using anonymous user profiles to contact children. User connections allow perpetrators to establish contact with children and start communication, while the sense of trust that mutual connections can create may also be exploited. Perpetrators may exploit network recommender systems, like publicly visible friends lists, to quickly infiltrate groups of children and use this access to blackmail and coerce them into further abuse. User groups can be used to target children, particularly groups for adolescents that discuss sexual themes. Direct messaging enables perpetrators to make contact with children and potentially develop relationships through frequent communication, often away from public view. While ephemeral and encrypted messaging makes proving and detecting perpetrators’ contact with children challenging. Network recommender systems can facilitate grooming by suggesting adult users to children and vice versa. Additionally, users and groups associated with child users may be suggested to perpetrators, based on their past activity and connections. Grooming can also often include coercing or manipulating a child into performing sexual acts over livestreams. Perpetrators can also use comments on posted or livestreamed content to build rapport and exchange contact details. Other functionalities contribute to these offences. For example, sending images through messaging functionalities can be used to persuade children to share self-generated indecent imagery (SGII). Visual media editing functionalities can also be used to disguise the identity of perpetrators when they are contacting a child. The ability to post or send location information can allow a perpetrator details needed
to physically approach their target, to gain a child’s trust by claiming common connections, or to use this knowledge to intimidate and threaten the child.
How grooming offences manifest online¶
2A.1 This section is an overview which looks at how the specified grooming offences manifest online, and how users may be at risk of harm.
Definition and Scale¶
2A.2 Grooming offences can include, but are not limited to, sexual or indecent communications with children, engaging in sexual activity with a child, or in the presence of a child, or coercing or inciting the child into sexual activity. The aim of online grooming is to manipulate or coerce children into engaging in sexual activity. This may occur solely online, through coercing children into producing and sharing self-generated indecent imagery (CSAM), or perpetrators may persuade children to meet them in person to sexually abuse the child.259 2A.3 The grooming process necessarily involves at least two components: identifying a child and contacting a child. Therefore, the presence of children on a service is a necessary initial enabler of grooming. While identifying a child may not constitute an offence, it is a crucial step towards committing grooming offences. The evidence presented here addresses both stages. 2A.4 The scale of online grooming is widespread, with IICSA describing it to be “of real and significant concern”.260 In 2023, the National Society for the Prevention of Cruelty to Children (NSPCC) reported nearly 34,000 recorded online grooming crimes against children over the past six years.261 A US study of undergraduates found that 17% (nearly 1 in 5) had experienced sexual solicitation as youths from adults they had chatted with online, and 23% (nearly 1 in 4) recalled a long intimate conversation with an adult stranger, potentially indicating online grooming.262 2A.5 Unsolicited sexual messages that children receive from unknown users is also an indicator of online grooming. Ofcom research found that 13% (more than 1 in 10) of 11 to 18-year-olds had received pictures or videos of naked or half-dressed people when communicating online, and 10% had been asked to share an intimate picture or video of themselves.263 A European study found that 68% (more than 3 in 5) of the 18-year-olds surveyed had experienced at least one sexual harm online during childhood, with 55% (more than half) being asked to engage online in sexually explicit activities they were uncomfortable with, or did not want to do.264
259 Child Exploitation and Online Protection Centre (CEOP), 2022. What is sexual grooming?. [accessed 14 November2024]. s 260 IICSA, 2020. The Internet: Investigation Report. [accessed 31 August 2023]. 261 Analysis of Freedom of Information requests sent to UK police forces: NSPCC, 2024. 82% rise in online grooming crimes against children in the last 5 years. [accessed 11 June 2024].
262 The study was of 1,133 undergraduate college students at two public institutions in the United States and asked about their experiences when under 18. Greene-Colozzi, E., Winters, G., Blasko, B. and Jeglic, E., 2020. Experiences and Perceptions of Online Sexual Solicitation and Grooming of Minors. A Retrospective Report. Journal of Sexual Abuse, 29:7, 836-854. 263 Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research. [accessed 14 November 2024]. 264 The study also found that 56% had received sexually explicit content from an adult they know or did not know: WeProtect, 2023. Estimates of childhood exposure to online sexual harms and their risk factors. [accessed 12 June 2024].
2A.6 There is evidence that incidences of grooming online are increasing. The NSPCC reported that more than 7,000 Sexual Communication with a Child offences were recorded by Police in 2023/24, an 89% increase since this offence came into force in 2017/18.265 2A.7 The scale of online grooming is likely underestimated due to complexities in how it presents, how it is experienced and challenges in identifying and reporting it, including the cross-border dimensions of offending where perpetrators and victims are often based in different countries. Such cases may be represented in these Police statistics, where perpetrators may not be UK citizens. Research has shown that grooming is significantly under-reported by victims and survivors for many reasons including shame, fear, and the lack of recognition that a crime has occurred. 266 Where it is reported, it is often many years after the abuse occurred.267 2A.8 Grooming is a behavioural offence that can take many different forms online and does not follow a set pathway. However, existing literature has identified some common patterns that occur in most cases. Children can be groomed to send CSAM and into being sexually abused online and/or offline through a single message, or over a prolonged period. Grooming can involve various stages, including forming friendships, flattery, developing trust, risk assessment, exclusivity, threats, and sexual conversations.268 While these stages are not fixed, and can often occur rapidly and/or interchangeably, they indicate crucial points where interventions could potentially disrupt an offender’s progression. The resulting abuse can be one-off, but can also often be sustained over a long period of time, with multiple offences being committed, and the abuse in some instances becoming increasingly severe. 2A.9 Grooming pathways often occur in stages which can take place on multiple different services.269 Many perpetrators try to move children to more private online spaces to continue grooming, using functionalities such as private chat, end-to-end encryption, as well as image sharing, to carry out sexual communication.270 Children are at risk at various points along the pathway, and different functionalities may present greater risks at different
265 While these figures indicate an increase in grooming attempts, they reflect, at least in part, an increased national policing response to grooming, and so an increase in the detection and recording of offences that may already have been occurring in previous years. NSPCC, 2024. Online grooming crimes against children increase by 89% in six years. [accessed 14 November 2024]. 266 Quayle, E., Jonsson, L., Lööf, L., 2012. Online behaviour related to child sexual abuse. Interviews with affected young people. Council of the Baltic Sea States, Stockholm: ROBERT project. [accessed 31 August 2023]; Katz, C., Piller, S., Glucklich, T., & Matty, D. E., 2021. “Stop Waking the Dead”: Internet Child Sexual Abuse and Perspectives on Its Disclosure. Journal of Interpersonal Violence, 36(9–10), NP5084–NP5104. [accessed 31 August 2023].
268 Whittle, H. C., Hamilton-Giachritsis, E. and Beech, A. R., 2015. A comparison of victim and offender perspectives of grooming and sexual abuse, Deviant behaviour, 36 (7), pp.539-564; Borj P., Raja, K., & Bours, P., 2023. Online Grooming detection: A comprehensive survey of child exploitation in chat logs. Journal of Knowledge Based Systems, 259, 110039. 269 The services used by perpetrators will differ according to the stage of the grooming journey. For example, the type of service where a perpetrator might seek to identify a potential child to groom may be a different from the one where they seek to exchange images with the child. Movement through the stages is often driven by the preparator’s motives.
270 Ofcom research found that 20% of 11-18-year-olds said they had communicated on more than one platform with the person with whom they had had their most recent potentially uncomfortable online contact experience. This ‘uncomfortable experience’ may not be grooming per se, but may be intimate image sharing, rude/abusive messages or being asked for personal information. The research also found that 10% of 11–18-year-olds said they had ever been asked to move their online conversation to another service by someone they did not know well or did not know at all: Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research; Ringenberg, T.R., Seigfried-Spellar, K.C., Rayz, J. M., and Rogers, M.K., 2022. A scoping review of child grooming strategies: Pre-and post-internet, Child Abuse & Neglect, 123, Article 105392.
stages. Thus, services must understand at what point in the grooming process their service could be exploited, to enable the implementation of appropriate safeguards.
2A.10 Some grooming perpetrators use online services to target large numbers of children, and one method deployed is the ‘scatter gun’ or ‘pyramid approach’. This is where perpetrators engage with many children (sometimes hundreds), mostly unknown to them, in quick succession. In these instances, the desired outcome for the perpetrator is to obtain a response from a proportion of the targeted children to engage them in conversation, thereby beginning the grooming process.271 This leads to a rapid escalation of the harm, with offences such as sexual communication sometimes being committed within minutes of the perpetrator and child making contact.272 2A.11 Other techniques used by perpetrators include relationship-building, such as ‘the boyfriend model, where the course of the interaction can last for days or even years’.273 Some perpetrators impersonate other young people or create fake online user profiles to build relationships with children and obscure their identity, using flattery or shared interests. In other cases, perpetrators may be truthful about who they are. Blackmail may also be used, which can make it difficult for the child to break contact with perpetrators of grooming or other CSEA offences online.274
Risks of harm to individuals presented by online grooming offences¶
2A.12 Grooming will affect each victim and survivor differently, but the effects are significant and long lasting. 2A.13 Grooming can lead to a range of negative psychological impacts. Qualitative studies based on interviews with victims and survivors of online grooming have reported impacts including self-harm, loss of confidence, aggression, and problems trusting others.275 2A.14 Feelings of self-blame are common as perpetrators often use manipulation and coercion to pressure victims into sending sexual material. This increases victims’ and survivors’ reluctance to disclose instances of online grooming.276 The cycle of abuse can also be escalated, with perpetrators encouraging the child to include other children, objects, or
271 Joleby, M., Lunde, C., Landström, Jonsson, L. S. 2021. Offender strategies for engaging children in online sexual activity, Child Abuse & Neglect, 120. [accessed 4 September 2023]. 272 Lorenzo-Dus, N., Izura, C., and Pérez-Tattam, R., 2016. Understanding grooming discourse in computer-mediated environments, Discourse, Context & Media, 12, pp.40-50. [Note: this research involves the analysis of chat logs between perpetrators and adults posing as children. These may not be truly reflective of interactions between children and perpetrators.] 273 Barnardo’s, 2017. Working with children who are victims or at risk of sexual exploitation: Barnardo’s model of practice. [accessed 10 August 2023]. 274 Hanson, E, 2017. The Impact of Online Sexual Abuse on Children and Young People: Impact, Protection and Prevention. in (2017) Online Risk to Children: Impact, protection and prevention (First Edition ed.), Blackwell, John Wiley & Sons, pp.98-122.
275 Whittle, H., Hamiliton-Giachritsis, C.& Beech, A., 2013. Victims’ Voices: The Impact of Online Grooming and Sexual Abuse. Universal Journal of Psychology 1(2), pp.59-71. [accessed 18 November 2024]. Ofcom, 2024. Online communications among children and young people: Qualitative research exploring experiences of sexualised messages online. 276 Hanson, E, 2017. The Impact of Online Sexual Abuse on Children and Young People: Impact, Protection and Prevention. in (2017) Online Risk to Children: Impact, protection and prevention (First Edition ed.), Blackwell, John Wiley & Sons, pp.98- 122.
animals in the sexual abuse, deepening the child’s complex feelings of shame and culpability.277
2A.15 Grooming can also affect victims’ and survivors’ attitudes towards being online. One study found that, rather than viewing the internet as a place of opportunity, victims’ and survivors' attitudes shifted towards a more negative view.278
Evidence of risk factors on user-to-user services¶
2A.16 We consider that the risk factors below are likely to increase the risk of harm relating to grooming. These risk factors are summarised in the grey box at the start of the chapter.
Risk factors: Service types¶
2A.17 Although grooming offences can happen on various services, research shows that the following service types are commonly used to facilitate or commit grooming offences:
discussion forums and chat room services, social media and video-sharing services, private¶
messaging services and gaming services.
Discussion forums and chat room services¶
2A.18 There is a range of evidence that discussion forum and chat room services are used to identify and establish contact with a child, before conversation moves to a service with more privacy, such as a messaging service. The NSPCC noted that in reports of online grooming by children, conversations were said to start in a “public online space such as a forum or group chat” before “becoming private”.279 This is supported by evidence from Internet Matters, which showed that online groomers may strike up a relationship with a child through discussion forums before asking them to move to another service to talk privately.280 2A.19 Research suggests that young people who struggle to form friendships and relationships offline compensate by seeking online interactions, including in in chat rooms.281 It is reasonable to assume that such an individual would be more vulnerable to approaches from an adult seeking to groom a child.
Social media services and video-sharing services¶
2A.20 Online grooming has been shown to take place on social media and video sharing services. In 2023, the NSPCC reported that, of the 34,000 online grooming crimes against children recorded in the last 6 years, where the means of communication was known, 26% took place on Snapchat and 47% took place on Meta-owned products such as Facebook and Instagram.282 Protect Children also found that 48% of respondents who had sought contact with a child online first did so on social media; this was the most popular approach identified.283
277 IICSA, 2020. The Internet: Investigation Report. [accessed 22 September 2023]. 278 Chiu, J. & Quayle, E., 2022. Understanding online grooming: An interpretative phenomenological analysis of adolescents’ offline meetings with adult perpetrators. Child Abuse & Neglect, 128. [accessed 18 November 2024]
279 NSPCC, 2020. The impact of the coronavirus pandemic on child welfare: online abuse. [accessed 10 August 2023]. 280 Internet.matters.org, n.d. Learn about online grooming. [accessed 10 August 2023]. 281 Wolak, J., Finkelhor, D., Mitchell, K. J., and Ybarra, M. L., 2008. Online ‘Predators’ and their victims, American Psychologist, 63(2) pp.111-128. [accessed 18 November 2024] 282 NSPCC, 2024. 82% rise in online grooming crimes against children in the last 5 years. [accessed 11 June 2024]. 283 The survey was conducted among individuals searching for CSAM on dark web search engines: Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024].
2A.21 Perpetrators may intentionally use social media and video sharing services to find and communicate with children, as a high proportion of children often use these services, an Ofcom report found that 60% of all children aged 3 to 17 game online.284 WeProtect described how many perpetrators who attempt to groom children online first identify targets on social media before moving conversations to a more private space.285 2A.22 Perpetrators may also use social media services and video-sharing services due to functionalities that allow for quick and easy forming of many user connections (and subsequent communication), enabling techniques such as the ‘scatter gun’ or ‘pyramid approach’. Some social media and video sharing services encourage new connections (see network recommender systems for more information), which perpetrators are known to exploit to quickly initiate contact with a child or children. 2A.23 Perpetrators may use social media services and video-sharing services to identify children, making use of live streaming functionalities (see live streaming for more information). Reports of online grooming made to the NSPCC mention children being approached on ‘social media networks’ and ‘livestreaming platforms’.286
Messaging services¶
2A.24 There is evidence that messaging services are online spaces where perpetrators initiate conversations with children. Protect Children found that 37% of respondents who had sought contact with a child online first did so on a messaging app.287 Data from the Office for National Statistics (ONS) has also shown that 74% of approaches to children online by someone they do not know first take place via messaging services.288 In research from the NSPCC exploring the growth of online grooming reported during the COVID-19 pandemic, ‘instant messaging apps’ played a role – either as the first service used to contact a child, or in conjunction with other services used to communicate – in many of the reports where children discussed how the perpetrators had built relationships with them.289 2A.25 There is also evidence to show that perpetrators seek to move their communication with children to private messaging services, after initiating the conversation in a more public online space. A US study of online grooming by Thorn in 2022 found that 65% of children surveyed reported having an online-only contact invite them to move from a public chat into a private conversation on a different platform.290 It noted that “private messaging apps warrant unique consideration for the role they play in meeting people online and how these relationships deepen for minors”.291
284 Ofcom, 2024. Children and Parents: Media Use and Attitudes Report. [accessed 24th September 2024]. 285 WeProtect, 2023. Global Threat Assessment 2023: Assessing the scale and scope of child sexual exploitation and abuse online, to transform the response. [accessed 12 June 2024]. 286 NSPCC, 2020. The impact of the coronavirus pandemic on child welfare: online abuse. [accessed 22 September 2023]. 287 The survey was conducted among individuals searching for CSAM on dark web search engines: Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 288 NSPCC response to November 2023 Illegal Harms Consultation.
289 NSPCC, 2020. 290 Half of the children surveyed (52%) reported having used a private messaging service to interact with an online-only contact, including 23% of the 9-12-year-olds, who had had daily interactions with an online-only contact using a private messaging service. 445 9–12-year-olds and 755 13- to 17-year-olds in the United States were surveyed. From this study it is not possible to infer that these were adult-led conversations, nor that grooming was involved: Thorn, 2022. Online Grooming: Examining risky encounters amid everyday digital socialization. [accessed 10 August 2023]. 291 Thorn, 2022. Online Grooming: Examining risky encounters amid everyday digital socialization. [accessed 10 August 2023].
2A.26 Perpetrators may seek to move their communication with children to messaging services to exploit the enhanced privacy of end-to-end encryption. WeProtect reported that many perpetrators first identify targets on social media, in chat rooms or in gaming environments before moving conversations to a private messaging app or an end-to-end encrypted environment to reduce the risk of detection.292 For more information, see ‘encrypted messaging’ in the User communications sub-section.
Gaming services¶
2A.27 Evidence shows that gaming services are used by perpetrators to identify and establish contact with children. Protect Children found that 41% of respondents who had sought contact with a child online first did so via an online game.293 NSPCC research on online grooming also highlights “voice or text chat services built into online multiplayer games” as methods used by perpetrators to approach children.294 2A.28 Gaming services may pose a particular risk because they are online spaces where communication with strangers is normalised, and often a core part of the gaming process, which may make children more vulnerable to approaches from perpetrators. Protect Children also noted that it is easier for perpetrators to hide their real identity in gaming, where many users do not reveal personal information.295 2A.29 CSEA professionals have also highlighted that gaming services can be exploited by those seeking to groom children, by using such services’ features to gain contact and establish trust with them online.296 Perpetrators have used in-game gifts and trades as a manipulation and coercion tactic, and instances of sexualised language and grooming have been observed in multi-player games.297
Risk factors: User base¶
User base size¶
2A.30 Both large and small user bases can pose risks for online grooming. On a large service, a perpetrator may use a scatter-gun approach, randomly targeting many due to the wider pool of potential victims. Services with a smaller user base may enable perpetrators to identify a victim more easily with specific characteristics or vulnerabilities.
User base demographics¶
nature and extent of the grooming threat is incomplete due to under-detection and under-reporting of this crime; children may not report due to a lack of awareness that they have
292 WeProtect, 2023. Global Threat Assessment 2023: Assessing the scale and scope of child sexual exploitation and abuse online, to transform the response. [accessed 12 June 2024]. 293 The survey was conducted among individuals searching for CSAM on dark web search engines: Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024].
294 NSPCC, 2020. The impact of the coronavirus pandemic on child welfare: online abuse. [accessed 22 September 2023]. 295 Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 296 Interpol, 2020. Threats and Trends Child sexual exploitation and abuse: covid-19 impact. [accessed 25 August 2023]. 297 5Rights, 2020. Risky by Design. [accessed 26 September 2023]; Hamilton-Giachritsis, C., Hanson, E., Helen, W., Alves- Costa, F., and Beech, A., 2020. Technology assisted child sexual abuse in the UK: Young people’s views on the impact of online sexual abuse. Children and Youth Services Review, 119.; Stonehouse, R., 2019. Roblox: 'I thought he was playing an innocent game', The BBC, 30 May. [accessed 2 October 2023].
been groomed, or due to feelings of fear, shame, or considering that they will not be believed. 298
2A.32 However, the available data suggests that user base characteristics including age, gender, disability, sexual orientation and gender identity and media literacy could lead to an increased risk of harm to individuals.
Age¶
2A.33 Age is an important risk factor in grooming offences. Perpetrators seeking to target children are drawn to services that children use.299 2A.34 There is evidence to suggest that children above the age of 13 may be at greater risk of being contacted by strangers than younger children, possibly due to the increasing ownership of phones and social media accounts around this age.300 Internet Matters found that the proportion of children contacted by strangers online rises from 18% of 11 to 12-year-olds, to 25% of 13 to 14-year-olds.301 2A.35 Nonetheless, younger children are at risk of online grooming. In 2023, the NSPCC reported that out of the 34,000 online grooming crimes recorded over the six years prior, 5,500 involved primary-aged school children as targets.302
Gender¶
2A.36 Girls have been shown to be at greater risk of experiencing grooming online. In 2023, the NSPCC reported that 83% of recorded online grooming offences over the past six years, where gender was known, were against girls.303 Ofcom research found that girls aged 16-18 were more likely than other groups to have encountered all ten of the potentially uncomfortable experiences asked about. 304 2A.37 Boys are less likely to report sexual abuse, often due to the perceived social stigma surrounding this type of crime, which can impact the accuracy of gender-related data. Evidence also suggests the type of sexual abuse experienced may vary by gender. A number of sources have found that a large portion of reports involving SGII derived from financially motivated sexual extortion (FMSE) involve boys.305 Analysis by the Canadian Centre for Child
298 Katz, C., Piller, S., Glucklich, T., & Matty, D. E., 2021. “Stop Waking the Dead”: Internet Child Sexual Abuse and Perspectives on Its Disclosure. Journal of Interpersonal Violence, 36(9–10), NP5084–NP5104. [accessed 31 August 2023]. 299 Kloess, J. A., Hamilton-Giachritsis, C. E. and Beech, A. R., 2019. Offence Processes of online sexual grooming and abuse of children via internet communication platforms, Sexual Abuse, 31(1), pp.73-96. [accessed 18 November 2024]. 300 An Ofcom report found that 95% of children aged 12-15 have smartphones, while 59% of 8–11-year-olds do: Ofcom, 2024. Ofcom children and parents media literacy. [accessed 25 September 2024]. 301 Internet Matters response to November 2023 Illegal Harms Consultation. 302 NSPCC, 2024. 82% rise in online grooming crimes against children in the last 5 years. [accessed 11 June 2024]. 303 Whittle et al. (2013) found that girls may be twice as likely to be groomed: Whittle, H. C., Hamilton-Giachritsis, C., Beech, A., and Collings, G., 2013. A Review of young people’s vulnerabilities to online grooming, Aggression and Violent Behavior, 18, pp.135-146; NSPCC, 2024. 82% rise in online grooming crimes against children in the last 5 years. [accessed 11 June 2024]. 304 24% of girls aged 16-18 had been asked to share intimate images or videos, compared to the average of 10% for all boys and girls aged 11-18. The uncomfortable experiences we asked about in the survey were: an unwanted friend or follow request; asked to share naked or half-dressed pictures or videos; asked to share personal information; a friend request from someone pretending to be someone else; pictures or videos of naked or half-dressed people; abusive, nasty, or rude messages, voice note or comments, asked to video call/chat with someone you have not spoken to before; asked to move your chat to a different app or platform by someone you don’t know well or don’t know at all; added to a group chat which includes people you don’t know well or don’t know at all; added to a group video call which includes people you don’t know. Source: Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research.
305 FBI National Press Office, 2022. FBI and Partners Issue National Public Safety Alert on Financial Sextortion Schemes. [accessed 22 September 2023]; IWF, 2023. The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023].
Protection (C3P) into FMSE found that, in cases where the victim’s gender was known, 98% of financial sextortion victims and survivors were male.306
Disability¶
2A.38 Neurodivergent children, and those with disabilities such as learning difficulties, could be more vulnerable to online grooming. Ofcom research also found that those aged 11 to 18 with limiting or affecting conditions were more likely than those without such conditions to report potentially uncomfortable experiences asked about in the research, including being asked to share intimate images or being sent intimate images.307 WeProtect also found that children with disabilities were more likely to experience online sexual harms in their childhood.308 Whilst not specific to online harm, the CSA Centre found that disabled participants were twice as likely as non-disabled participants to describe experiences of CSA.309 Professionals within charities working with children and young people who have experienced grooming and sexual abuse also highlight that children with special educational needs and disabilities (SEND) are considered to be at higher risk of engaging and responding to sexualised messages from adults.310
Sexual orientation and gender identity¶
2A.39 Evidence suggests that LGBTQIA+ children are at greater risk of online grooming, as they are potentially more likely to seek relationships online if they feel that they have little opportunity to explore their sexual orientation or gender identity offline. WeProtect found that respondents who identified as transgender/non-binary (59%) or LGBQ+ (65%) were more likely to experience online sexual harms in their childhood than those who identified as cisgender (47%) or non-LGBQ+ (46%).311 Ofcom research found that those aged 11 to 18 who identified as LGBTQ+ were more likely than those who identified as heterosexual or cisgendered to encounter potentially uncomfortable experiences online.312 Furthermore, research from US-based charity, Thorn, found that 32% (nearly one-third) of LGBTQ+ participants reported an online sexual interaction with someone they believed to be over 18, compared with 22% of non-LGBTQ+ participants.313 2A.40 There is also evidence to suggest LGBTQ+ victims and survivors face distinct barriers to disclosing and reporting child sexual abuse, leading to potential under-representation in reported cases. The IICSA found that reasons for under-reporting of abuse by LGBTQIA+ victims and survivors included internalised prejudice and stigma, and for gay and bisexual men, the continuing trauma of past criminalisation of homosexuality. 314
306 C3P, 2022. An analysis of financial sextortion victim posts published on R/Sextortion. [accessed 2 August 2023]. 307 Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research. [accessed 18 November 2024]. 308 WeProtect, 2021. Global Threat Assessment 2021. [accessed 22 September 2023]. 309 CSA Centre (Karsna, K. and Kelly, L.), 2021, The scale and nature of child sexual abuse: Review of evidence. [accessed 20 June 2024]. 310 Ofcom, 2024. Online communications among children and young people: Qualitative research exploring experiences of sexualised messages online 311 WeProtect, 2021. Global Threat Assessment 2021. [accessed 22 September 2023].
312 This included being asked to share intimate images or being sent intimate images: Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research. [accessed 18 November 2024]. 313 Thorn and Benenson Strategy Group, 2023. Youth Perspectives on Online Safety, 2022: an Annual Report of Youth Attitudes and Experiences. [accessed 12 June 2024]. 314 This report does not specify the online aspect of child sexual abuse, or grooming in particular, but given the prevalence of an online aspect in CSEA, it could be indicative of trends within online abuse: Independent Inquiry Child Sexual Abuse (Gibson, E., Knight, R. Durham, A. and Choudhury, I.), 2022. Engagement with lesbian, gay, bisexual, transgender and queer/ questioning + victims and survivors. [accessed 20 June 2024].
Socio-economic factors¶
2A.41 Perpetrators may seek out children who display certain vulnerabilities, such as being in care.315 Whilst not specific to online harm, the CSA Centre found those who had lived in a care home were nearly four times as likely to have experienced CSA than those who had not.316
Physical/mental health¶
2A.42 Children with mental health difficulties may be at greater risk of online grooming. Evidence suggests perpetrators may seek out children who display certain vulnerabilities online, including mental health difficulties.317 2A.43 Children with experience of an eating disorder may be more vulnerable to online grooming, as there is evidence to suggest that some perpetrators deliberately target them. An exploratory study on children’s vulnerability to human trafficking reported on a number of criminal and investigative journalism cases (across the UK, the Netherlands and Germany) where CSEA perpetrators posed as ‘anorexia coaches’318 to exploit sexual images and acts from young women and girls.319 Children with experience of an eating disorder are more likely to connect with ‘anorexia coaches’ and are therefore at greater risk of harm of sexual abuse. For more information, see the ‘Eating disorder content’ chapter in Volume 3 of our May 2024 Protection of Children Consultation.
Ethnicity¶
2A.44 There may be underreporting of CSEA among children from minority ethnic groups, which may affect the accuracy of the ethnicity-related data for this harm. Police and local authority data indicate that children from these groups are not routinely identified as victims of sexual exploitation, suggesting that their risks and experiences of harm may be overlooked.320 2A.45 The pattern of under-reporting may be due to barriers such as cultural stereotypes, racism, shame, and stigma, which may place these victims and survivors at greater risk of not getting adequate support. The IICSA found that cultural stereotypes and racism had contributed to child sexual abuse going unrecognised or ignored by professionals. Additionally, societal racism can make individuals in ethnic minority communities hesitant to report abuse, fearing reinforcement of negative stereotypes. Furthermore, participants described how shame, stigma, and a fear of ostracisation created further barriers to speaking out. 321
315 Whittle, H. C., Hamilton-Giachritsis, C., Beech, A., and Collings, G., 2013. A review of young people’s vulnerabilities to online grooming, Aggression and Violent Behavior, 18, pp.135-146; CSA Centre (Karsna, K. and Bromley, P.), 2023. Child sexual abuse in 2021/22: Trends in Official data. [accessed 31 July 2023]. 316 CSA Centre (Karsna, K. and Kelly, L.), 2021, The scale and nature of child sexual abuse: Review of evidence. [accessed 20 June 2024]. 317 Whittle, H. C., Hamilton-Giachritsis, C., Beech, A., and Collings, G., 2013. A review of young people’s vulnerabilities to online grooming, Aggression and Violent Behavior, 18, pp.135-146.; CSA Centre (Karsna, K. and Bromley, P.), 2023. Child sexual abuse in 2021/22: Trends in Official data. [accessed 31 July 2023]. 318 These users are known as ‘anorexia coaches’, ‘ana buddies’ or a similar equivalent. ‘Coaching’ can include requesting pictures and videos for ‘body checks’, weekly weigh-ins and enforcing strict rules about what food to eat and avoid. It can also include ‘punishment’ for not complying in the form of verbal abuse, and sometimes requesting sexual images.
319 Dettermeijer-Vermeulen, C., Esser, L. and Noteboom, F. 2016. Vulnerability up Close: An exploratory study into the vulnerability of children to human trafficking. [accessed 18 December 2023]. 320 Missing People, 2023. The ethnicity of missing people: Findings from police and local authority data, 2021-22. [accessed 17 June 2023]. 321 Many victims and survivors described how abuse had a significant effect on their sense of identity and belonging in the community they grew up, with some being ostracised from their communities and cut off from their friends and family. This report does not specify the online aspect of child sexual abuse, or grooming in particular, but given the prevalence of
Media literacy¶
2A.46 There is evidence to suggest that child users sometimes do not understand the risks associated with using a service, such as the risk of sharing personal information, and may not fully understand security settings.322 In these cases, the lack of information from a service on how certain functionalities work may increase the risk of online grooming, leading to uninformed decisions (for example, child users may be more inclined to post personal information without understanding the personal risks that this may entail).
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles¶
2A.47 User profiles, and the information that is presented on them, facilitate the commission of grooming offences as they help perpetrators identify children to target. Perpetrators have described selecting potential victims based on information provided in user profiles, such as profile pictures, name, age, and location.323 Malesky examined the online activity of 31 convicted sex offenders who had communicated with a child online and found that these offenders first viewed user profiles to identify potential victims.324 2A.48 User profiles can make information regarding the increased vulnerability of a child more visible to perpetrators and inform a perpetrator’s risk assessment. NCMEC uses a model of online grooming which highlights that the first step many perpetrators take is a type of evaluation in which they seek to understand a child’s personal characteristics.325 This can include assessing whether the child’s user profile indicates low self-esteem or a lack of supervision. 2A.49 Perpetrators can use multiple user profiles to re-victimise victims and survivors, and find new targets, by creating new accounts when their original account has been blocked, deleted, or removed. In response to Ofcom’s Illegal Harms consultation, GeoComply described how perpetrators have been able to re-victimise children and find new children on platforms even when they have already been banned or removed by a platform.326 2A.50 The risk of user profiles intersects with fake user profiles, as perpetrators can have multiple accounts under fake profiles at any one time, irrespective of whether an account has been banned or removed.
an online aspect in CSEA, it could be indicative of trends within online abuse: Independent Inquiry Child Sexual Abuse (Rodger, H., Hurcombe, R., Redmond, T. and George, R.), 2020. “People don’t talk about it”: Child sexual abuse in ethnic minority communities. [accessed 20 June 2024]. 322 Wolak, J., Finkelhor, D., Mitchell, K. J., and Ybarra, M. L. ,2008. Online ‘Predators’ and their victims, American Psychologist, 63(2) pp.111-128. [accessed 18 November 2024]. 323 Quayle, E., Allegro, S., Hutton, L., Sheath, M. and Lööf, L., 2014. Rapid skill acquisition and online sexual grooming of children, Computers in Human Behaviour, 39, pp.368-375. [accessed 18 November 2024].
324 The study found that offenders based their decision on who to contact on the presence of sexual content in a child’s profile; an explicit statement of age; the perceived neediness or submissiveness of the child; and young-sounding usernames: Malesky, L. A., 2007. Predatory online behavior: Modus operandi of convicted sex offenders in identifying potential victims and contacting minors over the internet. Journal of Child Sexual Abuse, 16, pp.23–32. [accessed 18 November 2024] 325 NCMEC (Australia) adapted the Winters & Jeglic Sexual Grooming Model for online perpetrators. Source: NCMEC, 2022. The new “Stranger Danger”: Tactics used in the online grooming of children. [accessed 11 August 2023]. 326 GeoComply Solutions response to November 2023 Illegal Harms Consultation.
Fake user profiles and anonymous user profiles¶
2A.51 Evidence suggests the ability to create a fake user profile, which allows users to present false representations of themselves, is widely regarded as a key tool used by perpetrators to facilitate the commission of grooming offences. Abusers can create fake user profiles and present themselves as desirable to their target by falsifying their age, name and location.327 This was recently seen in a high-profile case: a man was jailed in 2021 after posing as a teenage girl online and grooming 500 boys, blackmailing over 51 boys into sending indecent images of themselves, and coercing them into committing abusive acts against themselves and other children.328 Ofcom research also found that 15% of 11 to18-year-olds claim to have received a friend request from someone pretending to be someone else.329 Children and young people who have received sexualised messages from users perceived to be adults reported that accounts with heavily anonymised user profiles, for example with cartoon profile pictures and very few followers often used to send explicit images to them.330 2A.52 Perpetrators often use services that do not require them to disclose much personal information. This can include services that require only an email address or a username, which can often be easily falsified. networking
User connections¶
2A.53 The ability for users to connect with each other facilitates grooming, as it allows perpetrators to establish contact with child users and begin communicating, often making it a necessary pre-cursor to direct engagement with a child. Online services that utilise such functionality are implicated in significant numbers of cases of online grooming.331 2A.54 A specific use of user connection functionality by perpetrators is the ‘scatter-gun’ or ’pyramid approach’ where perpetrators use user connection functionalities to try to access large numbers of children. One study provided an example of an offender randomly adding children to initiate contact with them.332 Similarly, Internet Matters reported that social media services will often be used by perpetrators to target a large number of young users by sending out multiple connection requests.333 Specific functionality such as ‘quick add’ has also been noted as reducing barriers to adults connecting with children.334 2A.55 The visibility of user connections may facilitate grooming as these features instil a sense of ‘relationship’ among mutual connections, which can be exploited by abusers to appear as a trusted contact of mutual friends. Young people’s social networking predominantly involves
327 Bluett-Boyd, N., Fileborn, B., Quadara, A. and Moore, S., 2013. The role of emerging communication technologies in experiences of sexual violence: a new legal frontier? [accessed 11 August 2023]. 328 BBC News, 2021. David Wilson: Sex offender who posed as girls online jailed for 25 years. 10 February. [accessed 11 August 2023]. 329 Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research. [accessed 18 November 2024] 330 Ofcom, 2024. Online communications among children and young people: Qualitative research exploring experiences of sexualised messages online 331 For example, in a sample of 641 grooming cases in England and Wales from 2020 where the online service was known, Instagram was used 236 times (37%), Facebook, Instagram and WhatsApp combined were used 324 times (51%), and Snapchat was used in 20% of cases: NSPCC, 2020. Instagram most recorded platform used in child grooming crimes during lockdown. [accessed 18 October 2024]
332 Kloess, J. A., Hamilton-Giachritsis, C. E. and Beech, A. R., 2019. Offence Processes of online sexual grooming and abuse of children via internet communication platforms, Sexual Abuse, 31(1), pp.73-96. [accessed 18 November 2024]. 333 Internet.matters.org, n.d. Learn about online grooming. [accessed 10 August 2023]. 334 Young people interviewed about their experiences of receiving sexualised messages from adults (or perceived adults) felt this function enables adults to quickly add many users including children, widening the base of who they are engaging with. Source: Ofcom, 2024. Online communications among children and young people: Qualitative research exploring experiences of sexualised messages online
pre-established networks, and trust in the other users within the network. This sense of trust expands to those connected or ‘friended’ with others in the network 335 and can provide a false sense of security, for example, an abuser seeming to be ‘known’ to their social network.336 Differences in the nature of sexualised messages received from users with mutual connections compared to those with none have also been highlighted. Children and young people discussing their experiences of receiving sexualised messages from users perceived to be adults suggested that messages from users with no mutual connections were often more likely to be sexualised or explicit in the first instance. Messages from users with some mutual connections were more likely to start off ‘normal’, before becoming sexualised over time.337
2A.56 As well as being used to identify targets, user connections can be used to coerce children. Perpetrators may use user connection lists to demonstrate to children that they know who their family and friends are, which can enable them to manipulate, threaten and coerce children, for example by threatening to share intimate images of the child with their contacts.338
User search¶
2A.57 Search functions that enable children, or groups containing children, to be found on services increase the risk of grooming. These functions allow perpetrators to identify potential victims based on specific characteristics. Research suggests that the ability to target particular groups of potential victims aids in the grooming process.339
User groups¶
2A.58 Online communities may facilitate grooming as they provide a space for an abuser to approach a child in a discreet way, whether as part of a user group discussing a shared interest, or as part of a sexualised environment among adolescents.340 Evidence suggests there are online communities that are popular among adolescents that focus on explicit sexual discussions and obscene language 341 and engaging with other users in such an environment may desensitise users to sexual solicitations from perpetrators in these communities. 2A.59 User groups can be used by perpetrators to attract children. They can create groups centred around topics that appeal to young people to capture their interest, and may impersonate young people, potentially for several months to build trust.342 Some perpetrators work together in such groups to enable grooming.
335 Bluett-Boyd, N., Fileborn, B., Quadara, A. and Moore, S. 2013. The role of emerging communication technologies in experiences of sexual violence: a new legal frontier? [accessed 11 August 2023]. 336 Hamilton-Giachritsis, C., Hanson, E., Helen, W., Alves-Costa, F., and Beech, A., 2020. 337 Ofcom, 2024. 338 Safer Schools, 2023. Protecting Young People from Sextortion. [Accessed 1 September 2023].
339 Bluett-Boyd, N., Fileborn, B., Quadara, A. and Moore, S., 2013. The role of emerging communication technologies in experiences of sexual violence: a new legal frontier? [accessed 11 August 2023]. 340 de Santisteban, P., Del Hoyo, J., Alcázar-Córcoles. M. Á.,, Gámez-Guadix, M., 2018. Progression, maintenance, and feedback of online child sexual grooming: A qualitative analysis of online predators. Child Abuse Neglect. 80, pp.203-215. 341 Wolak, J., Finkelhor, D., and Mitchell, K. J., Ybarra, M. L., 2008. Online ‘Predators’ and their victims, American Psychologist, 63(2), pp.111-128. 342 de Santisteban, P., Del Hoyo, J., Alcázar-Córcoles. M. Á.,, Gámez-Guadix, M., 2018. Progression, maintenance, and feedback of online child sexual grooming: A qualitative analysis of online predators. Child Abuse Neglect. 80, pp.203-215
User communications¶
Livestreaming¶
2A.60 Livestreaming allows users to share content in real time. This can be broadcast to limited connections, open to all users of a site or it can be a one-to-one live video. 2A.61 Livestreaming can be used by perpetrators to enable sexualised conversations with children and/or incite children to engage in sexual activity, sometimes including other children, in real time. The NSPCC found that sexualised conversations take place when children livestream and, of those children who livestreamed, 6% (more than 1 in 20) had received requests to change or remove their clothes. The research found that primary-school-aged children were more likely than secondary-school-aged children to be asked to change or remove their clothes when livestreaming.343 2A.62 Livestreaming functionalities in combination with screen-recording can enable perpetrators to create permanent records of SGII. Once a perpetrator has explicit photos or videos of a child, these can then be used to blackmail children into sharing further images344or to financially extort them.345
Video calling¶
2A.63 Video calling can be used by perpetrators to ask children to engage in sexual activity or to create sexually explicit material while on the call. An NSPCC study found that of the 40,000 children aged 7 to 16 who participated in the study, 12% (more than 1 in 10) of children have video-called with someone they did not know. During these calls, 10% of primary-aged and 11% of secondary-aged children were asked to change or remove their clothes.346
Direct messaging¶
2A.64 Direct messaging can facilitate online grooming by allowing private, direct, and rapid communication between perpetrators and children. This can allow perpetrators to build relationships away from public view and parental supervision.347 2A.65 The privacy of direct messaging can reduce barriers such as social status and age that are present in face-to-face environments. This increased privacy can increase feelings of intimacy and allows for more freedom to broach sensitive topics such as sex.348 2A.66 Evidence has also shown that children and young people are experiencing potentially uncomfortable experiences on direct messages. Ofcom research found that among those aged 11 to 18 who reported such experiences, messages—both individual and group—were the primary method of communication with the other user(s).349 In qualitative research exploring the experiences of children and young people who had received sexualised
343 Survey of nearly 40,000 children aged 7 – 16 years old: NSPCC, 2018. Livestreaming and video-chatting. [accessed 22 September 2023].
344 NCA, Child sexual abuse and exploitation - National Crime Agency [accessed 23 September 2024]. 345 NSPCC 2024. Young people’s experiences of online sexual extortion or ‘sextortion’. [accessed 25 September 2024]. 346 NSPCC, 2018. Livestreaming and video-chatting. [accessed 22 September 2023]. 347 See case studies analysed in source: Kloess, J. A., Hamilton-Giachritsis, C. E. and Beech, A. R., 2019. Offence Processes of online sexual grooming and abuse of children via internet communication platforms, Sexual Abuse, 31(1), pp.73-96. 348 Wolak, J., Finkelhor, D., and Mitchell, K. J., Ybarra, M. L., 2008. Online ‘Predators’ and their victims, American Psychologist, 63(2), pp.111-128. 349 Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research.
messages from users perceived to be adults, all respondents had been contacted via direct messaging functionalities on various online services.350
2A.67 The ability to share images via direct messaging can enable grooming offences by encouraging children to share SGII. Evidence has shown that to normalise their requests for sexual images and to lower a child’s inhibitions, perpetrators may first share pornographic images with the child.351 This is followed by asking the child to send the perpetrator similar images of themselves. 2A.68 The use of emojis in direct messages can help perpetrators to incite children to engage in sexual activity. A study exploring discussions on LiveMe found that emojis were used to send sexually suggestive messages. Their analysis of over 39 million chat messages found that emojis were used as a form of communication to request sexually inappropriate and suggestive acts, such as the removal of clothes.352 The emojis used in these instances were clothing-related emojis, hand gestures and tongue emojis.
Ephemeral messaging¶
2A.69 The use of ephemeral messaging has also been noted in instances where children have received sexualised messages from users perceived to be adults. Young people reported feeling that services with this functionality were more conducive to receiving sexualised messages form adults, and the self-deleting nature of the content meant that showing the messages to others, for example adults they trusted, was more difficult.353
Encrypted messaging¶
2A.70 As has been introduced previously, perpetrators may seek to move their communication with children to online spaces that offer end-to-end encryption, which can make it harder to detect offenders’ contact with children. Law enforcement agencies have highlighted the impact that the increased prevalence of end-to-end encryption could have on detecting offenders and on child safety.354 The National Crime Agency (NCA) also estimated that most reports (92% from Facebook and 85% from Instagram) that are currently disseminated to UK police each year will be lost as a result of the roll out of end-to-end encryption.355 2A.71 Perpetrators may be using end-to-end encrypted messaging to share tips and advice on harmful behaviour with each other. Protect Children reported that the lack of text moderation has allowed for offenders to share grooming strategies, discuss offending and share identified social media accounts of vulnerable children with each other.356
Group messaging¶
2A.72 Group messaging, like direct messaging, allows adults to engage directly with children and develop relationships. Ofcom research found that nearly 1 in 4 children (23%) had been
350 Ofcom, 2024. Online communications among children and young people: Qualitative research exploring experiences of sexualised messages online 351 Thomas, K., Hamilton-Giachritsis, C., Branigan, P. and Hanson, E., 2023. Offenders' approaches to overcoming victim resistance in technology-assisted child sexual abuse, Child Abuse & Neglect, 141(2). 352 These chat messages were exchanged by more than 1.4 million users in 291,000 live broadcasts over two years: Lykousas, N. and Patsakis, C. 2021. Large-scale analysis of grooming in modern social networks. Expert systems with applications, 176.
353 Ofcom, 2024. 354 Virtual Global Taskforce, 2023. Statement on End-to-End Encryption. [accessed 16 August 2023]. 355 NCA, 2024. European police chiefs call for end-to-end encryption roll out to include public protection measures. [accessed 8 July 2024]. 356 Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024].
added to a group chat with people they did not know well or at all, making it the second most common and potentially uncomfortable online contact experience reported.357
Commenting on content¶
2A.73 Perpetrators can use comments on posted content as a means of building rapport with victims and survivors in the early stages of the grooming journey. Reports by BBC News and The Times found the presence of sexually explicit comments on children’s videos on TikTok and YouTube respectively.358 2A.74 There is also evidence to suggest that livestream comments are used to facilitate grooming offences, where hyperlinks or attempts to exchange contact details are shared in comments with the aim of getting the child to connect with the perpetrator on another service. As outlined in the Livestream section above, the NSPCC found that sexualised conversations take place when children livestream.359 Transcripts from the IWF have also shown that perpetrators leave sexualised comments when children livestream.360
Posting or sending location information¶
2A.75 Sharing a user’s current location can provide a perpetrator with the necessary information to physically approach their target. A service which automatically shares a child’s location on shared content, or which gives a child the ability to post or send their location, can give perpetrators information about a child’s frequent places, such as their school and home address. Perpetrators can use this information to gain the trust of children, or may use the knowledge to threaten the child, to further their abuse.
Transactions and offers¶
Accepting online payments¶
2A.76 Perpetrators may send money to a child or buy them gifts (either virtual or physical) to facilitate relationship building as part of the grooming process. This can be enabled by a service accepting online payments. Evidence shows that perpetrators give gifts to flatter children, and as a gesture to give the impression of affection to the child user.361 2A.77 Evidence also suggests that an increasing number of perpetrators are coercing children into sending SGII by offering them money. Thus, functionalities that allow children to accept online payments may facilitate this. Blackmailing child users into generating further sexual images was seen in the case of one perpetrator who posed as a rich businessman online and groomed children, inciting them to generate SGII with offers of financial payments. The perpetrator then threatened to distribute these images to the child’s friends and family unless the child sent further indecent images.362
357 Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research. 358 Shukman, H. and Bridge, M., 2018. Paedophiles grooming children live on YouTube, The Times, 10 December. [accessed 21 September 2023]; Silva, M., 2019. Video app TikTok fails to remove online predators, BBC News¸5 April. [accessed 16 August 2023].
359 NSPCC, 2018. Livestreaming and video-chatting [accessed 22 September 2023]. 360 King, J. 2022. The shocking transcripts that reveal how groomers sexually abuse children in their own rooms, Metro, 3 September 2022. [accessed 21 September 2023]. 361 Gámez-Guadix, M., De Santisteban, P., Wachs, S. and Wright, M., 2021. Unraveling cyber sexual abuse of minors: Psychometrics properties of the Multidimensional Online Grooming Questionnaire and prevalence by sex and age, Child Abuse and Neglect, 120. 362 BBC News, 2021. Abdul Elahi: Sexual blackmailer jailed for 32 years.[accessed 16 August 2023].
Content storage and capture¶
Screen recording or capturing¶
2A.78 Screen-recording and screen-capture functionalities can be deployed by perpetrators during video calls or livestreaming to non-consensually capture indecent images.363 They can then use these images to blackmail the child to generate further CSAM.
Content editing¶
Editing visual media¶
2A.79 Editing visual media can facilitate grooming offences by allowing perpetrators to disguise their identity, often posing as young people when first contacting children. They can do this by using video- or image-editing functionalities, such as filters, to change their appearance when calling a child, or when sending them photos of themselves. Deepfake technology and GenAI can assist in creating images and videos that may be uploaded to user-to-user services, making approaches to children appear more genuine and thereby increasing the likelihood of a response.364 GenAI ‘nudifying’ apps can also be used to take a child’s photos from their social media and create CSAM which can then be used to blackmail and ‘sextort’ them – for more information on these kinds of deepfakes, see the Intimate image abuse chapter, and for details on financial extortion resulting from media editing see ‘financially motivated sexual extortion’ in the introduction of this chapter.
Recommender systems¶
Network recommender systems¶
2A.80 Network recommender systems can play a role in facilitating grooming by suggesting child users to adults and adult users to children. Network recommender systems are used by services to recommend connections, and child users are likely to accept suggested connection recommendations, thereby expanding their online networks. Ofcom research found that 30% (nearly one-third) of children and young adults aged 11 to 18 said that they had added contacts via friend suggestions, quick adds, connections requests or cover functions.365 Where an adult user has an established pattern of adding a lot of child contacts to their network, the recommender system could suggest this adult user to other children. 2A.81 A perpetrator may join various groups focused on topics that appeal to children, in an attempt to make contact with them. Recommender systems may then suggest further user groups or connections, based on the perpetrator’s membership of these user groups.
363 While users can often screen record or capture content using third-party services, screen recordings and captures are shared on U2U services as user-generated content and some U2U services have dedicated screen recording and screen capturing functionalities.
364 Deepfakes are a specific type of media that involves the use of AI algorithms, particularly generative AI models, to modify videos, images or audio to create realistic synthetic content. This is often done by superimposing the face of a person onto the body of another person in a video or image as well as voice manipulation with lip syncing. Deepfakes are commonly shared as user generated content on user-to-user services but could also potentially be created using functionalities present on user-to-user services. Deepfake technology is currently used to create content that can be harmful; however, we acknowledge that it may also have positive use cases. 365 Ofcom, 2023. Understanding Online Communications Among Children – Quantitative Research.
Risk factors: Business models and commercial profiles¶
Revenue models¶
2A.82 Services that generate revenue through the sale of online gifts or tokens may be at greater risk of being exploited by perpetrators of online grooming. The C3P have outlined how some individuals who perpetrate online grooming offences send gifts and tokens to children as part of the grooming process and/or to incentivise the child to share sexual imagery.366 2A.83 Platforms that facilitate money transfers or other U2U payments may be attractive for those seeking to perpetrate financially motivated sexual extortion schemes. Many services will generate revenue from each of these transactions.367
366 Canadian Centre for Child Protection (C3P) response to November 2023 Illegal Harms Consultation. 367 Canadian Centre for Child Protection (C3P) response to November 2023 Illegal Harms Consultation.
2B Child Sexual Abuse Material (CSAM)¶
Warning: This chapter contains content that may be upsetting or distressing in relation to CSAM. Summary analysis for CSAM: how harms manifest online, and risk factors Online CSAM includes material depicting sexual activity, or indecent or prohibited imagery of children and can take the form of photographic images and videos, as well as non-photographic material, such as drawings and animations. CSAM can have a profound and long-lasting impact on children who are sexually abused, as well as on the wellbeing of adults and children who unintentionally view this material. Beyond the abuse itself, the presence, sharing and viewing of images and videos depicting the abuse can serve as a continual source of trauma for victims and survivors of CSEA. Victims and survivors may experience re-victimisation and can be affected by heightened sensitivity to photos and cameras. CSAM is often accessed intentionally, and the availability of such material online creates a permissive environment in which perpetrators may develop and act on their sexual interests. The availability of CSAM can lead to unintentional viewing, likely causing considerable distress. Children themselves may generate content that can be considered CSAM, which can cause them harm. UK law enforcement refers to this as self-generated indecent imagery (SGII). Service type risk factors: Any service can be used to distribute CSAM. Services enabling image or video sharing, text posting or hyperlink sharing pose particular risks. File-storage and file-sharing services, especially those that allow users to upload and share images through links, are considered particularly risky, as they enable large, curated collections of CSAM to be stored. While studies show that perpetrators also regularly encounter CSAM on social media and video sharing services,. other types of services also pose risks of CSAM offences, including discussion forums and chat rooms, messaging services and user-to-user pornography services. User base risk factors: As outlined in the grooming section above, services with a large user base can pose a grooming risk and therefore may be considered risky for the creation of first generation CSAM. However, evidence suggests that perpetrators also often use 84small and less-mature services to share CSAM, as these services may be less likely to have CSAM detection technology and processes in place. Child users on a service can be a risk factor for CSAM, as offenders may search for content uploaded by children on their personal accounts, some of which may be considered CSAM. Gender, disability, ethnicity, socio-economic factors and sexual orientation and gender identity can factor in how likely children are to be vulnerable to the way the different CSEA offences manifest online. Functionalities and recommender systems risk factors: Several functionalities can facilitate CSAM offences. Group messaging enables CSAM sharing or trading within communities of users, while direct messaging and the ability to post content, such as text and images, are also used by perpetrators to share and distribute CSAM. Encrypted messaging enables perpetrators to share CSAM with less risk of discovery, while ephemeral messaging complicates detection due to the disappearing nature of the content. Messages or posts can include hyperlinks to collections of CSAM saved on file-storage and file-sharing services. These hyperlinks can be shared with perpetrators, sometimes for a fee. Anonymous profiles can allow perpetrators to avoid being personally identified by a service when sharing or accessing CSAM. Livestreaming can allow abusers to create CSAM during livestream sessions or from SGII, which can then be widely distributed. This is particularly risky when combined with storage and screen capture functionalities. The ability to post goods or services for sale can be exploited to distribute CSAM, while cryptocurrency payments pose a growing threat, as using cryptocurrencies allow offenders to buy CSAM anonymously and evade detection. User profiles and unauthenticated user profiles can facilitate CSAM offences, by enabling abusers to target children. Additionally, they are also a tactic used to signpost other perpetrators to CSAM. Functionalities allowing users to download content, such as CSAM, enable users to store and view local copies of content on their devices, as well as to share it with others. Content recommender systems are also a risk factor in the viewing and discovery of CSAM, as a service’s algorithm could suggest CSAM-related content to users who are actively viewing CSAM videos. Business model risk factors: Low-capacity services, and services that are earlier in their business development lifecycle will be at greater risk of being used by perpetrators to share CSAM. Early-stage services are less likely to have established processes or resources to detect and remove CSAM from their services. If a service has insufficient focus on having effective moderation and verification processes in place, this can be exploited by perpetrators to share CSAM content. 85
How child sexual abuse material offences manifest online¶
2B.1 This section provides an overview of how CSAM manifests online, and how users may be at risk of harm. 2B.2 Online CSAM includes material depicting penetrative sexual activity, non-penetrative sexual activity, or indecent or prohibited imagery of children.368 This can take the form of photographic images and videos, as well as non-photographic material, such as drawings and animations. CSAM can also include deepfake imagery, ‘pseudo-photographs’369 and imagery created using generative AI tools accessed through extended reality technologies. CSAM is not limited to image-based material and can include materials that provide advice on grooming or abusing a child sexually, or material that is considered obscene and encourages the commission of other CSEA offences.370 2B.3 Perpetrators of CSAM may engage in creating, uploading, sharing, and distributing such material. They may also facilitate the creation, uploading, sharing, or distribution of CSAM by indicating that they possess it and by sharing links or advising others on where it can be found. 2B.4 While it is difficult to accurately estimate the volume of CSAM online, a number of sources show how widespread it is. NCMEC reported that it received over 36.2 million reports of suspected child sexual exploitation via its CyberTipline in 2023, a more than 12% increase on 2022, and the number of files included within the reports increased by 19% to more than 100 million.371 The IWF confirmed that it received 275,652 reports containing CSAM, links to CSAM, or advertised CSAM in 2023.372 Police data also shows that c.107,000 sexual offences against children were recorded by the police across England and Wales in 2022, a 7.6% increase on 2021 and a near quadrupling of the number recorded ten years prior. Police estimates suggest that online CSEA accounts for at least 32% of the recorded total.373 2B.5 Evidence suggests the presence of CSAM online is increasing. There have been year-on-year increases in the number of URLs which contain CSAM reported to the IWF, with an 8% increase between 2022 and 2023.374 2B.6 CSAM is present on both the dark and clear web. While CSAM can be found on the dark web, 97% of CSAM detected by the C3P Project Arachnid was hosted on the clear web.375 Similarly, in a survey of individuals searching for CSAM on dark web search engines, 77% of
368 Crown Prosecution Service, 2020. Indecent and Prohibited Images of Children. [accessed 18 August 2023]. 369 A pseudo-photograph is an image made by computer-graphics or otherwise which appears to be a photograph: Home Office, 2023. Indecent Images of Children: guidance for young people. [accessed 22 September 2023]. 370 For more information on what constitutes CSAM see the Illegal Content Judgement Guidance (ICJG - Volume 5, Chapter 26).
371 NCMEC, 2024. CyberTipline 2023 Report. [accessed 19 November 2024] 372 IWF, 2024. IWF Annual Report 2023 #behindthescenes. [accessed 8 July 2024]. 373 National Police Chiefs’ Council (NPCC), 2024. National Analysis of Police-Recorded Child Sexual Abuse & Exploitation (CSAE) Crimes Report - January 2022 to December 2022. [accessed 20 September 2024] 374 IWF, 2024. IWF Annual Report 2023 #behindthescenes. [accessed 8 July 2024]. 375 C3P, 2021. Project Arachnid: Online availability of child sexual abuse material. [accessed 21 August 2023]. 86
respondents report that they have encountered CSAM or links to CSAM somewhere on the clear web.376
2B.7 The severity of harm is rising as more extreme categories of CSAM are detected, particularly involving babies and toddlers. In 2022, the IWF reported a yearly increase in Category A material, which includes images of penetrative sexual activity, sexual activity with animals, or sadism.377 By the end of 2023, the IWF's hash database contained 10,393 unique image hashes of Category A material depicting children aged 0-2, and 554,553 Category A hashes in total, reflecting increases of 19% and 35%, respectively, compared to 2022. A hash serves as an indicator of the existence of such material.378 2B.8 Most of the CSAM presently detected on U2U services is content that has previously been shared, and sometimes reshared, hundreds of thousands of times over a period of many years.379 In contrast, ‘first-generation’ or ‘novel’ CSAM refers to material that is newly generated and which has not been previously shared, re-shared or detected.380 Perpetrators will often exchange first-generation CSAM in return for other new material. 2B.9 There is some evidence to suggest viewing abusive pornography can act as a gateway to perpetrators seeking out CSAM. The WeProtect Global Alliance’s 2023 Global Threat Assessment reported on emerging evidence of an association between the frequent viewing of pornography and progression to viewing CSAM.381 Similarly, the CSA Centre described how a common pathway into viewing CSAM involves initially viewing legal pornography, which gets more extreme and depicts younger individuals over time.382 Interviews with offenders who have viewed CSAM in the UK also indicated that most had not intentionally sought out CSAM, but that it was a result of entrenched pornography use and spiralling online behaviour.383 Further, the Lucy Faithfull Foundation has reported that of 3,400 callers to their Stop It Now Helpline in 2023, comprising adults and under 18s who said they had abused or were close to abusing, or were worried about their thoughts or behaviours, 25.9% (881) had self-reported a ’problem’ with pornography.384 2B.10 Perpetrators actively identify and share legal content that is linked to CSAM using various tactics to indicate their ability to share it, but without publicly uploading it. This practice, known as ‘contextual CSEA’ includes sharing personal information about CSAM victims and
376 Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 377 The IWF notes that of the reports it received in 2022, 255,588 were confirmed to have contained images or videos of children suffering sexual abuse. Of these, 51,369 were the most severe Category A images: IWF, 2023 The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 11 August 2023]. 378 IWF, 2023 The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023]. 379 This is in large part due to there being more consistent and developed systems for identifying known CSAM, such as the use of CSAM hashing databases, compared to identifying first-generation CSAM. 380 This includes material that has been produced by a perpetrator who has sexually abused a child in person or who has directed the in-person sexual abuse of a child, or by a child creating ‘self-generated’ CSAM – known as ‘self-generated indecent imagery’ (SGII).
381 We Protect Global Alliance, 2023. Global Threat Assessment 2023: Assessing the scale and scope of child sexual exploitation and abuse online, to transform the response. [accessed 10 June 2024]. 382 CSA Centre (Brown, S.), 2023. Key messages from research on child sexual abuse by adults in online contexts. [accessed 11 June 2024]. 383 The Police Foundation, 2022. Turning the Tide Against Online Child Sexual Abuse. [accessed 10 June 2024]. 384Lucy Faithfull Foundation response to November 2023 Consultation, p.30 87
survivors to help locate material, as well as sharing contextual images taken from a sexual abuse ‘series’ to imply possession of illegal material.385 Similar to tactics seen in other online harms, this practice—where perpetrators signpost others towards CSAM—is referred to as ‘breadcrumbing’.
Risks of harm to individuals presented by child sexual abuse material offences¶
2B.11 For victims and survivors, knowing that CSAM remains available online and that perpetrators may still be using it, can be a continuing source of trauma. Some describe ‘closure’ as impossible.386 Many describe feeling constantly in fear, and vulnerable, because their abuse exists as a permanent record online which others can view.387 2B.12 Survivors can be re-victimised by reliving their sexual abuse if they encounter the material online, or by the fear of being recognised by someone who has seen it. In a survey conducted by the C3P, 69% (more than two-thirds) of victims and survivors reported constant worry about being recognised, and almost a third had been identified online or in person by someone who had seen images of their abuse.388 Some victims and survivors reported being targeted and re-victimised by someone who had recognised them, including being propositioned or threatened. Victims and survivors also describe suffering from a heightened sensitivity to photos and cameras.389 2B.13 The impacts of SGII, both non-consensual and aggravated,390 can be wide ranging and severe. In the case of non-consensual SGII, the negative impacts on the child depicted are significant, often leading to mental health challenges391 and negative social consequences,392 particularly for girls, who will often face bullying, harassment, social exclusion, and victim-blaming.393 Many victims and survivors of aggravated SGII describe feelings of self-blame, negative psychological health, and heightened anxiety from knowing that the images remain online. 2B.14 The re-victimisation of survivors can be exacerbated by the increasing use of generative AI to create CSAM. Children who have been abused may experience re-victimisation as offenders
385 C3P, 2019. How we are failing children: Changing the Paradigm. [accessed 23 August 2023]. 386 CSA Centre (Brown, S.), 2023. Key messages from research on child sexual abuse by adults in online contexts. [accessed 11 June 2024].
389 NSPCC (Hamilton-Giachritsis, C., Hanson, E., Whittle, H. and Beech, A.), 2017. “Everyone deserves to be happy and safe”. A mixed methods study exploring how online and offline child sexual abuse impact young people and how professionals respond to it. [accessed 31 August 2023]. 390 See the sub-section ‘Cross-cutting harms’ for a definition of non-consensual and aggravated SGII. 391 Frankel, A., Bassm S., Patterson, F., Dai, T., Brown, D., (2018). Sexting, Risk Behaviour, and Mental Health in Adolescents: An Examination of 2015 Pennsylvania Youth Risk Behavior Survey Data. Journal of School Health, 88(3), pp.190-199.
392 From a qualitative study with 41 young people in south-east England. Setty, E. 2019. A rights based approach to Youth Sexting: Challenging, Risk, Shame, and the Denial of Rights to Bodily and Sexual Expression Within Youth Digital Sexual Culture. International Journal of Bullying Prevention, 1, pp.298-311. [accessed 18 November 2024]. 393 Ringrose, J., Regehr, K., Whitehead, S., 2022. ‘Wanna Trade?’ Cisheteronormative homosocial masculinity and the normalisation of abuse in youth digital sexual image exchange. Journal of Gender Studies, 31(2), pp.243 – 261. [accessed 18 November 2024]. 88
generate and distribute new sexual imagery of them.394 Offenders also use generative AI to create new CSAM from innocuous images of children online, so many more children may be at risk of being newly victimised.395
2B.15 CSAM created using generative AI may also have other impacts, including the potential to normalise abuse and overwhelm both authorities and platforms. The NSPCC has warned that such use of generative AI risks normalising the sexual abuse of children.396 The Australian eSafety Commissioner highlighted that as technology advances and enables the creation of increasingly realistic images, it may become harder for police forces and hot lines to identify children who are currently being abused and in need of urgent protection.397 The large scale at which AI-generated CSAM can be produced presents a major and complex challenge to detection and moderation.398 The NSPCC has also reported that some children have expressed nervousness at reporting AI generated images of themselves, or to speak to trusted adults, fearing they may not be believed when explaining that the images are artificially generated.399 2B.16 CSAM also has a broader impact on the population, as the unintentional viewing of CSAM – by both adults and children – is likely to cause considerable distress. 6% of British adults’ report having been exposed to CSAM online.400 A Childline report using data from UK counselling sessions found that young people who had accidentally accessed CSAM online often felt reluctant to confide in anyone about it. They feared they might not be believed or,
2B.17 Unintentional viewing of CSAM by adults may also risk creating a pathway to viewing more CSAM.402 For some, it may cause individuals to become desensitised to the material and fall into more regular, intentional viewing that they then find difficult to stop.403 A survey conducted by Protect Children, with people who view CSAM, found that 50% of respondents wanted to stop viewing CSAM.404
394 Lucy Faithfull Foundation, 2024. A call to end AI-generated child sexual abuse. [accessed 11 June 2024]. 395 In a legal decision on deepfake CSAM in Canada, the court noted how this technology could be used to victimise any child using photos stolen from social media or taken surreptitiously in public: C3P response to November 2023 Illegal Harms Consultation. 396 NSPCC response to November 2023 Illegal Harms Consultation. 397 eSafety Commissioner, 2023. Generative AI – position statement. [accessed 12 June 2024]. 398 NSPCC response to November 2023 Illegal Harms Consultation. 399 NSPCC response to May 2024 Protection of Children Consultation 400 The figure is higher for young adults, with 14% of 18-24-year-olds reporting having been exposed to CSAM online: IWF, 2022. More than one in 10 British young people exposed to online child sexual abuse. [accessed 25 August 2023]. 401 NSPCC, 2016. Online child sexual abuse images: Doing more to tackle demand and supply. [accessed 25 August 2023]. 402 Over half (51%) of respondents to a survey of CSAM users on the dark web reported that they had first encountered CSAM accidentally, meaning they were exposed to CSAM without actively searching for it. Insoll, T., Ovaska, O. & Vaarenen-Valkonen. 2021. CSAM Users in the Dark Web: Protecting Children Through Prevention. [accessed 24 September 2024].
403 Insoll, T., Ovaska, A. K., Nurmi, J., Aaltonen, M., and Vaaranen-Valkonen, N., 2022. Risk Factors for Child Sexual Abuse Material Users Contacting Children Online: Results of an anonymous multilingual survey on the dark web, Journal of Online Trust and Safety, 1(2). [Note: This research was carried out on the dark web, which is out of scope of the Act]. 404 Sample of 3,935: Suojellaan Lapsia, Protect Children (Insoll, T., Ovaska, A., and Vaaranen-Valkonen, N.), 2021. CSAM Users in the dark web: Protecting children through prevention. [accessed 25 August 2023]. 89
Evidence of risk factors on user-to-user services¶
2B.18 We consider that the risk factors below are liable to increase the risks of harm relating to CSAM. These are also summarised at the beginning of the chapter. 2B.19 The evidence used in this chapter is not necessarily tied to individual offences, but the analysis relates more broadly to perpetrator actions that can lead to CSAM being created and appearing on U2U services. The nature of CSAM offences is such that the presence of CSAM online is likely to be very closely linked to the offences of creating, possessing, distributing, and publishing CSAM.
Risk factors: Service types¶
2B.20 While any service that allows users to share images, videos or text can allow perpetrators to distribute CSAM, the following types of services in particular can be used to facilitate or commit offences related to CSAM: discussion forum and chat room services, social media services and video-sharing services, messaging services, file-storage and file-sharing services, and online user-to-user pornography services.
Discussion forums and chat-room services¶
2B.21 Discussion forums can be used to embed and advertise CSAM. The IWF reported that 5% of the child sexual abuse imagery it detected appeared on forums.405 In a separate 2018 study, specifically examining captures of livestreamed child sexual abuse, the IWF found that 73% (nearly three-quarters) of the images it discovered were embedded into 16 forums that were “dedicated to the distribution of captures of live-streamed child sexual abuse”. These forums were “at the centre of distribution networks for captures of live streamed child sexual abuse”.406 407
Social media services and video sharing services¶
2B.22 There is a range of evidence that shows CSAM is available on social media and video sharing services. In a survey of individuals searching for CSAM, 29% (nearly one-third) of respondents said they had encountered CSAM on a social media platform.408 Data from the NCMEC CyberTipline showed that, listed in order of level of reports, Facebook, Instagram, Twitter/X, Snapchat and TikTok accounted for 87% of all reports made in 2023.409 Furthermore, the NSPCC reported that, among cases where a social media site was identified, Snapchat accounted for 48% of the offences relating to the sharing and
405 IWF, 2023 The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023]. 406 Based on a sample of 2,082 images over three months: IWF, 2018. Trends in Online Child Sexual Exploitation: Examining the distribution of livestreamed child sexual abuse. [accessed 22 September 2023].
407 Discussion forums and chat rooms may appear as features within a service. This could in some cases present a greater risk as these services may not have the user-to-user interactions that they enable as a primary focus and may lack robust moderation or trust and safety procedures. 408 This was the second most popular location, behind pornography websites (32%): Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 409 NCMEC, 2023. . [accessed 23 September 2024]. 90
possession of indecent images of children. Facebook and Messenger accounted for 10% of such offences, Whatsapp for 12% and Instagram for 6%.410
2B.23 There is further evidence that demonstrates perpetrators are actively using social media platforms to search for and share CSAM. Protect Children found that, in a survey of individuals searching for CSAM, 32% (nearly one-third) of respondents said they had used social media platforms to search for, view, or share CSAM.411 2B.24 Social media services can be also used to find information on how to commit child sexual abuse offences, and signpost users to other services where CSAM is shared. In a survey of individuals searching for CSAM, 16% of respondents said they learnt how to access CSAM in the dark web on social media platforms.411
Messaging services¶
2B.25 There is evidence to suggest CSAM is being encountered on messaging services.412 Protect Children reported that, in a survey of individuals searching for CSAM, 12% had encountered CSAM on messaging apps.413 2B.26 There is further evidence to show that messaging services are being actively used by perpetrators to search for, view and share CSAM. One survey reported that messaging services are one of the online channels that perpetrators use to distribute CSAM.414 Protect Children also reported that 29% of respondents had used a messaging app to search for, view, or share CSAM.415 2B.27 The use of messaging services may have increased since the pandemic. Research by Interpol showed that there was an increase in the volume of CSAM sent via private messaging applications, as well as social media services, during the COVID-19 pandemic.416 2B.28 Messaging can be used in conjunction with other services to facilitate CSAM offences; for example, to communicate with victims or facilitators of livestreamed CSEA.417 Some offenders start this interaction via messaging, then move to another service to watch the livestream.418
410 This is according to freedom of information data obtained from UK police forces, relating to 2021/22: NSPCC, 2023. We’re calling for effective action in the Online Safety Bill as child abuse image crimes reach record levels. [accessed 11 June 2024]. 411 Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 412 Messaging services are defined in the glossary as: A user-to-user service type describing services that are typically centred around the sending and receiving of messages that can only be viewed or read by a specific recipient or group of people. 413 Protect Children (Lapsia, S.), 2024. 414 Lee, H. E., Ermakova, T., Ververis, V. and Fabian, B., 2020. Detecting child sexual abuse material: A comprehensive survey. Forensic Science International: Digital Investigation, 34. See ‘Risks of harm to individuals presented by CSAM offences’ for more information. 415 The survey was conducted among individuals searching for CSAM on dark web search engines: Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024].
416 Interpol is an international organisation that facilitates police cooperation on international crime: Interpol, 2020. INTERPOL report highlights impact of COVID-19 on child sexual abuse. [accessed 20 June 2024]. 417 Napier, S., Teunissen, C. and Boxall, H., 2021. Live streaming of child sexual abuse: An analysis of offender chat logs, Trends and issues in crime and criminal justice, 639. [accessed 25 August 2023]. 418 Napier, S., Teunissen, C. and Boxall, H., 2021. 91
2B.29 Messaging services with end-to-end-encrypted messaging can make the exchange of CSAM harder to detect. For more information, see Risk factors: Functionalities and recommender systems and End-to-end encrypted messaging.
File-storage and file-sharing services¶
2B.30 File-storage and file-sharing services419, particularly services that allow users to upload and share access to images and videos, present a significant risk for hosting CSAM because perpetrators can store it on these services. In 2023, INHOPE reported that approximately 39% of the CSAM it detected was hosted by ‘image hosts’. They also reported that 5% of the CSAM detected was hosted by ‘file hosts’, although this figure has been as high as 26% in previous years. INHOPE suggested that this reduction from previous years was likely to be due to difficulties in detecting illegal content on file hosting services that require payment.420 The IWF found that 89% of images or videos detected of livestreamed child sexual abuse were stored on an ‘image-hosting service’. Other types of file-storage and file-sharing services such as ‘cyberlockers’ and ‘image stores’ made up 4% and 1% of cases respectively of the child sexual abuse imagery that the IWF reviewed in 2023.421 2B.31 File-storage and file-sharing services may also enable the sharing of CSAM, as perpetrators can distribute URLs directing users to these collections.422 A study reported that this is done through URLs which are shared to services such as image boards, offender forums and ‘chats’.423 According to the IWF, “image hosts allow users to upload still images which are assigned a unique URL and can be embedded to display on third-party websites, such as forums or social networking sites”.424
User-to-user pornography services and dating sites¶
2B.32 Perpetrators are encountering CSAM on user-to-user pornography services. Protect Children found that, in a survey of individuals searching for CSAM, roughly one third (32%) of perpetrators had encountered CSAM on pornography websites; making it the most common type of online service identified in the survey for where CSAM was encountered on the clear web.425 2B.33 Evidence suggests perpetrators are also using user-to-user pornography services to disseminate CSAM. A 2021 report by the C3P stated that Serverel, which is the hosting
419 File-storage and file-sharing services are defined in the glossary as: ‘User-to-user service type describing services whose primary functionalities involve enabling users to store digital content and share access to that content through links.’ 420 INHOPE are a global network of organisations working to tackle CSAM: INHOPE, 2023. Annual report 2023. [accessed 7 August 2024]. 421 IWF, 2023. Annual report 2023. [accessed 7 August 2024]. 422 We understand that perpetrators may be more likely to choose these services if they are encrypted, or if access is time limited. 423 GCHQ Government Communications Headquarters, “A thematic overview of how the internet facilitates the distribution of Child Sexual Abuse Material.” GCHQ Government Communications Headquarters, 2022. As cited in Dorotic. M. and Johnsen, J. W., 2023. Child Sexual Abuse on the Internet. Report on the analysis of technological factors that affect the creation and sharing of child sexual abuse material on the Internet. [accessed 25 August 2023].
424 IWF, 2018. Trends in Online Child Sexual Exploitation: Examining the Distribution of Captures of Live-streamed Child Sexual Abuse. [accessed 22 September 2023]; IWF found that 77% of child sexual abuse imagery appeared on imaging hosts: IWF, 2023. The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023]. 425 Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 92
provider for at least 1,200 unique websites sharing adult content, received 66,824 removal notices for hosting post-pubescent CSAM. Overall, more than 72,000 pieces of content were targeted for removal on Serverel sites (including both pre- and post-pubescent CSAM).426
2B.34 Perpetrators also use online user-to-user pornography services to seek out CSAM. This is evidenced by the chatbot released on Pornhub, which is used to intercept searches using known CSA search terms. The chatbot was launched in March 2022, and was used in 173,904 search attempts in the first 30 days after its launch.427 2B.35 Dating services can be exploited to solicit livestreamed abuse of children. A 2022 Australian study of 9,987 people who had used mobile or website dating platforms in the past five years, found that 12.4% of respondents reported receiving requests to exploit their own children or children they had access to.428 These requests included seeking sexual information, images, or videos of children.
User base size¶
2B.36 Evidence suggests that both large and small services can pose a risk of CSAM. Some of the most prolific sharing of CSAM occurs in services with large user bases. In addition, evidence suggests that perpetrators seeking SGII will often target services with larger user bases. As described earlier in this chapter, perpetrators seeking to groom children for the purposes of creating SGII will often use services with a larger user base, allowing them to target a larger number of children using the ‘scattergun approach’.429 2B.37 However, larger service sizes and user bases may correlate with greater detection and removal efforts, and as a result, the rates of detected CSAM on smaller services (with lower detection capabilities) may not be representative of the volume of CSAM present on those services. Intelligence suggests that perpetrators often seek out services with smaller user bases, particularly services that are less mature, as these services may have fewer CSAM detection technologies or processes in place. In addition, some services with a smaller user base offer users’ specific functionalities which may not be available on services with larger user bases, such as the ability to post content without a registered account. Perpetrators may target these services to exploit such functionalities.
User base demographics¶
2B.38 The following section outlines the key evidence on user base demographic factors and risks of harm, which can include protected characteristics. Services should consider the intersecting influence of demographic factors on risk, which can be contextual, complex, and involve multiple factors.
426 C3P, 2021. Project Arachnid: Online availability of child sexual abuse material. [accessed 21 October 2024]. 427 IWF, 2022. Internet Watch Foundation, Stop It Now, and Pornhub launch first of its kind chatbot to prevent child sexual abuse. [accessed 30 August 2023]. 428 Teunissen, C., Boxall, H., Napier, S. & Brown, R. 2022. The sexual exploitation of Australian children on dating apps and websites. [accessed 24 September 2024]. 429 NCA, 2021. National Strategic Assessment of Serious and Organised Crime. [accessed 31 August 2023]. 93
Age¶
2B.40 Services with a young user base may be at increased risk of CSAM offences. Content uploaded by children may be sought out by potential perpetrators, and this content may be classifiable as CSAM. Further, as is discussed above, perpetrators can target services with a younger user base to identify children for the purposes of grooming, which may result in the production and sharing of CSAM, including SGII. 2B.41 Analysis by the IWF indicates that children of all ages are at risk. In a review of unique indecent image hash matches by age for 2023, the youngest child in the image was assessed to be aged 0-2 in 35,153 instances, with 227,437 images for children aged 3-6, and over 900,000 instances for both children aged 7-10 and those aged 11-13. There were also 96,322 unique indecent images of children where the youngest child was assessed as being aged 14-15, and 45,638 of children aged 16-17.430
Gender¶
2B.42 Gender is a risk factor for CSAM as most children depicted in CSAM are girls. The IWF found that 96% of the reports processed in 2022 depicted exclusively girls. Furthermore, many of the SGII reports (64%) received by the IWF in 2022 related to girls aged 11 to 13.431 2B.43 While most children depicted in CSAM are girls, there is some evidence to suggest that the content that does depict boys tends to be more severe. The IWF found that content depicting boys tended to be of higher severity (based on CSAM categories) than content depicting girls.432 2B.44 In terms of offender demographics, most perpetrators are male. The IWF found that, of the CSAM they analysed, where a perpetrator was visible, they tended to be male.433 One study by the United States Sentencing Commission also found that 94.3% of offenders involved in CSAM production were male.434
Ethnicity¶
2B.45 Evidence has shown that children from a wide range of ethnic backgrounds are at risk. A study by ECPAT International and Interpol found that, of the analysed CSAM in their study in which ethnicity was determinable, both the majority of children (76.6%) and the majority of perpetrators (78.8%) were white.435 In terms of other ethnicities, 10.1% of children were classified as Hispanic or Latino, 9.9% were Asian, and 2.1% were Black. For perpetrators, the research found that 12.2% were Hispanic or Latino, 4.2% were Black and 3.2% were Asian. 2B.46 As has already been described, there is evidence to suggest there may be an under-reporting of children from minority ethnic groups being identified as victims of CSEA, which may affect
430 Unique Image Analysis | IWF 2023 Annual Report [accessed 20 November 2024] 431 IWF, 2023 The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023]. 432 IWF, 2023. The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023].
433 IWF, 2022. The Annual Report 2021. [accessed 30 August 2023]. 434 United States Sentencing Commission, 2021. Federal Sentencing of Child Pornography: Production Offenses. [accessed 22 September 2023]. 435 ECPAT and Interpol, 2018. Towards a global indicator on unidentified victims in child sexual exploitation material: summary report. [accessed 30 August 2023]. 94
the accuracy of the ethnicity-related data for this harm. For more information, see Grooming, Risk factors: User base, ethnicity.
Socio-economic factors¶
2B.47 The socio-economic background of children, particularly when intersecting with gender, has been found to increase the risk of SGII occurring. Evidence has found that girls, particularly from less privileged backgrounds, are at greater risk of experiencing the non-consensual sharing of SGII.436
Disability¶
2B.48 Children with a disability may be more likely to be depicted in CSAM. Research indicates that perpetrators target and exploit the vulnerability of children with disabilities in order to sexually abuse them,437 which can result in the production of CSAM. It is estimated that children with disabilities are nearly three times as likely to be sexually abused than children without disabilities.438 In particular, research has found that children with disabilities or who are neurodivergent are more vulnerable to pressure from others to produce SGII.
Sexual orientation and gender identity¶
2B.49 LGBTQIA+ young people may be less open about and less supported with their feelings, and their isolation may make them more vulnerable to sexual exploitation, including pressure from others to produce SGII. 439 2B.50 As has already been described, there is evidence to suggest there may be an under-reporting of LGBTQIA+ individuals as being identified as victims of CSEA, which may affect the accuracy of reported data for this demographic for this harm. 440 For more information, see ‘Grooming, Risk factors: User base - sexual orientation and gender identity.’
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles¶
2B.51 User profiles and the accompanying statements on them, such as biographies, can facilitate CSAM offences. A study into TikTok, which is used predominantly by younger users, found that some TikTok user profiles included statements of interest in naked images and the exchange of sexual videos.441
436 Revealing Reality, 2022. Not Just Flirting: The unequal experiences and consequences of nude image-sharing by young people. [accessed 30 August 2023]. 437 Independent Inquiry Child Sexual Abuse, 2022. Child sexual exploitation by organised networks: Investigation Report. [accessed 30 August 2023]. 438 Vera Institute of Justice (Smith, N. and Harrell, S.) 2013. Sexual Abuse of Children with Disabilities: A National Snapshot. [accessed 30 August 2023].
439 Independent Inquiry Child Sexual Abuse, 2022. The Report of the Independent Inquiry into Child Sexual Abuse. [accessed 25 September 2024]. 440 Independent Inquiry Child Sexual Abuse, 2022. 441 Cox, J, 2018. December 6. TikTok, the app super popular with kids, has a nudes problem. VICE, 6 December. [accessed 20 August 2023]. 95
2B.52 The ability to create multiple user profiles can enable perpetrators to overcome measures such as strikes and blocking, by creating multiple profiles from which they access and share CSAM.
Fake user profiles¶
2B.53 Fake user profiles can be created by perpetrators to impersonate victims and survivors, so that perpetrators can connect with each other and share advice. The NSPCC has warned of ‘tribute sites’ and ‘tribute user profiles’, that impersonate victims and survivors so that abusers can connect with each other and share advice.442 This is a CSEA breadcrumbing technique that facilitates the commission of CSAM offences.
Anonymous user profiles¶
2B.54 Anonymous user profiles, and the ability for unregistered users to post content anonymously, may allow perpetrators to avoid being personally identified by a service when sharing or accessing CSAM, and thereby avoid any potential content escalation or legal investigation. This may mean perpetrators are likely to choose services that allow users to share content without registering through creating an account, as this affords users a greater degree of anonymity. This was identified in a study into technologies used to commit child sexual abuse offences: 82% (more than 4 in 5) of the offenders surveyed indicated that anonymity was of at least moderate importance for conducting CSAM offences.443 2B.55 Perpetrators may leverage various forms of location-altering technology to hide their location and identity, to enable them to conduct illicit activities anonymously and therefore evade detection. GeoComply have highlighted how cybercriminals are using anonymising technology, such as Virtual Private Networks (VPNs), to commit sexual offences against children online.444
User networking¶
User groups¶
2B.56 Perpetrators can take advantage of user groups and use them as spaces to exchange CSAM, as well as ideas, advice, and tradecraft tips regarding abusive behaviour. The NSPCC noted that CSAM can be shared through online communities, and this behaviour can become “normalised or even encouraged” as like-minded people who share a sexual interest in children connect online.445 For some closed user groups, sharing CSAM, including new, first-generation CSAM, is sometimes a condition of entry. Protect Children also found that perpetrators were joining thematic communities online that posted and traded violent material.446
442 NSPCC response to Ofcom 2022 Call for evidence: First phase of online regulation.
443 Steel, C., Newman, E., O’Rourke, S. and Quayle, E., 2022. Technical Behaviours of Child Sexual Exploitation Material Offenders, Journal of Digital Forensics, Security and Law, 17(1). [accessed 18 November 2024] 444 VPNs are Virtual Private Networks: GeoComply Solutions 2024 response Ofcom Illegal Harms consultation. 445 NSPCC, 2019. Online abuse: learning from case reviews. [accessed 30 August 2023]. 446 Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 96
2B.57 In addition to sharing image CSAM directly within user groups, perpetrators also share links. Large virtual communities of offenders have been seen to share millions of items of CSAM indirectly via hyperlinks.447 2B.58 There is evidence to suggest user groups are forming that are dedicated to the abuse of AI tools to generate CSAM, which may increase the rate and ease by which the technology is exploited for these purposes. Newly created sections of online forums have been observed where members advise and request information on acquiring child sexual abuse-related material from AI systems.448
User communications¶
Livestreaming¶
2B.59 As has been introduced previously, livestreamed CSEA is a widespread problem, both in the UK and globally. This is where offenders view, comment on and direct the sexual abuse of children, in real time. It could be through one-to-one conversations (video call) or content that is broadcast live to a wider online audience. There is a substantial evidence base detailing the role that livestreaming plays in the commission of sexual exploitation of children, and offences relating to child sexual exploitation are discussed in chapter 9: Human trafficking and chapter 10: Unlawful immigration. 2B.60 Livestreaming can be used by perpetrators to evade detection because once the livestream is over, unless it was recorded, there may be little evidence of it. This risk has been noted by WeProtect, who have also highlighted how most platforms do not monitor livestreams.449 2B.61 The risk of livestreaming interacts with screen capturing and recording functionalities, as it allows for the livestreaming of CSEA to be used to create CSAM. The IWF’s report on livestreaming analysed over 2,000 indecent images of children taken from livestreams and 98% of these showed children who appeared to be 13 years or under.450 2B.62 Livestreaming in conjunction with messaging functionalities could present added risks, as it allows perpetrators to make specific requests while an offence is taking place. The NSPCC found evidence that children who livestream are sometimes asked to perform sexual acts.451 Of those children who livestream, 6% had received requests to change or remove their clothes.452 2B.63 As previously discussed, cases of livestreamed child sexual abuse have often been identified as being streamed from South-East Asia, with perpetrators in Western countries (including
447 Westlake, B. G., & Bouchard, M., 2016. Liking and hyperlinking: Community detection in online child sexual exploitation networks. Social science research, 59, pp.23-36. 448 Active Fence, 2023. How predators are abusing generative AI. [accessed 12 June 2024]. 449 In August 2022, the Australian e-Safety Commissioner issued the first mandatory transparency notices to Microsoft, Skype, Snap, Apple, Meta, WhatsApp, and Omegle, four of which have livestreaming or video call/conferencing services (note Omegle is no longer an active service). Responses revealed that of these four, three do not currently use tools to detect livestreamed child sexual abuse or exploitation: We Protect Global Alliance response to November 2023 Illegal Harms Consultation.
450 IWF, 2018. Trends in Online Child Sexual Exploitation: Examining the Distribution of Captures of Live-streamed Child Sexual Abuse. [accessed 22 September 2023]. 451Although not specified, it is reasonable to assume that these requests were received via messaging functionalities either publicly or privately, depending on the functionalities of the services being used and the tactics used by the perpetrator. 452 24% of all children have done a livestream broadcast: NSPCC, 2018. Livestreaming and video-chatting. [accessed 22 September 2023]. 97
the UK) accessing the material, generally in exchange for payment. 453 Globally, the UK has previously been estimated to be the third largest consumer of this form of livestreamed child sexual abuse.454
2B.64 The livestreaming of child sexual abuse does not solely take place in South-East Asia. The IWF’s research on livestreaming encountered many captures of livestreamed child sexual abuse which involved white girls, from apparently relatively affluent Western backgrounds, often appearing to be alone in their bedrooms.455
Direct messaging¶
2B.65 Direct messaging can allow perpetrators to share CSAM with one another. Interpol found that there was an increase in the volume of CSAM circulating via direct messaging on private messaging services or ‘message applications’ during the COVID-19 pandemic in 2020.456 2B.66 As outlined above, direct messaging is also a risk in the context of livestreaming, as it allows perpetrators to make specific requests while an offence is taking place.
Encrypted messaging¶
2B.67 Encrypted messaging makes the exchange of CSAM hard to detect,457 which may increase the likelihood perpetrators will seek out spaces with encrypted messaging to disguise their activity. Protect Children reported that messaging apps are often favoured by offenders due to the perceived security and privacy offered by end-to-end encryption, which allows them to commit crimes with apparent reduced fear of detection or law enforcement presence.458 2B.68 Increased use of encrypted messaging may make the detection and reporting of CSAM more difficult. The IWF reported that Meta’s suspected CSAM reports dropped by 58% between 2020 and 2021, during which time the IWF noted that Meta stopped ‘voluntarily scanning’ its services. The IWF has made the case that this scenario is similar to the situation where end-to-end encryption is rolled out and automated detection tools no longer work.459 The NCA has estimated that most reports currently provided to UK police will be lost with the introduction of end-to-end encryption, impacting 92% of reports from Facebook and 85% from Instagram.460 2B.69 Evidence suggests increased use of end-to-end encryption may have a number of adverse impacts on child safety. Protect Children reported that it would hinder law enforcement efforts to identify and rescue victims and survivors and make it ‘virtually impossible’ to
453 International Justice Mission, 2020. Online Sexual Exploitation of Children in the Philippines: Analysis and Recommendations for Governments, Industry, and Civil Society. [accessed 22 September 2023]. 454 IICSA, 2020. The Internet: Investigation Report. [accessed 22 September 2023]. 455 IWF, 2018. Trends in Online Child Sexual Exploitation: Examining the Distribution of Captures of Live-streamed Child Sexual Abuse. [accessed 23 August 2023]. [Note: this research was funded by Microsoft]. 456 Interpol, 2020. Threats and trends child sexual exploitation and abuse: COVID-19 impact. [accessed 22 September 2023]. 457 The exact scale of sharing and distribution of CSAM over encrypted messaging is difficult to quantify, as it cannot be tracked across services. Services offering end-to-end encryption have no means of accessing encrypted content. As such, technologies intended to mitigate the harm (such as hashing technology and content classifiers) cannot be applied within encrypted spaces and illegal content cannot be detected.
458 Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 459 IWF, 2022. Not all Encryption is the same: social media is not ready for End-to-End Encryption. [accessed 30 August 2023]. 460 NCA, 2024. European police chiefs call for end-to-end encryption roll out to include public protection measures. [accessed 8 July 2024]. 98
detect and remove CSAM, allowing for the cycle of revictimisation to continue. The report also noted it may contribute to the online disinhibition effect, as offenders become confident that their illegal activity cannot be detected.461
Ephemeral messaging¶
2B.70 Ephemeral messaging functionalities can make it harder for CSAM to be detected due to the disappearing nature of the content. 2B.71 Ephemeral messaging can be used by perpetrators to coerce children into producing and sharing sexual images, reassuring them that by using ephemeral messaging the image cannot be saved. However, it is common that perpetrators receiving such messages will screenshot the image. 2B.72 Ephemeral messaging has been found to facilitate SGII being shared consensually, as well as distributed non-consensually. Young people using these features may believe that their images are safer by sharing them in this format, in that there will be no permanent record of them, however the evidence suggests that users can deploy tactics to circumnavigate this feature.462
Group messaging¶
2B.73 Perpetrators tend to operate within groups or networks which trade content with each other. Group messaging functionalities can be used as part of these networks to share CSAM, enabling the spread of CSAM to multiple perpetrators. Research has found that abusers traded and shared CSAM in group chats on messaging applications.463 2B.74 Group messaging encompasses ‘chat functions’ which can be used by networks of perpetrators to share CSAM URLs with one another. Services may vary in their level of oversight or moderation of these channels, which presents a further risk if content sharing is not detected by human or automated moderation systems. 2B.75 Group messaging functionalities can be used to non-consensually share SGII to large groups
Commenting on content¶
2B.76 The ability to comment on content can be used to ‘breadcrumb’, whereby offenders use legal content to create a trail to direct like-minded individuals to illegal content, and thereby
461 The report described how this may result in the emergence of large-scale CSAM communities: Protect Children (Lapsia, S.), 2024. Tech Platforms Used by Online Child Sexual Abuse Offenders. [accessed 20 June 2024]. 462 Revealing Reality, 2022. Not Just Flirting: The unequal experiences and consequences of nude image-sharing by young people. [accessed 22 September 2023].
463 Steel, C., Newman, E., O’Rourke, S. and Quayle, E., 2020. An Integrative Review of Historical Technology and Countermeasure Usage Trends in Online Child Sexual Exploitation Material Offenders, Forensic Science International: Digital Investigation, Volume 33. [accessed 18 November 2024]. 464 Revealing Reality, 2022. Not Just Flirting: The unequal experiences and consequences of nude image-sharing by young people. [accessed 22 September 2023]. 465 Revealing Reality, 2022. 99
facilitate perpetrators’ access to CSAM.466 Some perpetrators use livestreaming comment functionality to start sexualised conversations with children and/or incite children to engage in sexual activity, sometimes including other children, in real time. The NSPCC found that of those children who livestreamed, 6% (more than 1 in 20) had received requests to change or remove their clothes.467
Posting content (text, images, video)¶
2B.77 The ability to post content, in this case text, videos and images, is a key enabler of the commission of CSAM offences. Abusers can post visual CSAM, and links or URLs to CSAM, on both open and closed channels of communication. The IWF reported that in 2022, 77% of the CSAM reports it dealt with were from services which hosted images.468 These services typically allow users to post images which can subsequently be shared through a unique URL. Evidence indicates that such URLs are often ‘embedded’, presumably by being posted, on discussion forums (see Risk factor: service type section for more information).
Transactions and offers¶
Posting goods or services for sale¶
2B.78 The use of CSAM within an advert is a CSAM offence in itself, as well as enabling the commission of further CSAM offences. 2B.79 Perpetrators arranging the livestreaming of child sexual abuse for offenders to purchase may use online functions where goods and services are posted. Facilitators may include CSAM in their posts to advertise and attract offenders.
Online payments and cryptocurrency payments¶
2B.80 The ability to make online payments, as well as cryptocurrency payments, can enable CSAM offences. Cryptocurrencies or other exchange mechanisms, like vouchers or payment codes, can enable offenders to buy CSAM anonymously and evade detection. The IWF noted that the number of websites found to accept cryptocurrency payments for CSAM has doubled in most years since 2015.469 Other potential payment options for CSAM include direct payment mechanisms, such as credit card or money transfer services.
Content exploring¶
Hyperlinking¶
2B.81 URLs, both in the form of hyperlinks and plain text, can be used by perpetrators to share CSAM between other individuals or more widely. Perpetrators can create links to CSAM stored on a file-storage and file-sharing service and share these across forums and in areas
466 WIRED (Orphanides, K. G.) 2019. On YouTube, a network of paedophiles is hiding in plain sight. [accessed 22 September 2023]. 467 Survey of nearly 40,000 children aged 7 – 16 years old: NSPCC, 2018. Livestreaming and video-chatting. [accessed 22 September 2023].
468 “These sites provide ‘storage’ for images which either appear on dedicated websites or are shared within forums”. Source: IWF, 2023. The Annual Report 2022 #Behind the Screens: A deep dive into the digital and social emergency happening #BehindTheScreens, in children’s bedrooms. [accessed 22 September 2023]. 469 IWF, 2022. Websites offering cryptocurrency payment for child sexual abuse images ‘doubling every year’. [accessed 22 September 2023]. 100
of otherwise legitimate services. Evidence has shown that large virtual communities of offenders have been seen to share millions of items of CSAM indirectly via hyperlinks.470
2B.82 Hyperlinks can facilitate access to CSAM as they can be used to direct abusers to CSAM hosted on third-party sites. The NSPCC refers to this as ‘digital breadcrumbing’, where abusers use services to signpost other abusers to CSAM hosted on other sites.471 This includes the use of Quick Response (QR) codes, working in a similar way to hyperlinks.472 2B.83 The volume of link-sharing related to child sexual abuse also appears to be increasing, with offenders seemingly moving partly away from curating personal collections of CSAM and instead look to access ‘on-demand’ CSAM.473 2B.84 As noted previously, there is evidence of links to CSAM content being posted on social media sites in a ‘scattergun’ approach. The spamming of links to CSAM material drives up web traffic, and income, for those hosting CSAM content.474
Building lists or directories¶
2B.85 The ability to create lists or directories in folders and save them on file-sharing services can be used by perpetrators to collect particular kinds of CSAM. Using hyperlinks, perpetrators can then easily share their collections with other perpetrators.
User-generated content searching¶
2B.86 Autocomplete suggestions in a U2U service’s search box can suggest searches for CSAM content. In 2018, Facebook’s autocomplete search terms were found to suggest child abuse videos and ‘under-age girls performing sex acts.’475 2B.87 In addition, some offenders use search functions on user-to-user pornography services to search for terms associated with CSAM.476
Content editing¶
Editing visual media¶
2B.88 As introduced previously, generative AI is being used by perpetrators to edit images and videos to produce CSAM. Perpetrators may use deepfakes, forms of audio-visual content that have been generated or manipulated using AI. Non-CSAM deepfakes are discussed in the Intimate image abuse chapter. 2B.89 The ability to edit images and videos can be used alter legal pornography, making participants appear as children, generating new CSAM material. Deepfake and GenAI technology be used to produce this type of CSAM, meaning that services allowing users to
470 Westlake, B. G., & Bouchard, M., 2016. Liking and hyperlinking: Community detection in online child sexual exploitation networks. Social science research, 59, pp.23-36. 471 NSPCC, 2022. Time to act: An assessment of the Online Safety Bill against the NSPCC’s six tests for protecting children. [accessed 30 August 2023]. 472 NSPCC response to Ofcom 2022 Call for evidence: First phase of online safety regulation. 473 WeProtect, 2023. Link-sharing and child sexual abuse: understanding the threat. [accessed 24 September 2024]
474 IWF, 2022. Public warned as ‘disturbing’ new trend risks exposure to child sexual abuse material online. [accessed 30 August 2023]. 475 Hern, A., 2018. Facebook apologises for search suggestions of child abuse videos, The Guardian, 16 March. [accessed 30 August 2023]. 476 IWF, 2022. Internet Watch Foundation, Stop It Now, and Pornhub launch first of its kind chatbot to prevent child sexual abuse. [accessed 19 November 2024]. 101
create such content through deepfake, GenAI or other content editing functionalities present risks.
Content storage and capture¶
Downloading content¶
2B.90 The ability to download content allows perpetrators to store and view local copies of CSAM on their computers and devices, as well as share it with others. This functionality enables perpetrators to build very large collections of CSAM. Indeed, the NCA has found some perpetrators who have downloaded over a million child sexual abuse images to their devices.477
Screen capturing or recording¶
2B.91 There is evidence of perpetrators capturing images from livestreams and capturing images of SGII videos from victims and survivors who have been groomed and coerced, which can then be distributed to other sites online.478 Over a three-month period in 2017, the IWF found 2,082 child sexual abuse captures from livestreams online. Of these, 96% of the images were of children on their own, typically in a house; 98% of the images depicted children assessed as being 13-years-old or under; and 40% of the images were categorised as Category A or B. 479
Recommender systems¶
Content recommender systems¶
2B.92 Recommender systems generally rely on user behaviour, such as viewing history, as an input into personalised content recommendations. As such, recommender systems are a risk factor in CSAM offences, as it is possible that a service’s recommender system could suggest CSAM-related content to users who are actively viewing CSAM videos. In these instances, CSAM must be present in the content pool that the recommender system is sourcing, ranking, and serving content to users from. 2B.93 There is evidence of users being recommended inappropriate, but not necessarily illegal, content.480 Salter and Hanson described how, if YouTube detects a user who seeks out and watches content of young children, the recommender system generates a playlist of similar content.481
477 For example: Luck, F., 2023. Former GP caught with 1.2m indecent images of children jailed. BBC News, 23 February. [accessed 30 August 2023]. 478 While users can often record or capture content using third-party services, screen recordings and captures are shared on U2U services as user-generated content and some U2U services have dedicated screen recording and screen capturing functionalities. 479 IWF, 2018. Trends in Online Child Sexual Exploitation: Examining the distribution of lives-streamed child sexual abuse. [accessed 22 September 2023]
480 Fisher, M. and Taub, A., 2019. ‘On YouTube’s digital playground, an open gate for pedophiles’, New York Times, 5 June. [accessed 27 September 2023]. 481 Salter, M. and Hanson, E., 2021. “I Need You All to Understand How Pervasive This Issue Is”: User Efforts to Regulate Child Sexual Offending on Social Media. Chapter 42 in (Bailey, J., Flynn, A. and Henry, N.) Emerald International Handbook of Technology Facilitated Violence and Abuse, pp.729–748. 102
Risk factors: Business models and commercial profiles¶
Revenue models¶
Advertising-based model¶
2B.94 Advertising features on user-to-user services can be used by perpetrators of CSEA. Research by the IWF found that legitimate adverts can be inadvertently used to fund websites ‘dedicated to child sexual abuse’482 if those engaged in arranging the adverts (advertising agencies, brands and advertising exchanges) do not do enough to prevent their adverts’ placement on such sites.483
Commercial profile¶
Low-capacity and early-stage services¶
2B.95 Low-capacity services, and services that are earlier in their business development lifecycle may be at greater risk of being used by perpetrators to share CSAM. Evidence suggests that low-capacity and early-stage services may be at risk of enabling CSAM content. This is because they are less likely to have technical and financial resources for risk management (for example, investment in the automated and/or manual moderation processes necessary to identify and combat CSAM content). We consider that perpetrators may seek out these spaces to share and view CSAM undetected.
Growth strategy¶
2B.96 Evidence suggests that services which prioritise and emphasise growth, for example prioritising user growth, may put insufficient resources towards effectively moderating harmful content and preventing offenders from exploiting the service.
482 Home Office, 2018. Advertisers urged to help tackle online child sexual exploitation. [accessed 22 September 2023]. 483 The research found that “one in ten websites dedicated to child sexual abuse host adverts for legitimate brands, including some household names” and the preliminary research into a sample of child sexual exploitation websites found that 57 of 100 websites contained adverts: Home Office, 2018. Advertisers urged to help tackle online child sexual exploitation. [accessed 22 September 2023]. 103
Section 3 Hate¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for hate offences: how harm manifests online, and risk factors Hate offences can be experienced by many people, in particular minorities and other protected groups. The offences can be targeted at one or more individuals, or wider communities. Exposure to hateful content, even if it may not meet legal thresholds, can have a severe impact on those to whom it is directed. The psychological effects of hateful content include shock, anger, suicidal thoughts, shame, exhaustion and fear, which can lead to further behavioural changes. Other experiences include financial harm and reputational damage. There is also evidence to suggest that, in some contexts, exposure to hateful content can entrench prejudices and incite acts of violence. Service type risk factors: Different types of services are associated with a risk of hate offences. Social media services and online gaming services pose a particular risk of hate offences. Video-sharing services and private messaging services have also been identified as spaces that are commonly used to commit or facilitate offences related to hate, targeting minorities and other protected groups. User base risk factors: Both large and small services play a role in disseminating hateful content. Users can first build a community of like-minded individuals to share provocative content on a large social media service, without breaching the service’s terms and conditions for hateful conduct. They may then direct their user networks to smaller and less-moderated services. Users’ race and ethnicity, gender, religion, disability status and sexual orientation are all risk factors influencing someone’s experience of hateful content. But age may also play a role. Functionalities and recommender systems risk factors: Evidence shows that the ability to create anonymous user profiles is a risk factor, but this is a complex issue. Anonymity can provide individuals with an environment in which they can speak and act more radically and propagate harm towards other users. But in many instances, hateful content is shared by users who are identifiable but unknown to the target. Usernames on identifiable user profiles can also be used to reference hate, while the ability to edit them can allow perpetrators to avoid enforcement action by recreating terminated profiles with slight edits to the original username. 104
Content recommender systems are another risk factor. Because these systems are generally designed to optimise user engagement, in some circumstances they promote content which may be hateful in nature because such negative or inflammatory content tends to have increased user engagement compared to benign content. Additionally, because recommender systems offer personalised feeds, they can also promote ideas or ideologies that users have already engaged with, which can increase confirmation bias and create ‘filter bubbles’. Hate offences are often committed via direct responses or comments on posted content. This functionality can enable the amplification of hate, enabling multiple people to direct abuse towards a target. The ability to livestream is also a risk factor as hateful content can be broadcast in real time. While direct messaging can be used to carry out hate offences in a highly targeted manner. Other functionalities pose risks for hate offences. Evidence suggests that content tagging, using hashtags, is also a risk factor when services allow hashtags with hateful language to go unmoderated or unenforced. User groups can enable offenders to spread hateful content amongst like-minded users. Hyperlinks allow users to move easily from mainstream to more niche hateful spaces. Business model risk factors: Advertising-based revenue models with an incentive to maximise user engagement may sometimes advertently, or inadvertently, promote hateful content. However, advertisers can be sensitive to their adverts being associated with hateful content on a service and can use their economic leverage to require a service to protect against hateful content. Other revenue models may increase the risk in a different way; for instance, subscription models may be limited to a small group of like-minded users (subscribers) who share common views and so may be more tolerant of hateful content.
Introduction¶
Research into online hate mainly focuses on hateful content that is prohibited by the terms and conditions of given services. Given this, the evidence described in this chapter includes sources that may cover non-statutory definitions relating to hate.
Relevant offences¶
484 Section 18 of the Public Order Act 1986.
485 Section 19 of the Public Order Act 1986. 486 Section 21 of the Public Order Act 1986. 487 Section 29B of the Public Order Act 1986. 488 Section 29C of the Public Order Act 1986. 489 Section 29E of the Public Order Act 1986. 490 Sections 31 and 32 of the Crime and Disorder Act 1998. 106
transgender identity. Crimes can be prosecuted as a hate crime if the offender has demonstrated, or been motivated by, hostility on the basis of these characteristics, and this may also have implications for sentencing.491 We acknowledge that the experience of hateful offences is deeply personal and varies for each individual, and there may be a wider range of motivating factors not captured by the aforementioned definitions.
How hate offences manifest online¶
491 The Crown Prosecution Service, n.d. Hate Crime. [accessed 7 June 2023]. 492 Refers to offences that have been recorded as hate crimes (flagged as being motivated by at least one of the five centrally monitored hate crime strands) and also been flagged as an online crime. We note that this is likely to be a broader definition than the specific priority offences that we are considering in this chapter. 493 The Equality Act 2010 protects discrimination against someone with protected characteristics, which refers to: age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, and sexual orientation.
494 Home Office statistics on Online Hate Crimes were last given in 2017/2018, when experimental figures were reported for 30 out of 44 police forces. 495 Home Office, 2018. Hate crime, England and Wales 2017/18. [accessed 2 April 2023]. 496 For instance, in Northern Ireland, hate crime data from the Police Service of Northern Ireland shows the prevalence of sectarian hate. 107
characteristics such as race, religion, disability, sexuality, or gender identity.497 As a further, specific example, analysis conducted by the Woolf Institute estimates that on just one user-to-user service, 495,000 explicitly antisemitic posts are made viewable to UK users per year.498
497 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 11 November 2024]. 498 Community Security Trust and Antisemitism Policy Trust, 2021. X: the extent and nature of antisemitism on X in the UK. [accessed 18 June 2024]. 499 Ofcom, 2023. Qualitative research into the impact of online hate. [accessed March 2023]. 500 Williams, M. and Reya, M 2019. Hatred behind the screens: a report on the rise of online hate speech. [accessed 23 March 2023]. 501 Ofcom, 2023. Qualitative research into the impact of online hate. [accessed March 2023]. 502 The Alan Turing Institute (Vidgen, B., Chung, Y-L, Johansson, P., Kirk, H.R., Williams, A., Hale, S.A., Margetts, H., Röttger, P. and Sprejer, L., 2022. Tracking abuse on X against football players in the 2021-22 Premier League season. [accessed 27 September 2022]. 503 Kearns, C, Sinclair, G, Black, J, Doidge, M, Fletcher, T, Kilvington, D, Liston, K, Lynn, T. and Rosati, P. A scoping review of research on online hate and sport. Community and Sport. 11 (2) [accessed 19 January 2023]. 504 Institute for Strategic Dialogue and CASM Technology, 2024. Evidencing a rise in anti-Muslim and anti-migrant online hate following the Southport attack. [accessed 15 October 2024] 505 TellMAMA, Greatest Rise in Reported Anti-Muslim Hate Cases to Tell MAMA since Oct 7th. [accessed 22 October 2024] 506 Hope not Hate, 2024. Doubling Down on Division, Anti Muslim hatred in the UK since 7th October. [accessed 22 October 2024] 507 Community Security Trust, 2024. Antisemitic Incidents Report January-June 2024. [accessed 22 October 2024] 508 LGBTQ+ is the acronym recognised by Stonewall.
509 This term is recognised by the authors of the research to describe people whose gender is not the same as, or does not sit comfortably with, the sex they were assigned at birth – Stonewall, n.d. List of LGBTQ+ terms. [accessed 8 September 2023]. 510 Between February and April 2017, 5,375 lesbian, gay, bi and trans (LGBT) people across England, Scotland and Wales completed an online questionnaire about their life in Britain today. Source: Stonewall, 2017. LGBT in Britain: Hate crime and discrimination. [accessed 15 March 2023]. 108
witnessed homophobic, biphobic or transphobic abuse or behaviour online that was directed at other people.
Risks of harm to individuals presented by the hate offences¶
511 Respondents who reported having seen “Hateful, offensive or discriminatory content that targets a group or person based on specific characteristics like race, religion, disability, sexuality or gender identity; e.g. hate speech” online within the four weeks before completing the survey. Please note that this could include content which may not meet the threshold for illegal hate. Q. Which, if any, of the following have you seen or experienced online in the last 4 weeks? This includes any images, videos, audio or text, either comments, posts or messages you have seen and/or those shared directly to you. Source: Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 15 October 2024]. 512 See ‘Risk factors: user base’ section for more information.
513 Ofcom, 2023. Qualitative research into the impact of online hate. [accessed March 2023]. 514 Ofcom, 2023. 515 Ofcom, 2022. How people are harmed online: Testing a model from a user perspective. [accessed 2 November 2022]. 516 LGBT+ is the acronym recognised by Galop. 517 Galop (Hubbard, L.), 2020. Online Hate Crime Report 2020: Challenging online homophobia, biphobia and transphobia. [accessed 12 September 2022]. 109
not for racially motivated crimes. Although the results were inconsistent, they did “point to the potential for using online behaviour to identify offline risk”.518
Evidence of risk factors on user-to-user services¶
Risk factor: Service types¶
Social media services¶
518 Cahill, M., Migacheve, K., Taylor, J., Williams, M., Burnap, P., Javed, A., Liu, H,. Lu, H. and Sutherland, A., 2019. Understanding online hate speech as a motivator and predictor of crime. [accessed 8 June 2023]. 519 Council on Foreign Relations, 2018. Hate speech on social media: global comparisons. [accessed 22 May 2023]. 520 See section above on Risks of harm to individuals presented by the hate offences for more information. 521 With some exceptions, these are generally focused on content moderation metrics and government requests for user data and records. Source: Harling, A, Henesy D. and Simmance, E, 2023. View of Transparency Reporting: The UK Regulatory Perspective, Journal of Online Trust & Safety, 1(5). [accessed 22 September 2023].
522 Further information on metrics can be found on individual platforms’ published transparency reports. 523 Institute for Strategic Dialogue (O’Connor, C.), Hatescape: An In-Depth Analysis of Extremism and Hate Speech on TikTok. [accessed 22 October 2024]. 524 Poole, R, Giraud, E. and Quincey E, 2021. Tactical interventions in online hate speech: The case of #stopIslam, New Media and Society, 23(6) [accessed 12 Jan 2023]. 110
less-moderated social media services where hateful ideologies can be discussed more openly.525
Video-sharing services¶
Private messaging services¶
525 N, Velasquez., R, Leahy., N, Johnson Restrepo., Y, Lupu., R, Sear., N, Gabriel., O, K, Jha., B Goldberg. and N, F, Johnson., 2021. Online hate network spreads malicious COVID-19 content outside the control of individual social media platforms. Scientific Reports, 11 (11549). [accessed 20 February 2023]. 526 Reed, A, Whittaker, J, Votta, F. and Looney, S., 2019. Radical filter bubbles: social media personalisation algorithms and extremist content. [accessed 19 February 2023]. 527 A third of users (32%) said they had witnessed or experienced hateful content. Hateful content was most often directed towards a racial group (59%), followed by religious groups (28%), transgender people (25%) and those of a particular sexual orientation (23%). Source: Ofcom and Yonder, 2021. User Experience of Potential Online Harms within Video Sharing Platforms. [accessed 20 September 2024]. 528 For example, between April and June 2024 Google removed approximately 34,000 channels for “Hateful or abusive” violations of their community policy. Note that this accounted for only 1% of all channel removals in this time period. Source: Google, 2024. Transparency report. [accessed 20 September 2024]. 529 Institute of Strategic Dialogue, 2023. 43-fold increase in anti-Muslim YouTube comments following Hamas’ October 7 attack. Institute of Strategic Dialogue, 2023. Rise in antisemitism on both mainstream and fringe social media platforms following Hamas’ terrorist attack. [accessed 10th May 2024] Narratives of Hate: Post-7 October Antisemitism and Anti-Muslim Hate on Social Media. [accessed 18th June 2024].
530 ADL (Kumbleben, M., Woolley, S. and Engler, M.), 2020. Computational propaganda and the 2020 U.S. Presidential election: Antisemitic and anti-Black content on Facebook and Telegram.[accessed 2 November 2022]. 531ADL (Kumbleben, M., Woolley, S. and Engler, M.), 2020. 111
Online gaming services¶
Risk factors: User base¶
User base size¶
532 Ofcom, 2022. How people are harmed online: Testing a model from a user perspective. [accessed 2 November 2022]. 533 Research included an online survey of gamers, n=622, focus groups with six avid gamers and focus groups with six experts. United Nations Office of Counter-Terrorism, 2022. Examining the Intersection Between Gaming and Violent Extremism. [accessed 3 June 2023]. 534 Miller, C. and Silva, S., 2021. Extremists using video-game chats to spread hate, BBC, 23 September. [accessed 22 September 2023]..
535 Miller, C. and Silva, S., 2021. 536 The Alan Turing Institute (Vidgen, B., Margetts, H. and Harris, A.), 2019. How much online abuse is there? A systematic review of evidence for the UK. [accessed 12 July 2023]. 537 Velasquez, N., Leahy, R., Johnson Restrepo, N., Lupu, Y., Sear, R., Gabriel, N., Jha, O. K., Goldberg, B., and Johnson, N. F., 2021. Online hate network spreads malicious COVID-19 content outside the control of individual social media platforms, Scientific Reports, 11 (11549) [accessed 20 February 2023]. 112
User base demographics¶
538 Ofcom, 2024. Online Experiences Tracker - Wave 6. [accessed 11 November 2024]. 539 Some of the content seen by users may not meet the legal threshold for the relevant offences we are considering in this chapter. Source: Ofcom, 2022. Online Experiences Tracker Data tables waves 1 and 2. [accessed 18 July 2023]. 540 Multi-stage national probability sample of 2,000 people. 541 The Alan Turing Institute (Vidgen, B, Margetts. and H, Harris, A.), 2019. How much online abuse is there? A systematic review of evidence for the UK. [accessed 12 July 2023]. 542 Ofcom, 2024. Online Experiences Tracker - Wave 6. [accessed 11 November 2024].
543 The Alan Turing Institute (Vidgen, B., Margetts, H. and Harris, A.), 2019. 544 Keywords for toxic messages include racial and misogynistic slurs. 545 Glitch, 2023. The Digital Misogynoir Report: Ending the dehumanising of Black women on social media, p.35. [accessed 18th June 2024] 546 A term created by American sociologist Kimberlé Crenshaw to describe how people can face different kinds of discrimination at the same time due to their "intersecting” or overlapping personal characteristics. 113
of hateful abuse. For instance, Muslim women and Jewish women may be more susceptible to being targeted with hate due to their religious and gender characteristics, much like how Black women would be due to their racial and gender characteristics. Since intersectional hate manifests through the coming together of an individual’s or community's personal characteristics (such as race, religion, gender, and class) it creates a unique and more complex type of discrimination.
547 Ofcom, 2024. Online Experiences Tracker - Wave 6. [accessed 11 November 2024]. 548 Ofcom / Coventry Youth Activists meeting, 16 April 2024.
549 Ofcom, 2024. Online Experiences Tracker - Wave 6. [accessed 11 November 2024] – 18+ only, of the 26% who has seen hateful, offensive or discriminatory content 550 Galop (Hubbard, L.), 2020. Online Hate Crime Report 2020: Challenging online homophobia, biphobia and transphobia. [accessed 12 September 2022]. 551 Nationally representative survey of Americans, 1,974 respondents: ADL, 2020. Online hate and harassment report: The American experience 2020. [accessed 20 October 2022]. 114
religion-based harassment (from 11% to 22%) while race-based harassment had increased from 15% to 25%.
Risk factors: Functionalities and recommender systems¶
User identification¶
Anonymous user profiles¶
User profiles¶
552 Galop (Hubbart, L.), 2020. 553 Mondal, M, Silva, L. and Benevenuto, F., 2018. Characterising usage of explicit hate expressions in social media, New Review of Hypermedia and Multimedia, 24(2). [accessed 18 November 2022].
554 Revealing Reality, on behalf of the Department for Media, Culture and Sport. 2022. Abuse and Anonymity. [accessed 23 October 2024]. 555 eSafety Commissioner, n.d. Anonymity and identity shielding [accessed 22 May 2023]. 556 eSafety Commissioner, n.d. 557 eSafety Commissioner, n.d. 115
suggests that usernames have been used as a tool by users to spread racial slurs.558 Users have also returned to a service after enforcement action by slightly editing their username. The Institute for Strategic Dialogue (ISD) found that accounts which had been banned from TikTok sometimes returned to the service under an edited username.559 Users can often easily find the banned user via their username, even after the original account has been banned.560
User communications¶
Livestreaming¶
Direct messaging¶
Commenting on content¶
558 Institute for Strategic Dialogue (O‘Connor, C.), 2021. Hatescape: An in-depth analysis of extremism and hate speech on TikTok. [accessed 9 March 2023]. 559 Institute for Strategic Dialogue (O‘Connor, C.), 2021. 560 Institute for Strategic Dialogue (O‘Connor, C.), 2021. 561 Online Safety Act Network, 2024. Disinformation and disorder: the limits of the Online Safety Act. [accessed 30 October 2024]. 562 Zhou, Y. and Farzan, R., 2021. Designing to stop live streaming cyberbullying: A case study of Twitch live streaming platform. [accessed 22 September 2023].
563 Institute for Strategic Dialogue (O’Connor, C.), Hatescape: An In-Depth Analysis of Extremism and Hate Speech on TikTok. [accessed 22 October 2024]. 564 Zhou, Y. and Farzan, R., 2021. Designing to stop live streaming cyberbullying: A case study of Twitch live streaming platform. [accessed 22 September 2023]. 565 Ofcom, 2022. The Buffalo Attack: Implications for Online Safety. [accessed 18 January 2023]. 566 Ofcom, 2023. Qualitative research into the impact of online hate. [accessed 22 September 2023]. 116
trigger an increase in online hate speech facilitated by commenting functionalities. For example, ISD research found that in the wake of the Israel and Gaza conflict, anti-Muslim and antisemitic comments on posts published on a video sharing service rose sharply.567
Reacting to content¶
Posting content¶
567 Institute of Strategic Dialogue, 2023. 43-fold increase in anti-Muslim YouTube comments following Hamas’ October 7 attack. [accessed 15 October 2024]; Institute of Strategic Dialogue, 2023. Rise in antisemitism on both mainstream and fringe social media platforms following Hamas’ terrorist attack. [accessed 10th May 2024]; Institute of Strategic Dialogue, 2024. Narratives of Hate: Post-7 October Antisemitism and Anti-Muslim Hate on Social Media. [accessed 18th June 2024]. 568 Ofcom’s research into X abuse of Premier League football players found that many users send just one abusive tweet. Source: The Alan Turing Institute, (Vidgen, B., Chung, Y-L, Johansson, P., Kirk, H.R., Williams, A., Hale, S.A., Margetts, H., Röttger, P. and Sprejer, L.), 2022. Tracking abuse on X against football players in the 2021-22 Premier League season. [Accessed 27 September 2022]. 569 Galop (Hubbard, L.), 2020. Online Hate Crime Report 2020: Challenging online homophobia, biphobia and transphobia. [accessed 12 September 2022]. 570 Both terms refer to more than one person directing abusive comments towards an individual in a semi-co-ordinated manner.
571 Revealing Reality, on behalf of the Department for Media, Culture and Sport. 2022. 572 Poole, R., Giraud, E., and Quincey E., 2021. Tactical interventions in online hate speech: The case of #stopIslam, New Media and Society, 23(6). [accessed 12 Jan 2023]. 573 Institute for Strategic Dialogue and CASM Technology, 2024. Evidencing a rise in anti-Muslim and anti-migrant online hate following the Southport attack. [accessed 30 Sep 2024]. 117
hashtags and containing antisemitic comments can reach large audiences on these services.574
User networking¶
User groups¶
User-generated content exploring¶
Hyperlinking¶
Content tagging¶
574 CCDH, 2021. Failure to protect: How tech giants fail to act on user reports of antisemitism. [accessed 20 September 2023]. 575 Uyheng, J., Bellutta, D. and Carley, K., 2022. Bots amplify and redirect hate speech in online discourse about racism during the Covid-19 pandemic. Social Media + Society, 8(3). [accessed 4 September 2022]. 576 Revealing Reality, on behalf of the Department for Media, Culture and Sport. 2022.
577 Gaudette, T., Scrivens, R. and Venkatesh, V., 2020. The role of internet in facilitating violent extremism: Insights from former right-wing extremists. Terrorism and Political Violence. 7 (34). [accessed 22 September 2023]. 578 A guest appearance refers to a person collaborating with a service user and taking part in the video. They can often be notable online personalities or ‘influencers’. 579 Data & Society (Lewis, R.), 2018. Alternative influence: broadcasting the reactionary right on Youtube. [accessed 20 February 2023]. 118
services will promote certain trending topics to wider audiences.580 Some users promoting hate know how to exploit algorithms, increasing the risks of harm to other users, as content they would not normally search for is likely to appear on their feeds under ‘trending’ topics.581
User-generated content editing¶
Editing visual media¶
Recommender systems¶
Content recommender systems¶
580 Institute for Strategic Dialogue (O’Connor, C.), Hatescape: An In-Depth Analysis of Extremism and Hate Speech on TikTok. [accessed 22 October 2024]. 581 This relates to services which employ trending functionalities.
582 Antisemitism Policy Trust, 2021. Instagram: Bad Influence. [accessed 20 December 2022]. 583 BBC News, 2016. Man jailed for harassing Labour MP Luciana Berger. BBC News, 8 December. [accessed 22 September 2023]. 584 Institute for Strategic Dialogue (O’Connor, C.), Hatescape: An In-Depth Analysis of Extremism and Hate Speech on TikTok. [accessed 22 October 2024]. 585 CST, 2020. Hate fuel: The hidden online world fuelling far right terror. 11 June. [accessed 7 June 2023]. 119
recommender system may then promote this content, which could be divisive, untrue, or incendiary.586
Risk factors: Business models and commercial profile¶
Revenue models¶
Advertising-based revenue model¶
586 Munn, L., 2020. Angry by design: Toxic communication and technical architectures. Humanities and Social Sciences communications. 7 (53) [accessed 3 May 2023].
587 “Fringe” content was coded as radical content without justification of violence, may also include profanity laden nicknames that go beyond political discourse, or historical revisionism. 588 Whittaker, J., Looney, S., Reed, A., Votta, Fabio., 2021. Recommender systems and the amplification of extremist content. [accessed 10 October 2024] 589 Munn, L, 2020. 120
toxic communication and technical architectures590 sets out the argument that social media services which rely on advertising revenues try to increase these revenues through increasing user interaction with the platform. An effective way of doing this is to display, and facilitate, a large quantity of controversial content, which could include hateful content. Material that creates outrage or strong reactions can encourage user engagement and by extension, increase the amount of time spent viewing the material.
Subscription-based revenue models¶
590 Munn, L., 2020. Angry by design: Toxic communication and technical architectures. Humanities and Social Sciences communications. 7 (53) [accessed 3 May 2023]. 591 World Federation of Advertisers, 2020. Marketing leaders take action on harmful online content. [accessed 22 September 2023]. 592Conscious Advertising Network, n.d. About Us. [accessed 22 September 2023]. 593 Stanford (Thiel, D. and Mcain, M.), 2022. Gabufacturing dissent: An in-depth analysis of Gab. [accessed 3 May 2022]. 121
Section 4 Harassment, stalking, threats and abuse¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for harassment, abuse, threats, stalking and threatening communications offences: How harms manifest online, and risk factors This chapter covers offences relating to online harassment, abuse, threats, stalking or threatening communications which are unwanted behaviours that can cause alarm and distress to other individuals, or put them in fear of violence. Ofcom’s Online Experiences Tracker showed that 12% of people aged 13+ surveyed, reported they had witnessed ‘one off abusive behaviour or threats in the past four weeks.594 Additionally, the Oxford Internet Survey in 2019 reported that 27% of people had been exposed to abusive content. 595 They can cause significant harm to individuals. Psychological impacts can include mental and emotional distress, isolation, and feeling unsafe both online and offline. Service type risk factors: Research indicates that social media services are used to commit and facilitate various forms of abuse and harassment, including threats to kill, and misleading information that can result in violence. Evidence also indicates that users regularly experience severe abuse, including physical threats, stalking, and sustained harassment, on online gaming services. These offences are also perpetrated on private messaging services and online dating services. Userbase risk factors: Evidence suggests that anyone can be subjected to these behaviours. However, research shows that women are particularly subjected to harassment. In a study surveying 4500 women globally, 38% of women aged 18 to 74 stated that they had personal experiences of “online violence”.596 In Plan International’s study of over 14,000 girls and women aged 15 to 25, 58% of respondents reported they experienced some form of online harassment.597 Their exposure to these offences is often more prevalent, severe, and can cause greater effect, compared to men,
594 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024].
595 27% of respondents had seen cruel or hateful comments or images posted online. Sample of 2,000 people. Source: The Alan Turing Institute (Vidgen, B., Margetts, H., Harris, A.), 2019. How much online abuse is there? [accessed 28 September 2023]. 596 The Economist Intelligence Unit. 2021, ‘Measuring the Prevalence of Online gender-based violence against Women’, [accessed 29 July 2024]. 597 Plan International, 2020, State of the World’s Girls 2020: Free to be Online? [accessed 29 July 2024]. 122
especially among certain groups such as women in the public eye, or women in the online gaming community. Gender also intersects with age and race as a risk factor, with evidence suggesting that young women, and those in minority ethnic groups, are at highest risk of harassment and abuse. Functionalities and recommender system risk factors: Several functionalities of User-to-user (U2U) services can be used in specific ways to perpetuate harassment, stalking and violent threats. While anonymity can be a source of protection, evidence suggests that anonymous user profiles may encourage harmful contact by making users feel freer to violate social norms. User profiles, and the information that is often displayed on them, can help facilitate stalking. Perpetrators can also gain unauthorised access to victims and survivors’ accounts to impersonate them through their user profile. Cases of harassment and stalking often involve perpetrators creating multiple and often fake user profiles to contact individuals against their will and to be omnipresent in their lives. Perpetrators can circumvent blocking and moderation by creating new accounts and their associated user profiles, thereby continuing to harass, stalk or threaten victims and survivors, causing significant fear and distress. In some cases, perpetrators can leverage the user connections functionality by connecting with second and third-degree connections of the victim or survivor, to access content that is otherwise not publicly available, thereby giving a perpetrator visibility of a target's profile without connecting with them directly. User connections also enable perpetrators to build online networks through network recommender systems, which can be leveraged to facilitate harassment and abuse. Individual perpetrators can incite their network to join the abuse of an individual.598 Additionally, content recommender systems can play a role in spreading harassment, threats and abusive content and in increasing the risks of perpetrators engaging in this content. The ability to post or send location information can provide information that allows perpetrators to target and monitor victims and survivors for the purposes of harassment, stalking and threats of violence. Reposting or forwarding content can enable content to be circulated that is likely to provoke harassment or abuse from certain audiences. Abusive messages can also be communicated via direct messaging, in both private and public contexts. Abuse and harassment can also occur via comments on content. Perpetrators also exploit user tagging to harass their victims and survivors by incessantly tagging their usernames in abusive and threatening messages.
598 This is also known as a ‘pile-on’, in which an individual is attacked by a large number of users. This form of harassment can cause significant harm to victims and survivors. 123
Introduction¶
Relevant offences¶
4.6 Act.
599 Crown Prosecution Service, 2023. Stalking or Harassment. [accessed 28 September 2023]. 600 The offences of fear or provocation to violence and threats to kill will be discussed together. This is because evidence of these threats perpetrated on services often relates to general threats to violence. 124
601 Section 16 of the Offences against the Person Act 1861. 602 Section 38 of the Criminal Justice and Licensing (Scotland) Act 2010 (asp 13). 603 Section 4 of the Public Order Act 1986. 604 Section 4A of the Public Order Act 1986. 605 Section 5 of the Public Order Act 1986. 606 Section 2 of the Protection from Harassment Act 1997; Article 4 of the Protection from Harassment (Northern Ireland) Order 1997 (S.I. 1997/1180 (N.I. 9)).
607 Section 2A of the Protection from Harassment Act 1997; Section 39 of the Criminal Justice and Licensing (Scotland) Act 2010 (asp 13). 608 Section 4 of the Protection from Harassment Act 1997; Article 6 of the Protection from Harassment (Northern Ireland) Order 1997 (S.I. 1997/1180 (N.I. 9)). 609 Section 4A of the Protection from Harassment Act 1997. 125
Harassment can also be by two or more people against an individual, or harassment against more than one victim or a group.610
• Stalking similarly involves a course of conduct and is a specific form of harassment. It may be understood as a pattern of fixated, obsessive, unwanted and repeated (FOUR) behaviour which is intrusive.611 Examples of behaviours associated with stalking include following a person or contacting or attempting to contact a person.
How harassment, abuse, threats and stalking offences manifest online¶
610 Crown Prosecution Service, 2023. Stalking or Harassment. [accessed 28 September 2023]. 611 Crown Prosecution Service, 2023. 612 A survey conducted by the Suzy Lamplugh Trust, a UK-based charity specialising in stalking, found that 75% of respondents who had experienced stalking had experienced both online and offline stalking behaviours: Suzy Lamplugh Trust. 2021. Unmasking stalking: a changing landscape. [accessed 28 September 2023]. 613 Crown Prosecution Service, 2023. 126
partners614, while other data suggests that a significant proportion of intimate partner stalking victims are stalked during the relationship.615 616 Research by the Victims’ Commissioner for England and Wales found that victims of online abuse often experience multiple types of other harms – with victims of cyberstalking in particular experiencing more harms on average.617
Harassment, abuse and threats¶
614 Crown Prosecution Service, 2020. Stalking analysis reveals domestic abuse link. [accessed 10 October 2024]. 615 Paragon, 2024. Domestic abuse and stalking. [accessed 10 October 2024]. 616 The Domestic Abuse Commissioner also notes that stalkers in a domestic context (intimate partners or ex-intimate partners) ought to be recognised as a distinctive category in respect of the prevalence and risk posed to victims. Source: Domestic Abuse Commissioner’s response to November 2023 Illegal Harms Consultation. 617 The Victims’ Commissioner (Storry, M., Poppleton, S.), 2022. The Impact of Online Abuse: Hearing the Victims’ Voice. [accessed 25 July 2024].
618 Suzy Lamplugh Trust. 2021. Unmasking stalking: a changing landscape. [accessed 25 July 2024]. 619 Scottish Government. 2018. Scottish Crime and Justice Survey 2017/18. [accessed 28 September 2023]. 620 The Alan Turing Institute (Vidgen, B., Margetts, H., Harris, A.), 2019. How much online abuse is there? [accessed 28 September 2023]. 621 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024]. 127
online abuse and harassment.622 The Oxford Internet Survey in 2019 reported that 27% of people had been exposed to abusive content (including hate speech).623
Stalking¶
622 41% of respondents reported experiencing at least one of physical threats, stalking, sexual harassment, sustained harassment, purposeful embarrassment, or offensive name-calling. Note: Where possible, UK data has been used throughout this chapter. However, when this is limited, evidence for comparable cultures has been used, namely the US, Australia and Canada. Where evidence is not UK-based, this will be clearly stated. This US-based study from Pew Research includes single occasions as measures of harassment, although in the UK the harassment offence is a course of conduct occurring on two or more occasions. Source: Pew Research (Vogels, E.), 2021. The State of Online Harassment. [accessed 28 September 2023]. 623 27% of respondents had seen cruel or hateful comments or images posted online. Sample of 2,000 people. Source: The Alan Turing Institute (Vidgen, B., Margetts, H., Harris, A.), 2019. How much online abuse is there? [accessed 28 September 2023]. 624 The Economist Intelligence Unit. 2021, ‘Measuring the Prevalence of Online gender-based violence against Women’, [accessed 29 July 2024]. 625 Plan International, 2020, State of the World’s Girls 2020: Free to be Online? [accessed 29 July 2024]. 626 Pew Research (Vogels, E.), 2021. The State of Online Harassment. [accessed 28 September 2023]. 627 Gamergate, a networked harassment campaign that targeted women in the video game industry, is a high-profile example of this that is often cited when studying these issues. Source: Romano, A., 2021. What we still haven’t learned from Gamergate, Vox, 7 January. [accessed 29 July 2024].
628 Fox J., Cruz C., & Lee J., 2015. Perpetuating online sexism offline: Anonymity, interactivity, and the effects of sexist hashtags on social media. Computers in Human Behavior, 52, 436-442. [accessed 19 November 2024]. 629 Office for National Statistics, 2022. Stalking: findings from the Crime Survey for England and Wales. [accessed 28 September 2023]. 630 National Police Chief Council, 2024. Violence Against Women and Girls (VAWG) National Policing Statement. [accessed 29 July 2024]. 128
Research shows that stalking rarely occurs online only. Studies indicate that stalking which starts online tends to move offline and stalking that occurs predominantly offline will also have online elements. The National Stalking Helpline reports that 100% of the stalking cases reported to the helpline now involve a ‘cyber’ element, and a study into modes of cyberstalking and cyber harassment found that ‘proximal’ stalking which begins offline but also includes online elements was the most common among 278 victims in the UK. Information gained through online stalking (such as location coordinates from default geo-tagging of images shared on social media) can enable offline stalking and violence.631 632 633
Risks of harm to individuals presented by these offences¶
631 Suzy Lamplugh Trust, 2021. Unmasking stalking: a changing landscape. [accessed 25 July 2024]. 632 Brown A., Gibson M., & Short E. 2017. Modes of Cyberstalking and Cyberharassment: Measuring the negative effects in the lives of victims in the UK Annual Review of Cybertherapy and Telemedicine. [accessed 19 November 2024]. 633 Sheridan L., Blaauw E., &Davies G. 2003. Stalking: Knowns and Unknowns. Trauma, Violence & Abuse 4(2). [accessed 19 November 2024]. 634 Titherade, N. and Thomas, E., 2021. Stalking rises during Covid pandemic - police, BBC News, 13 July. [accessed 12 July 2023].
635 Suzy Lamplugh Trust, 2021. 636 Crown Prosecution Service, 2023. Stalking or Harassment. [accessed 28 September 2023]. 637 Refuge, 2021. Unsocial Spaces. [accessed 28 September 2023]. 638 Patton, D. U., Pryooz, D., Decker, S., Frey, W. R. and Leonard, P., 2019. When Twitter Fingers Turn to Trigger Fingers: a Qualitative Study of Social Media-Related Gang Violence. [accessed 28 September 2023]; Crest, 2022. Calouri, J., Hutt, O., Olajide, P. and Kirk, E, 2022; 129
Harassment, abuse and threats¶
639 The Crime Survey for England and Wales found that while almost one in ten men (9.5%) have experienced stalking, almost a quarter of women (23.3%) over the age of 16 have experienced this offence. The Crime Survey for England and Wales also collects information on the prevalence of stalking with an online element, and found that 8.3% of women had experienced this, and 3.1% of men. Source: Office for National Statistics, 2022. Stalking: findings from the Crime Survey for England and Wales. [accessed 28 September 2023]. 640 McEwan, T. E., Mullen, P. E., MacKenzie, R. D., Ogloff, J. R. P. 2009. Violence in stalking situations. Psychol Med. [accessed 13 September 2024]. 641 Victims Commissioner for England and Wales. The Impact of Online Abuse: Hearing the Victims’ Voice. [accessed 24 October 2024]. 642 This is when adjusted for socioeconomic status. Source: McManus, S., Bebbington, P. E., Tanczer, L., Scott, S. and Howard, L. M. 2021. Receiving threatening or obscene messages from a partner and mental health, self-harm, and suicidality: results from the Adult Psychiatric Morbidity Survey. Social Psychiatry and Psychiatric Epidemiology: the international journal for research in social and genetic epidemiology and mental health services. [accessed 29 July 2024]
643 Amnesty International, 2017. Social media can be a dangerous place for UK women. [accessed 28 September 2023]. 644 Data & Society Research Institute/CiPHR (Lenhart, A., Ybarra, M., Zickuhr, K. and Price-Feeney, M.), 2016. Online Harassment, digital abuse and cyberstalking in America. [accessed 28 September 2023]. 645 UNESCO (Posetti, J., Aboulez, N., Bontcheva, K., Harrison, J. and Waisbord, S.), 2020. Online Violence Against Women Journalists. [accessed 28 September 2023]. 130
found that 40% of victims and survivors of online harassment said they had experienced at least one of these types of isolation or disconnectedness as a result:646 • 27% of victims and survivors experienced trouble in a relationship or friendship because of something that was posted about them online; • 20% had shut down an online account or profile because of online harassment or abuse; and • 13% of victims and survivors felt less connected to information and 13% felt less connected to friends or family because their phone or internet use was limited because of harassment or abuse.
Stalking¶
646 Data & Society Research Institute/CiPHR (Lenhart, A., Ybarra, M., Zickuhr, K. and Price-Feeney, M.), 2016. Online Harassment, digital abuse and cyberstalking in America. [accessed 28 September 2023]. 647 Amnesty International, 2017. Social media can be a dangerous place for UK women. [accessed 28 September 2023]. 648 Data & Society Research Institute/CiPHR, 2016. 649 Pew Research, 2021. The State of Online Harassment. [accessed 28 September 2023]. 650 A study in Australia explored how ethnicity and gender intersect, with specific impacts (such as threats of deportation, ‘honour’ killing, or culturally-specific humiliation) affecting women with ethnically diverse backgrounds. Source: eSafety Commission Australia, 2019. eSafety for Women from Culturally and Linguistically Diverse Backgrounds. [accessed 28 September 2023]. 651 The authors reviewed 358 cases of homicide where the victim was a woman, and collected information to identify key stalking, control and risk markers in each case. They excluded cases where the victim was not specifically targeted, i.e. cases of mistaken identity, where the homicide outside the UK and circumstances could not be verified, or where the homicide occurred in the course of another crime such as a robbery. Source: Suzy Lumplugh Trust (Monckton Smith, J., Szymanska, K., and Haile, S.), 2017. Exploring the Relationship between Stalking and Homicide. [accessed 19 November 2024].
652 McFarlane, J., Campbell, J.C., Wilt, S., Ulrich, Y., & Xu, X. 1999. Stalking and Intimate Partner Femicide. Homicide Studies, 3(4), 300-316. [accessed 19 November 2024]. 131
61% of stalking victims are being or have been stalked by a former partner, with only 7% stating that the perpetrator was a stranger.653 The National Stalking Helpline further reports that “45% of those who contact the Helpline are being stalked by a former partner and further third have had some sort of prior acquaintance with their stalker.”654
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services¶
653 Suzy Lamplugh Trust, 2021. Unmasking stalking: a changing landscape. [accessed 29 July 2024]. 654 Suzy Lamplugh Trust, n.d. What is stalking? | Suzy Lamplugh Trust. [accessed 29 July 2024]. 655 Kaur, P., Dhir, A., Tandon, A., Alzeiby, E.A. and Abohassan, A.A., 2020. A systematic literature review on cyberstalking. An analysis of past achievements and future promises, Technological Forecasting and Social Change, 163. [accessed 28 September 2023]. 656 These impacts are described by a feminist writer and campaigner who faced an extensive online abuse and harassment: “At its height I struggled to eat, to sleep, to work. I lost about half a stone in a matter of days. I was exhausted and weighed down by carrying these vivid images, this tidal wave of hate around with me wherever I went... the psychological fall-out is still unravelling” Source: Criado Perez, C. 2015. Caroline Criado-Perez’s speech on cyber-harassment at the Women’s Aid conference. The New Statesmen, 27 September. [accessed 28 September 2023].
657 Suzy Lamplugh Trust, 2021. Unmasking stalking: a changing landscape. [accessed 29 July 2024]. 658 Short E. and Maple C., 2011. The impact of cyberstalking: review and analysis of the ECHO pilot project. Proceedings of the IADIS International Conferences – Web Based Communities and Social Media. [accessed 29 July 2024]. 659 Flynn A., Powell A., Hindes S. (2021). Technology-facilitated abuse: A survey of support services stakeholders. [accessed 29 July 2024]. 132
services, but it happens a lot more on social media services.660 According to UNESCO, “social media companies are the main enablers of online violence against women journalists”, which includes sexual violence and ‘gendered profanities’.661
Online gaming services¶
Online dating services and private messaging services¶
660 Some research shows that stalking through social media services is becoming more common. A study by the Suzy Lamplugh trust found that stalking using social media jumped from 59% before the first Covid-19 pandemic lockdown to 82% after it. Note small sample of 111 victims of stalking in survey. Source: Suzy Lamplugh Trust. 2021. Unmasking stalking: a changing landscape. [accessed 28 September 2023]. The NPCC reports that the majority of online-facilitated VAWG incidents that are reported to the police is classified as stalking, and is committed via social media services. See ‘How harms manifest online’ section for more information. Source: National Police Chiefs Council, 2023. Violence Against Women and Girls: Strategic Threat Risk Assessment 2023. [accessed 28 September 2023]. 661 Posetti, J., Shabbir, N., Maynard, D., Bontcheva, K. and Aboulez, N., 2021. The chilling effect. Global trends in online violence against women. [accessed 28 September 2023]. 662 Anti-defamation League, 2021. Hate is No Game: Harassment and Positive Social Experiences in Online Games 2021. [accessed 28 September 2023]. 663 Marwick, A, 2021. Morally Motivated Networked Harassment as Normative Reinforcement. [accessed 28 September 2023]. 664 See ‘Risk factors: functionalities and recommender systems’ section for more information. Source: McLean, L. and Griffiths, M. D., 2018. Female Gamers’ Experience of Online Harassment and Social Support in Online Gaming: A Qualitative Study, International Journal of Mental Health and Addiction, 17, 970-994. [accessed 28 September 2023].
665 McLean, L. and Griffiths, M. D., 2018. Female Gamers’ Experience of Online Harassment and Social Support in Online Gaming: A Qualitative Study, International Journal of Mental Health and Addiction, 17, 970-994. [accessed 28 September 2023]. 666 See ‘Risk factors: user base’ section for more information. Source: Cote, C, 2017. ‘‘I Can Defend Myself’’: Women’s Strategies for Coping With Harassment While Gaming Online, Games and Culture, 12(2). [accessed 28 September 2023]. 667 Gray, K.L. 2021. Intersectional Tech: black users in digital gaming. LSU Press. 133
women (aged 18 to34), with 57% receiving an explicit message, 44% being called an offensive name, and 19% receiving threats to physical harm.668
Risk factors: user base¶
User base demographics¶
668 Online dating users refers to respondents who say they have ever used an online dating site or app (n=2,094). Source: Anderson, M., Vogels, E., Turner, E, 2020. The Virtues and Downsides of Online Dating. [accessed 28 September 2023]. 669 Suzy Lamplugh Trust, 2021. Unmasking stalking: a changing landscape. [accessed 28 September 2023]. 670 See ‘Risk factors: functionalities and recommender systems’ section for more information. Source: UNESCO (Posetti, J., Aboulez, N., Bontcheva, K., Harrison, J. and Waisbord, S.), 2020. Online Violence Against Women Journalists. [accessed 28 September 2023].
671 Comprises waves 1 and 2 combined data set. Source: Ofcom, 2023. Experiences of using online services. [accessed 28 September 2023]. 672 According to this study, 64% of US adults aged 18-30 had experienced any form of harassment, compared to 41% for the whole adult population. Nearly half (48%) of 18-30 year olds had experienced behaviours classified by the study as more severe. These include being include being physically threatened, stalked, sexually harassed or harassed for a sustained period of time. Source: Pew Research, 2021. The State of Online Harassment. [accessed 28 September 2023]. 134
assault.673 A study of misogynoir674 on several social media platforms found that 20% of the one million posts collected about women were highly toxic.675
673 Amnesty International, 2017. Social media can be a dangerous place for UK women. [accessed 28 September 2023]. 674 “Misogynoir” is a term used to describe discrimination against Black women. 675 Glitch, 2023. The Digital Misogynoir Report: Ending the dehumanising of Black women on social media. [accessed 29 July 2024] 676 Amnesty International, 2017. Social media can be a dangerous place for UK women. [accessed 28 September 2023]. 677 UNESCO (Posetti, J., Aboulez, N., Bontcheva, K., Harrison, J. and Waisbord, S.), 2020. Online Violence Against Women Journalists. [accessed 28 September 2023]. 678 Inter-Parliamentary Union, 2016. Sexism, harassment and violence against women parliamentarians. [accessed 28 September 2023]. 679 Amnesty International UK. 2017. Black and Asian Women MPs Abused More Online. [accessed 29 July 2024].
680 Office for National Statistics, 2022. Stalking: findings from the Crime Survey for England and Wales. [accessed 28 September 2023]. 681 Suzy Lamplugh Trust, 2021. Unmasking stalking: a changing landscape. [accessed 28 September 2023]. 682 See ‘Risk factors: functionalities and recommender systems’ section for more information. Source: McLean, L. and Griffiths, M. D., 2018. Female Gamers’ Experience of Online Harassment and Social Support in Online Gaming: A Qualitative Study, International Journal of Mental Health and Addiction, 17, 970-994. [accessed 28 September 2023]. 135
suggesting that the risks of harm to individuals are higher for players who present as female in these environments.683
683 Cote, C, 2017. ‘‘I Can Defend Myself’’: Women’s Strategies for Coping With Harassment While Gaming Online, Games and Culture, 12(2). [accessed 28 September 2023]. 684 Office for National Statistics, 2022. Stalking: findings from the Crime Survey for England and Wales. [accessed 28 September 2023]. 685 Pew Research, 2021. The State of Online Harassment. [accessed 28 September 2023]. 686 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024]. 687 Pew Research, 2021. 688 Pew Research, 2021. 689 Black women are 84% more likely to be targets of abusive tweets than white women and 60% more likely to receive problematic tweets. Source: Glitch response to 2022 Ofcom Call for Evidence: First phase of online safety regulation; A US study collected a sample of Twitter data between 2015 and 2017. Analysing these 25,000 tweets, they found it took on average 18 seconds to detect an insulting, negative tweet directed at Black women, and 16 seconds to locate such a message aimed at Latina women, leading the authors to conclude that aggressive messages towards women of colour were easily accessible and visible on the social media platform. Source: Francisco, S. and Felmlee, D.H., 2022. What Did You Call Me? An Analysis of Online Harassment Towards Black and Latinx Women, Race and Social Problems, 14, 1-13. [accessed 28 September 2023].
690 Glitch, 2023. The Digital Misogynoir Report: Ending the dehumanising of Black women on social media. [accessed 29 July 2024] 691 Glitch, 2023. 692 Office for National Statistics, 2022. 693 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024]. 136
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles¶
694 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024]. 695 FRA EU Agency for Fundamental Rights, 2020. LGBTI Survey Data Explorer. [accessed 28 September 2023]. 696 LGBT Youth Scotland. Life in Scotland for LGBT Young People in 2022. [accessed 24 October 2024]. 697 Note this data set comprised Wave 1 and 2 of the Online Experiences Tracker in order to provide a large enough sample size for this analysis. Source: Ofcom, 2023. Experiences of using online services. [accessed 28 September 2023].
698 Data & Society Research Institute/CiPHR (Lenhart, A., Ybarra, M., Zickuhr, K. and Price-Feeney, M.), 2016. Online Harassment, digital abuse and cyberstalking in America. [accessed 28 September 2023]. 699 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024]. 700 Clevenger, S. and Gilliam, M, 2020. Intimate partner violence and the internet: Perspectives. Chapter in Holt, T. J. and Bossler, A. M. (eds.), The Palgrave Handbook of International Cybercrime and Cyberdeviance. [accessed 28 September 2023]. 137
monitored on social media services; the US Department of Justice in 2019 found this to be true of 31.9% of stalking victims and survivors.701
Fake user profiles¶
Anonymous user profiles¶
701 U.S Department for Justice (Morgan, R. and Truman, J.), 2022. Stalking Victimization, 2019. [accessed 28 September 2023]. 702 Yardley, E. 2021. Technology-facilitated domestic abuse in political economy: a new theoretical framework, Violence against women, 27 (10). [accessed 28 September 2023]. 703 Two qualitative studies from Australia found a high prevalence of harassment through creating multiple social media profiles in their samples of domestic abuse victims and survivors. The woman’s ex-partner created a false profile, using pictures from when they were together, and sent messages to her friends to discredit her reputation. This left her fearing for her safety, leading her to shut herself off from her family and social media altogether. In another example, the woman would block her former partner on social media, but he would “delete his whole account, not just deactivate, delete the whole account… so therefore that email address was no longer on the platform’s system. Then he could go and make a new account and contact me again.” Source: eSafety Commission Australia, 2019. eSafety for Women from Culturally and Linguistically Diverse Backgrounds. [accessed 28 September 2023]; Dragiewicz, M., Harris, M., Woodlock, D., Salter, M., Easton, H., Lynch, A., Campbell, H., Leach, J. and Milne, L., 2019. Domestic Violence and communication technology: victim experiences of intrusion, surveillance and identity theft. [accessed 28 September 2023]. 704 Refuge, 2022. Marked as Unsafe. [accessed 28 September 2023]. 705 Kale, S, 2022. 11 years, 10 arrests, at least 62 women: how did Britain’s worst cyberstalker evade justice for so long? The Guardian, 30 March. [accessed 28 September 2023]. 706 The writer and campaigner Caroline Criado Perez led a campaign to get more women on bank notes yet received intense backlash; at its peak receiving 100 to 200 tweets a minute, many of them abusive. Source: Criado Perez, C., New Statesmen, 2015. Caroline Criado-Perez’s speech on cyber-harassment at the Women’s Aid conference. The New Statesman, 27 September. [accessed 28 September 2023].
707 McGlynn, C. and Woods, L., 2022. Violence against women and girls (VAWG) Code of Practice. [accessed 28 September 2023]. 708 Suler, J, 2004. The online disinhibition effect. [accessed 28 September 2023]. 138
being unidentifiable as individuals, aligned themselves strongly with group identities and norms in ways that enabled trolling behaviour.709
User networking¶
User connections¶
709 Synnott, J., Coulias, A. and Loannou, M., 2017. Online trolling the case of Madeleine McCann. [accessed 28 September 2023]. 710 UNESCO (Posetti, J., Aboulez, N., Bontcheva, K., Harrison, J. and Waisbord, S.), 2020. Online Violence Against Women Journalists. [accessed 28 September 2023]. 711 Lapidot-Lefler, N. and Barak, N., 2012., Effects of anonymity, invisibility, and lack of eye-contact in toxic online disinhibition. Computers in human behaviour, 28(2). [accessed 28 September 2023]. 712 Ritter, 2014. Deviant Behavior in Computer-Mediated Communication: Development and Validation of a Measure of Cybersexual Harassment, Journal of computer-mediated communication, 19 (2). [accessed 28 September 2023]. 713 An example includes a user falsely accused of being a Russian disinformation theorist by a highly followed conspiracy theorist on Twitter. He describes: “when someone has 25K Twitter followers, they pile on really quickly, and it sort of becomes, especially in this case where it’s very conspiratorially-minded thinking, that the accusations and the allegations sort of start to compound and build up on each other.” He described being ‘really upset’ and ‘very alone’ as a result. This example shows how user networks can amplify harassment to a level that causes fear and distress to victims. Source: Marwick, A, 2021. Morally Motivated Networked Harassment as Normative Reinforcement. [accessed 28 September 2023].
714 Some posts provide step-by-step instructions for creating a believable fake profile and befriending accounts connected to targeted individuals, allowing the perpetrator to access content visible to ‘friends-of-friends'. Strategies can be highly targeted. As one poster describes: “In my neck of the woods there are a lot of local bars that have 1000+ friends and guess 139
User tagging¶
User communication¶
Livestreaming¶
Direct messaging¶
Commenting on content¶
what? Every one of those 1000+ friends has now given access to those 1000+ people that allow friends-of-friends to see their info”. [Note: Facebook and Google part funded this research through gifts]. Source: Tseng, E., Bellini, R., McDonald, N., Danos, M., Greenstadt, R., McCoy, D., Dell, N and, Ristenpart, T., 2020. The Tools and Tactics Used in Intimate Partner Surveillance: An Analysis of Online Infidelity Forums. [accessed 28 September 2023]. 715 For example, a UK political comedian described a ‘pile-on’ of violence and abuse against her following a media appearance on a television debate programme. She described how: “after the debate he continued to be rude about me on Twitter. That reached a whole new level. In the following 48 hours, I received 165 pages of Twitter abuse. Suddenly it went insane. In that, there were four or five death threats, rape threats, and things like that.” Source: Amnesty International, 2018. Online Violence against Women. [accessed 28 September 2023]. 716 See ‘Risk factors: functionalities and recommender systems’ section for more information. Source: McLean, L. and Griffiths, M. D., 2018. Female Gamers’ Experience of Online Harassment and Social Support in Online Gaming: A Qualitative Study, International Journal of Mental Health and Addiction, 17(970-994). [accessed 28 September 2023].
717 Zhou, Y. and Farzan, Y., 2021. Designing to stop live-streaming cyberbullying. [accessed 28 September 2023]. 718 Zhou, Y. and Farzan, Y., 2021. 719 UNESCO (Posetti, J., Aboulez, N., Bontcheva, K., Harrison, J. and Waisbord, S.), 2020. Online Violence Against Women Journalists. [accessed 28 September 2023]. 720 Suzy Lamplugh Trust, 2021. Unmasking stalking: a changing landscape. [accessed 28 September 2023]. 140
or abusive comments to an individual. Harassment can also occur where a user sends multiple comments to the same individual which are not hateful or abusive but are calculated to produce alarm or distress and are oppressive and unreasonable.
Posting content¶
Posting or sending location information¶
721 The Alan Turing Institute (Vidgen, B., Margetts, H. and Harris, A.), 2019. How much online abuse is there? [accessed 28 September 2023]. 722 This US-based study from Pew Research includes single occasions as measures of harassment, although in the UK the harassment offence is a course of conduct occurring on two or more occasions. Source: Pew Research, 2014. Online Harassment. [accessed 28 September 2023]. 723 YouTube, 2022. YouTube Community Guidelines enforcement – Google Transparency Report. [accessed 25 August 2023]. 724 A study on tweets sent directly to US candidates during the 2020 US election found that 15%-39% of all female candidates’ tweets were abusive, compared to 5-10% percent of male candidates. Source: Institute for Strategic Direction (Guerin, C. and Maharasingam-Shah, E.), 2020. Public Figures, Public Rage: Candidate abuse on social media. [accessed 28 September 2023].
725 Gosse, C., Veletsianos, G., Hodson, J., Houlden, S., Dousay, T.A. and Lowenthal, P.R., Hall, N., 2020. The hidden costs of connectivity, Learning, media and technology, 46(3). [accessed 28 September 2023]. 726 Patton, D. U., Pryooz, D., Decker, S., Frey, W. R. and Leonard, P., 2019. When Twitter Fingers Turn to Trigger Fingers: a Qualitative Study of Social Media-Related Gang Violence. [accessed 28 September 2023]. 727 Gunn, R., Tzani, C., Ioannou, M., Synnott, J. and Fumagalli, A., 2021. Cyberstalking among social media users: Perceptions, prevalence and characteristics. [accessed 28 September 2023]. 141
tracking might be carried out through spyware devices or apps.728 Some U2U services host geo-tagging functionalities729 and this information can be communicated to users.
Re-posting or forwarding content¶
Recommender systems¶
Content and network recommender systems¶
728 [Note: Facebook and Google part funded this research through gifts]. Source: Tseng, E., Bellini, R., McDonald, N., Danos, M., Greenstadt, R., McCoy, D., Dell, N., Ristenpart, T. 2020. The Tools and Tactics Used in Intimate Partner Surveillance: An Analysis of Online Infidelity Forums. [accessed 28 September 2023]. 729 Geotagging is the process of adding location data to media such as photos and videos, such as the coordinates of where a photograph or video has been taken. This occurs on most smartphones and tablets. Source: Paladin, n.d. Cyber and Digital Safety: are you a victim of cyberstalking? [accessed 28 September 2023].
731 Waldman, A, 2021. Navigating Privacy on Gay-Oriented Mobile Dating Applications. [accessed 28 September 2023]. 732 Context collapse in this example describes a piece of content being taken out of context to an audience it was not intended for. Source: Marwick, A., Boyd, D. 2010. I tweet honestly, I tweet passionately: Twitter users, context collapse and the imagined audience. [accessed 28 September 2023].
733 Thrasher, S.W, 2017. Yes there is a free speech crisis. But it’s victims are not white men. The Guardian, 5 June. [accessed 28 September 2023]; Gosse, C., Veletsianos, G., Hodson, J., Houlden, S., Dousay, T.A., Lowenthal, P.R. and Hall, N., 2020. The hidden costs of connectivity: nature and effectis of scholars’ online harassment, Learning, media and technology, 46(3). [accessed 28 September 2023]. 734 Thompson, J. D., Cover, R. 2021. Digital hostility, internet pile-ons and shaming: A case study. Convergence: The International Journal of Research into New Media Technologies, 28(6). [accessed 10 October 2024] 142
communication functionality and recommender systems work together to create this particular scenario.
Risk factors: Business models and commercial profiles¶
735 Institute for Strategic Dialogue (Thomas, E. and Balint, K.), 2022. Algorithms as a Weapon Against Women: How YouTube Lures Boys and Young Men into the ‘Manosphere.’ [accessed 10 July 2024]
736 Griffin, J., 2021. Incels: Inside a dark world of online hate, BBC News, 13 August. [accessed 11 July 2024]; Institute for Strategic Dialogue (Thomas, E. and Balint, K.), 2022. Algorithms as a Weapon Against Women: How YouTube Lures Boys and Young Men into the ‘Manosphere.’ [accessed 10 July 2024]. 737 Ellery B. and Mitib A., 2023. Social medic companies profit from misery spread by misogynistic influencers, The Times, 9 January. [accessed 29 July 2024]. 143
Section 5 Controlling or coercive behaviour (CCB)¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for controlling or coercive behaviour: how harm manifests online, and risk factors Online services offer new ways for perpetrators to coerce and control partners, former partners, or their children. In England and Wales, an estimated 1.4 million women and 751,000 men aged 16 years and over experienced domestic abuse in the year ending March 2023, equivalent to approximately 5.7% of women and 3.2% of men.738 The risks of harm from controlling or coercive Behaviour (CCB) are broad and can be life-threatening; they can affect a victim’s or survivor’s mental health, and affect their life in other ways, including their income. Harm is often caused by the perpetrator’s omnipresence in an individual’s life. Common risk factors are present in the evidence; however, services should be aware that the offence is often perpetrated in complex and personal ways. CCB can also involve other offences, including threats and intimate image abuse (see the Harassment, stalking, threats and abuse chapter, and the Intimate image abuse chapter for further information). Service type risk factors: Social media services offer perpetrators multiple ways to monitor victims and survivors, and to pursue campaigns of targeted abuse. CCB often happens across several social media services simultaneously. Messaging services enable perpetrators to be a constant presence in the lives of victims and survivors. Online dating services and online user-to-user pornography services can also be used in cases of coercive control, particularly if the abuse involves the sharing of intimate images. User base risk factors: Due to its significance in coercive control and other offences affecting women in particular, user base demographics are included as a general risk factor in the risk profiles. This is partly because CCB sits within a wider culture of gendered violence and misogyny. Gender is a risk factor, with women being more commonly and more severely affected. In England and Wales, in the year ending March 2023, 73.5% of domestic abuse-related crimes were female, and between 2020 and 2022,
738 Office for National Statistics, 2023. Domestic abuse victim characteristics, England and Wales: year ending March 2023. [accessed 20 November 2024]. Controlling or coercive behaviour is outlined as abuse in the Domestic Abuse Act 2021. 144
67.3% of domestic homicide victims were female.739 Young women, as well as women from minority ethnic and racial backgrounds, appear to be most at risk. Research indicates that disability and low socio-economic status can increase risk among women. While our evidence suggests that LGBTQ+ communities may be more at risk of CCB, more research is needed into possible ‘hidden groups’, which include male victims and survivors. Functionalities and recommender systems risk factors: Several functionalities enable monitoring practices. The most prominent are fake user profiles, which perpetrators can use to impersonate victims and survivors, as well as other individuals, to gain access to the target’s account, to monitor and harass victims and survivors. Sending abusive or threatening direct messages – sometimes incessantly and across multiple services – can cause fear and distress to victims and survivors and make them feel that the perpetrators have a constant presence in their lives. User connections allow users to build online networks, both around the perpetrators, and the victims and survivors. These networks can extend perpetrators’ ability to coerce and control victims and survivors, for example by creating an environment for public humiliation, or getting contacts to join in with monitoring or harassment. Location tracking is also common in cases of CCB, so posting or sharing location information represents a risk factor for this offence. Posting content gives perpetrators the ability to publicly post negative or personal information about victims and survivors, as well as to non-consensually share intimate images as part of campaigns of abuse.
Introduction¶
739 Office for National Statistics, 2023. Domestic abuse victim characteristics, England and Wales: year ending March 2023. [accessed 20 November 2024]. 145
5.4 legal definition of CCB. The language of coercive control is also used to explore how domestic abuse manifests online, based on the understanding that domestic violence is coercive, controlling, and profoundly contextualised in relationship dynamics, cultural norms, and structural inequality.742 To stay aligned with the evidence, we use the terminology from the research when citing studies in this chapter.
Relevant offences¶
740 An offence under section 76 of the Serious Crime Act 2015 (controlling or coercive behaviour in an intimate or family relationship). 741 Stark, E., 2009. Coercive control: the entrapment of women in personal Life. [accessed 21 September 2023].
742 For example, see: Dragiewicz, M., Burgess, J., Matamoros-Fernández, A., Salter, M., Suzor, N. P., Woodlock, D., Harris, B., 2018., Technology facilitated coercive control: domestic abuse and the competing roles of digital media platforms. [accessed 21 September 2023]; Harris, B. and Woodlock, D., 2022. Digital coercive control: Insights from two landmark domestic abuse studies. The British Journal of Criminology, 59(3). [accessed 21 September 2023]. 743 Refuge, 2021. Unsocial Spaces. [accessed 21 September 2023]. 744 Section 76 of the Serious Crime Act 2015. 146
How controlling or coercive behaviour manifests online¶
745 The phrase “victims and survivors” is used as many survivors find the word “victim” disempowering. However, the phrase “victim-survivors” also encompasses those who did not survive CCB.
746 It is important to note that the experience of partners is significantly more reported on than those of children in domestic abuse contexts. This is reflected in the evidence base available for this chapter. Research is limited on other groups such as children, male victims, minority ethnic groups, disabled people and LGBTQ+ people. This should be considered when reading this chapter. 747 Crown Prosecution Service, 2023. Legal Guidance: Controlling or Coercive behaviour in an intimate or family relationship. [accessed 5 September 2023]. 147
748 CIGI (Dunn, S., Vaillancourt, T. and Brittain, H.) 2023. Supporting Safer Digital Spaces. [accessed 18 November 2024]. 749 Henry, N., Vasil, S., Flynn, A., Kellard, K. & Mortreux, C. 2021. Technology-Facilitated Domestic Violence Against Immigrant and Refugee Women: A Qualitative Study. Journal of Interpersonal Violence. [accessed 05 September 2024]
750 Refuge, 2021. 751 Powell, A., Flynn, A., & Hindes, S. 2022. Technology-facilitated abuse: National survey of Australian adults’ experiences. [accessed 26 November2024]. 752 Women’s Aid is a charity working to end domestic abuse against women and children. 753 Women’s Aid (Laxton, C.), 2014. Virtual World, Real Fear: Women's Aid Report into Online Abuse, Harassment and Stalking. [accessed 23 September 2023]. 148
non-consensual intimate image-sharing (17%) or threats to do so (14%); unauthorised access to their online accounts (21%); doxxing (11%) and deepfakes (4%).754 755
754 Refuge, 2021. 755 Support services in Australia reported similar kinds of abuse experiences by victim-survivors they engage with, with 28% of women responding to the survey reporting experiencing multiple types of abuse from the same perpetrator: Powell, A., Flynn, A., & Hindes, S. 2022. 756 Powell, A., Flynn, A., & Hindes, S. 2022. 757 Powell, A., Flynn, A., & Hindes, S. 2022. Technology-facilitated abuse: National survey of Australian adults’ experiences. [accessed 26 November2024]. 758 Refuge, 2021. 759 Woodlock, D. et al, 2020.
760 Todd, C., Bryce, J & Franqueira, V. N. L. 2020. Technology, Cyberstalking and Domestic Homicide: Informing Prevention and Response Strategies. Policing and Society, accessed [15.09.2024]. 761 Refuge, 2021. 762 Refuge, 2021; Woodlock, D. et al, 2020. 763 Refuge, 2021. 764 Refuge, 2021. 149
765 In the Australian support service survey, respondents reported that perpetrators of tech abuse, in their experience, were most commonly men and boys, with a vast majority reporting that men (83%) and boys (35%) were using tech abuse to target victims ‘a lot of the time’, across both intimate partner, family member and/or acquaintance contexts: Flynn et al. (2021) Stakeholder survey. 766 Freed, D. et al. (2018) ‘’A stalker’s paradise’: How Intimate Partner Abusers Exploit Technology’, ACM Conference on Human Factors in Computing Systems (CHI 2018), accessed [05.09.2024]. 767 Tseng, E., Bellini, R., McDonald, N., Danos, M., Greenstadt, R., McCoy, D., Dell, N., Ristenpart, T., 2020. The Tools and Tactics Used in Intimate Partner Surveillance: An Analysis of Online Infidelity Forums. [accessed 21 September 2023]. 768 Women’s Aid. 2023. Influencers and Attitudes. [accessed 4 September 2024]. 769 In the previously referenced Women’s Aid survey (Women’s Aid, 2023) the questions refer specifically to ’Andrew Tate content’, due to the relevance of this influencer at the time of fieldwork, and recommend taking findings around this content to be representative of some types of existing online misogynistic content. 770 Women’s Aid, 2023.
771 A lack of reporting can be compounded by insufficient data recording practices; the National Police Chiefs’ Council (NPCC) highlights the challenges in assessing the scale of technology-enabled violence against women and girls (VAWG), due to data recording practices. It notes that while on average 10% of VAWG offences are recorded as occurring online, this is likely to be an underestimate, as most VAWG offences are likely to have a digital component. National Police Chiefs Council, 2023. Violence Against Women and Girls: Strategic Threat Risk Assessment 2023. [accessed 21 September 2023]. 772 Refuge, 2021. 150
may not identify as victims and survivors if asked directly; responses are higher if women are given specific examples of abuse to relate to.773
Risks of harm to individuals presented by controlling or coercive behaviour¶
773 Refuge reports (Refuge, 2021) that while more than one in three women (36%) reported experiencing forms of online abuse when asked about specific examples, only one in five (21%) self-identified as experiencing abuse on an online platform. Similarly, an Australian study found that many victims and survivors of intimate partner stalking do not identify stalking behaviour as such: Woodlock, D. 2017. The Abuse of Technology in Domestic abuse and Stalking. Violence against women, 23(5). [accessed 21 September 2023]. 774 Harris, B. and Woodlock, D., 2022. 775 Woodlock, D. 2017. Fernet, M. Lapierre, A., Hébert, M. and Cousineau, M.-M., 2019. A systematic review of literature on cyber intimate partner victimisation in adolescent girls and women, Computers in Human Behaviour 100. [accessed 21 February 2024]. 776 Refuge found that 47% of victims and survivors reported that someone had access to their social media account against their wishes: Refuge, 2022. Marked as Unsafe. [accessed 21 September 2023]. 777 Refuge, 2022. 778 An Australian study (Woodlock, D. 2017) reports a perpetrator hacking a victim’s and survivor’s account, before sharing intimate images with male contacts. Refuge reports (Refuge, 2021) a case in which ‘Laurel’s’ partner accessed her social media accounts and impersonated her online, while intercepting and deleting messages to make her question her memory. Laurel was also physically abused by her partner, but spoke about the tech abuse and gaslighting as being the worst part of her experience.
779 The most common forms of CCB online identified by Refuge are (Refuge, 2021): online harassment; stalking, monitoring and location-tracking; threats of physical and sexual violence; having accounts hacked or controlled; online impersonation; sharing of intimate images or videos without consent, or threats to share; having personal details shared online without consent, also known as ‘doxxing’. 151
780 Office for National Statistics, 2018. Women most at risk of experiencing partner abuse in England and Wales: years ending March 2015 to 2017. [accessed 16 September 2024] 781 In the year ending March 2023, the victim was female in 73.5% of domestic abuse-related crimes: Office for National Statistics, 2023. Domestic abuse victim characteristics, England and Wales: year ending March 2023. [accessed 20 November 2024]. 782 Powell, A., Flynn, A., & Hindes, S. 2022. Technology-facilitated abuse: National survey of Australian adults’ experiences. [accessed 26 November2024]. 783 Powell, A., Flynn, A., & Hindes, S. 2022. 784 As noted by SafeLives, a UK charity dedicated to ending domestic abuse, it is important to understand that different parts of this community can experience abuse in different ways: SafeLives, 2021. Transgender victims’ and survivors’ experiences of domestic abuse. [accessed 18 November 2024]. 785 Bisexual women were found to be 3 times more likely to experience abuse as heterosexual women: SafeLives, 2021. Bisexual victims and survivors’ experiences. [accessed 16 September 2024]. 786 Office for National Statistics, 2018. 787 Office for National Statistics, 2018.
788 Brown, M. L., Reed, L. A., Messing, J. L., 2018. Technology-Based Abuse: Intimate Partner Violence and the Use of Information Communication Technologies in Ryan Vickery, J., Everbach, T. (eds). #NastyWomen: Reclaiming the Twitterverse from Misogyny. [accessed 21 September 2023]. 789 Fernet, M. Lapierre, A., Hébert, M. and Cousineau, M.-M., 2019. 790 Refuge, 2021. 791 Refuge, 2021. 152
found that individuals who have received offensive or threatening messages are more than five times more likely to have had suicidal thoughts.792
792 McManus, S., Bebbington, P.E., Tanczer, L., Scott, S. and Howard, L.M. 2021. Receiving threatening or obscene messages from a partner and mental health, self-harm and suicidality: results from the Adult Psychiatry Morbidity Survey. [accessed 18 November 2024]. 793Woodlock, D., McKenzie, M., Western, D. and Harris, B., 2020. Technology as a Weapon in Domestic abuse: Responding to Digital Coercive Control. Australian Social Work, 73 (3). [accessed 21 September 2023]. 794 Woodlock, D. et al., 2020. 795 All-Party Parliamentary Group on Domestic Violence: as referenced in Women’s Aid, 2017. Tacking domestic abuse in the digital age. [accessed 21 September 2023]. 796 Refuge, 2021. 797 Honour-based abuse is a crime or incident committed to protect or defend the 'honour' of a family or community: Metropolitan Police, n.d. What is honour-based abuse?. [accessed 5 September 2023]. 798 Refuge, 2021. 799 Henry, N., Vasil, S., Flynn, A., Kellard, K. & Mortreux, C. 2021. 800 Refuge found (Refuge, 2021) that 38% of women who experienced abuse on social media from a partner or former partner said they felt unsafe or less confident online as a result.
801 A domestic abuser service provider describes the risks associated with blocking perpetrators. “You have to be really careful. You can’t even really tell anyone to block anyone ‘cause that could escalate things as well. It is literally a case-by- case basis. Some victims know. I’m keeping him sort of subdued by just taking his behaviours, but if I react then maybe he’ll react.”: Sugiura, L., Blackbourn, D., Button, M., Hawkins, C., Tapley, J., Frederick, B., Nurse, J. R. C., Rahime, B. S., 2021. Computer Misuse as a Facilitator of Domestic Abuse. [accessed 21 September 2023]. 802 Refuge, 2021. 153
a common trigger803; 66% of women who experienced tech abuse from an intimate partner said they were an ex-partner at the time of the abuse, compared to 18% who said the perpetrator was a partner at the time of the abuse.804
Evidence of risk factors on user-to-user services¶
Risk factor: Service types¶
Social media services¶
Messaging services and dating services¶
803 Woodlock, D. 2017. 804 Refuge, 2021. 805 Refuge, 2021.
806 Refuge, 2021. 807 An Australian study found multiple cases of perpetrators using the social media pages of shared friends, family or even their children for monitoring purposes: Woodlock, D. 2017. 808 Younger persons are more likely to use functionalities associated with social media services (unauthorised access to accounts, creation of fake profiles), whilst older people are more likely to use physical covert devices: Sugiura, L. et al., 2021. 154
of victims. Direct messaging is central to messaging services, as well as many social media services and dating services.
User-to-user pornography services¶
Risk factors: User base¶
User base size¶
User base demographics¶
809 Sugiura, L. et al., 2021. 810 Refuge, 2021. 811 Ipsos, Ipsos iris online audience measurement service, (BG) WhatsApp (web and app) and (APP) Facebook Messenger (app only), September 2024, UK online adults aged 18+. Note that Facebook’s messenger service can be accessed through the main Facebook site and app which is not included in this data.
812 Refuge, 2022. 813 Ipsos, Ipsos iris online audience measurement service, brand group (BG) TikTok, and (BG) Snapchat, September 2024, UK online adults aged 18+. 814 Refuge, 2022. 155
815 A literature review on intimate partner violence found that the majority of studies lacked adequate information about demographic characteristics such as age and geographical region: Grimani, A., Gavine, A., and Moncur, W. 2022. An Evidence Synthesis of Covert Online Strategies Regarding Intimate Partner Violence, Trauma, Violence and Abuse, 23(2). [accessed 21 September 2023]. 816 Young women aged between 16 and 19 (7.6%) and 20 and 24 (7.4%) were significantly more likely to have experienced partner abuse in the 12 months before interview than women aged between 45 and 54 (5.6%) or between 55 and 59 (4.4%): Office for National Statistics, 2018.https://www.ons.gov.uk/peoplepopulationandcommunity/crimeandjustice/articles/womenmostatriskofexperiencing partnerabuseinenglandandwales/yearsendingmarch2015to2017 817 Refuge, 2022. 818 Refuge, 2022. 819 Powell, A., Flynn, A., & Hindes, S. 2021. Stakeholder survey. Technology-facilitated abuse: A survey of support services stakeholders. [accessed 26 November 2024].
820 Walby, S. and Towers, J., 2018. Untangling the concept of coercive control: Theorizing domestic violent crime. Criminology and Criminal Justice, 18(1). [accessed 21 September 2023]. 821 Refuge, 2021. 822 Stakeholder survey (2022). 823 Refuge, 2021. 156
man (62.1%) compared to 31.1% saying the perpetrator was a woman.824 For example, one study of 96 cases of domestic abuse recorded by the police found that men are more likely to be repeat perpetrators, and more likely than women to use physical violence, threats and harassment.825 Data from Galop shows that male perpetration is also higher among LGBTQ+ communities. A survey of LGBTQ+ victims and survivors found that 71% of individual perpetrators identified as male and 29% as female.826
824 Powell, A., Flynn, A., & Hindes, S. 2022. Technology-facilitated abuse: National survey of Australian adults’ experiences. [accessed 26 November2024]. 825 Hester, M., 2013. Who Does What to Whom? Gender and Domestic abuse Perpetrators in English Police Records. European Journal of Criminology, 10(5). [accessed 21 September 2023]. 826 From a sample of 626 LGBTQ+ victims and survivors based in Greater London. Galop, (Magić, J., Kelley, P.), 2018. LGBT+ People’s Experiences of Domestic Abuse: a report of Galop’s domestic abuse advisory service. [accessed 21 September 2023]. 827 Sugiura, L. et al., 2021. 828 Furthermore, it is recognised that male survivors of domestic violence are often not able to access support or are not taken seriously due to a lack of recognition of the problem. 829 Almost three-quarters (74%) of domestic violent crime victims were female and 82% of domestic violent crimes were against women in Crime Survey for England and Wales data reviewed from 2008 to 2013, but there were still 79,473 men experiencing 219,118 cases of domestic abuse in this sample: Donovan, C., and Barnes, R., 2021. Re-tangling the concept of coercive control: A view from the margins and a response to Walby and Towers (2018). Criminology and Criminal Justice, 21(2). [accessed 21 September 2023].
830 Office for National Statistics, 2018. 831 Brookfield, K. et al. (2024), Technology-Facilitated Domestic Abuse: An under-recognised safeguarding issue? The British Journal of Social Work 54.1, accessed [05 September 2024]. 832 Office for National Statistics, 2018. 157
and being part of smaller communities where reputational damage can have a greater effect than in other communities.833
Risk factors: Functionalities and recommender systems¶
User identification¶
Fake user profiles¶
833 Woodlock, D. et al, 2020. 834 A national UK LGBTQI+ anti-violence charity Galop highlights the specific issues of partner abuse unique to the experiences of LGBTQI+ people, such as the threat of disclosure of sexual orientation and gender identity to family, friends, or work colleagues: Galop, n.d. Domestic Abuse. [accessed 5 September 2023]. 835 Refuge, 2021.
836 Office for National Statistics, 2018. 837 eSafety Commissioner (Harris, B., and Woodlock, D.), 2021. For my safety: experiences of technology-facilitated abuse among women with intellectual disability to cognitive disability. [accessed 4 September 2024]. 838 WESNET, 2022. How tech abuse affects women with disabilities. [accessed 4 September 2024]. 839 National Center for Transgender Equality, 2015. US Transgender Survey. [accessed 8 September 2023]. 840 Sugiura, L. et al., 2021. 158
that represent fictitious people or real people known to victims and survivors.841 Refuge reports that 29% of victims and survivors have been impersonated.842
User networking¶
User connections¶
841 The study lists examples of this, such as a man who set up accounts on swingers’ and dating accounts in a woman’s name with her workplace listed to discredit her, or a woman setting up a fake account under her ex-partners name and sending abusive messages to herself, before reporting this to the police: Sugiura, L. et al., 2021. 842 Refuge, 2022. 843 Refuge response to Ofcom 2022 Call for Evidence: First phase of online safety regulation. This included the quotation from a target of CCB: “When I was pregnant I was getting threats about my child. A lot of (the messages) were fake accounts – so it was over 40 accounts […] I reported three times. […] He’d send me voicemails - you can do that on [social media platforms]. He made other accounts where he threatened to kill me and then he messaged my family on [social media platforms]”.
844 Refuge, 2022. 845 Refuge, 2021. 846 Tseng, E., Bellini, R., McDonald, N., Danos, M., Greenstadt, R., McCoy, D., Dell, N., Ristenpart, T., 2020. 847 Melton, H, 2007. Stalking in the context of intimate partner abuse: In the victims’ words. Feminist Crimonolgy 2(4). [accessed 21 September 2023]. 159
User communications¶
Direct messaging¶
Posting content (text, images)¶
848 19% of victims and survivors said that the family of their partner or ex-partner was involved in the abuse, and 8% said their partner’s friends were involved: Refuge, 2021. 849 Dragiewicz, M., Harris, M., Woodlock, D., Salter, M., Easton, H., Lynch, A., Campbell, H., Leach, J., Milne, L., 2019. Domestic Violence and communication technology: victim experiences of intrusion, surveillance and identity theft. [accessed 21 September 2023]; Grimani, A., Gavine, A. and Moncur, W. 2022. 850 Refuge, 2021. 851 Woodlock, D. 2017.
852 Dragiewicz, M. et al., 2019. 853 Problematic use of this functionality requires an understanding of context for it to be identified as CCB. Repeated direct messages, like frequently messaging one’s partner to check their location, can be harmless or abusive, depending on the overall context of the relationship. Dragiewicz, M. et al., 2019. 854 Refuge, 2021. 855 Dragiewicz, M. et al., 2019. 160
Posting or sending location information, user groups, user events, user tagging¶
856 The Refuge study (Refuge, 2022) also provides qualitative examples of doxxing in the context of domestic abuse. For example, ‘Paula’, whose former partner waged a campaign of harassment, publicly accused her of lying about the domestic abuse that she faced and encouraging others to abuse her. Direct threats of harm were made, and her name and address were publicly shared from the abuser’s account. 857 Woodlock, D. 2017. 858 An individual described how her former partner publicly claimed she had given him a sexually transmitted infection – this information was read by her teenage son’s friends, among other people. The same study found perpetrators publicly shaming victims and survivors as ‘punishment’ for transgressions. Practitioners report behaviours such as a ‘status update’ where the perpetrator blames his problems on the victims and survivors, calls them names and accuses them of shameful behaviour. This can result in ‘comments’ of support to him from family and friends, leaving victims and survivors feeling isolated and ‘ganged up on’ by an entire community. Woodlock, D. 2017. 859 Refuge, 2022.
860 Sharratt, E., 2019. Intimate image abuse in adults and under 18s. [accessed 21 September 2023]. 861 Office for National Statistics, 2019. Domestic abuse and the criminal justice system, England and Wales: November 2019. [accessed 8 September 2023]. 862 For example, in one case provided by the Law Commission, an ex-partner set up a fake Facebook account in their ex- partner’s name and uploaded intimate images of her, which were then viewed and copied to pornography sites, where on one website the picture was viewed over 48,000 times: Sharratt, E, 2021. 161
10) of perpetrators reported ‘tracking someone through GPS’.863 This section draws on evidence from the Harassment, stalking, threats and abuse chapter. More detail on cyberstalking can be found in this chapter.
Content editing¶
Editing visual media¶
Recommender systems¶
Content recommender systems¶
863 Gunn, R., Tzani, C., Ioannou, M., Synnott, J., Fumagalli, A., 2021. Cyberstalking among social media users: Perceptions, prevalence and characteristics. [accessed 21 September 2023]. 864 Refuge, 2022. 865 Woodlock, D. 2017. 866 Sugiura, L. et al., 2021. 867 Chaulk, K., Jones, T. 2011., Online Obsessive Relational Intrusion: Further Concerns About Facebook. Journal of Family Violence, 26. [accessed 21 September 2023]. 868 Deepfakes are a specific type of media that involves the use of AI algorithms, particularly generative AI models, to modify videos, images or audio to create realistic synthetic content. This is often done by superimposing the face of a person onto the body of another person in a video or image as well as voice manipulation with lip syncing. Deepfakes are commonly shared as user generated content on user-to-user services but could also potentially be created using functionalities present on user-to-user services. Deepfake technology is currently used to create content that can be harmful; however, we acknowledge that it may also have positive use cases.
869 Refuge, 2021. 162
partner’s accounts or stalk partners”.870 Recommender systems may increase the risk of perpetrators coming across content that can be used for abusive purposes (such as spyware or information related to their partners or former partners) if the algorithm recommends content based on a perpetrator’s previous search history or interaction with content. This, in turn, increases the risk of perpetrators finding content that enables them to commit abusive or controlling behaviour.
Risk factors: Business models and commercial profiles¶
870 Sugiura, L., Blackbourn, D., Button, M., Hawkins, C., Tapley, J., Frederick, B., Nurse, J. R. C., Rahime, B. S., 2021. Computer Misuse as a Facilitator of Domestic Abuse. [accessed 21 September 2023]. 163
Section 6 Intimate image abuse¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for intimate image abuse: how harms manifest online, and risk factors This chapter looks at offences relating to non-consensually sharing or threatening to share intimate images. Between 2019 and 2022, 24 police forces recorded a total of 13,860 intimate image offences, with more offences recorded in the first six months of 2022 than in all of 2020.871 Such acts can have serious negative impact on individuals, causing mental health issues, with considerable distress and anxiety experienced by victims and survivors. This can include shame, helplessness, self-blame, isolation and humiliation, and damage to their professional lives, including their finances, education and employment. This intimate image abuse, and the harm it causes, is disproportionately experienced by women. But there is growing trend in financially motivated sexual extortion (‘sextortion’) in which men are most often the targets. Service type risk factors: Research indicates that intimate image abuse occurs particularly on user-to-user pornography services, with findings indicating that these services may be at a higher risk of being used by perpetrators to commit the offence. Studies show that intimate image abuse also occurs frequently on social media services, file-storage and file-sharing services, and dating services. Discussion forums and chatrooms have been found to allow its users to form online communities where members are able to discuss and share intimate images, including deepfakes. While messaging services are often used by offenders to non-consensually share intimate ages, and to threaten victims with sharing their intimate images. User base risk factors: Intimate image abuse is a gendered offence; women are more likely to be depicted in the images, and the person committing the offence is more likely to be a man. There is evidence that age can be a risk factor, with higher reported cases of intimate image abuse occurring among women aged 18 to 24. Other user groups that face an increased risk of intimate image abuse include people from minority ethnic and racial backgrounds. Additionally, research indicates that disability,
871 Refuge (2023) ‘Intimate image abuse – despite increased reports to the police, charging rates remain low’ accessed [20 November 2024]. 164
cultural and linguistic diversity, sexual orientation and low socio-economic status can increase risk of and from intimate image abuse. Functionalities and recommender systems risk factors: Intimate image abuse is primarily committed by posting images and videos where perpetrators share intimate images non-consensually. This risk can be exacerbated by functionalities such as re-posting and forwarding content, direct messaging and group messaging, where intimate images can be further shared with larger audiences. Some services allow users to screen capture, record, or download content, such as intimate images, which can then be shared on other services. Several additional functionalities on user-to-user (U2U) services can facilitate the non-consensual sharing of intimate images. Encrypted messaging can be used to share intimate images and more easily evade detection. Livestreaming can be used in the commission of intimate image abuse, where videos of people engaging in sexual activities are broadcast online without their consent. User groups allow like-minded individuals to form communities and potentially share intimate image abuse content with one another. The ability to label or tag content can facilitate intimate image abuse, as these tags can be manipulated by users to ensure that intimate images are shown to users who are more likely to know the person depicted. Perpetrators can create fake user profiles that impersonate their targets as well as anonymous user profiles that gives them added confidence in sharing intimate images without being identified. With the advent of Generative AI, functionalities that allow for the editing of visual media has become central to the creation of deepfake intimate images. These functionalities can vary from purpose-built websites to apps, or bots on U2U services.
Introduction¶
wider societal effects of the harm caused to individuals because of exposure to the content or activities that amount to relevant offences. In this case, this would relate to the normalisation of intimate image abuse in some contexts.
Relevant offences¶
872 Intimate image abuse can also be referred to as ‘revenge porn’ or ‘image-based sexual abuse’. ‘Revenge porn’ is a commonly-used term, but does not adequately capture the power dynamics and the type of content involved in intimate image abuse and we do not therefore use it in our risk assessment. ‘Image-based sexual abuse’ is often used to describe a broader range of harms than those covered in this chapter, including offences such as cyberflashing and the production or sharing of child sexual abuse material. We will therefore not be using the term ‘image-based sexual abuse’ in this chapter. These offences are covered separately in the chapters ‘Cyberflashing’ and ‘Child sexual exploitation and abuse (CSEA)’. 873 Manipulated images and videos, such as deepfakes, are considered within the scope of this offence. Any photograph or video which appears to depict an intimate situation should be treated as a photograph or video actually depicting such a situation. For more detail, refer to the refer to the Illegal Content Judgements Guidance (ICJG). 874 Section 66(B) of the Sexual Offences Act 2003; section 2 of the Abusive Behaviour and Sexual Harm (Scotland) Act 2016 (asp 22).
875 We are aware that interpretations of what is ‘intimate’ can vary among different cultural and religious groups, but Ofcom’s interpretation is based on the OSA legislation. The definition of intimate state in the Act is broad and can include "doing a thing that a reasonable person would consider to be sexual"; "the person in an act of urination or defecation"; and "the person carrying out an act of personal care associated with the person's urination, defecation or genital or anal discharge". The majority of our evidence focuses on sexual images. 876 Revenge Porn Helpline (Ward, Z.), 2021. Intimate image abuse, an evolving landscape. [accessed 3 August 2023]. 166
Intimate image abuse can form part of a wider continuum of online and offline behaviours by a partner or former partner, and often exhibits the gender dynamics of partner abuse/domestic abuse.877 These are closely linked to other behaviours explored in greater detail in the Threats, Harassment and Stalking and Coercive Controlling Behaviour chapters.
How intimate image abuse offences manifest online¶
Non-consensual sharing of intimate images¶
891 Internet Watch Foundation. 2024. ’Teenage boys targeted as hotline sees ’heartbreaking’ increase in child ’sextortion’ reports’. [accessed 10 October 2024]. Note: Intimate Image abuse offences involving those under the age of 18 are covered by CSEA offences [LINK CHAPTER]. 892 Ministry of Justice, 2022. New laws to better protect victims from abuse of intimate images. [accessed 4 August 2023]. 893 Revenge Porn Helpline, 2024. Revenge Porn Helpline 2023 Report. [accessed 10 October 2024]. 894 Ministry of Justice, 2022. New laws to better protect victims from abuse of intimate images. [accessed 3 August 2023]. 895 Refuge, 2023. Intimate image abuse – despite increased reports to the police, charging rates remained low. [accessed 31 August 2023].
896 Sextortion is a form of blackmail that involves threatening to publish sexual information, photos or videos about someone. Source: Metropolitan Police, n.d. Sextortion. [accessed 4 August 2023]. 897 Moore, A., 2022. ‘I have moments of shame I can’t control’: the lives ruined by explicit ‘collector culture’, The Guardian, 6 January. [accessed 3 August 2023]. 898 Revenge Porn Helpline (Ward, Z.), 2021. Intimate image abuse, an evolving landscape. [accessed 3 August 2023]. 169
image-boards, community forums and specific ‘revenge sites’ and varying in theme as well as in functionality.899
Deepfake intimate image abuse¶
899 Henry, N., Flynn, A. 2019. Image-Based Sexual Abuse: Online Distribution Channels and Illicit Communities of Support. Violence against women 25(16). [accessed 10 October 2024]. 900 UK Safer Internet Centre, 2024. Sextortion Report – August 2022 to August 2024. [accessed 18 November 2024]. 901 Revenge Porn helpline, 2021. RPH cases and trends of 2021. [accessed 09 October 2024]. 902 Revenge Porn Helpline (Huber, A. and Ward, Z.), 2024. Non-Consensual Intimate Image Distribution: Nature, Removal, and Implications for the Online Safety Act. [accessed 10 October 2024] 903 Henry, N., Flynn, A. 2019. Image-Based Sexual Abuse: Online Distribution Channels and Illicit Communities of Support. Violence against women 25(16). [accessed 10 October 2024].
904 Flynn, A., Powell, A., Scott, A. J., and Cama, E. (2022). Deepfakes and Digitally Altered Imagery Abuse: A Cross-Country Exploration of an Emerging form of Image-Based Sexual Abuse. The British Journal of Criminology, 62(6). [accessed 06 February 2024]. 905 My Image My Choice, 2024. Deepfake Abuse: Landscape Analysis 2023-24. [accessed 14 August 2024] 906 My Image My Choice, 2024. 907 Ofcom, 2024, Deepfake Defences: Mitigating the Harms of Deceptive Deepfakes, accessed [18 August 24] 170
Institute found that 19% (nearly 1 in 5) of adult respondents who had encountered a deepfake online said it was a non-consensual sexual deepfake.908
Risks of harm to individuals presented by intimate image abuse¶
908 Sippy, T., Enock, F. E., Bright, J. and Margetts, H. Z. 2024 Behind the Deepfake: 8% Create; 90% Concerned: Surveying public exposure to and perceptions of deepfakes in the UK. [accessed 18 August 2024]. 909 Deeptrace, 2019. The State of Deepfakes: Landscape, threats and impact. [accessed 08 November 2024] 910 Sensity AI, 2020. Automating Image Abuse: Deepfake bots on Telegram. accessed [15 February 2024]. 911 Indecent images of children (under 18s) would be Child Sexual Abuse Material (CSAM), and are covered in the chapter Child Sexual Abuse and Exploitation (CSEA) 912 My Image My Choice (2024) Deepfake Landscape Analysis 2023-2024. [accessed 20 September 2024]. 913 Where ‘nudify’ services allow users to share generated content with other users of the service, the nudify service could be considered to be a regulated user-to-user service and therefore the provider of the service would need to comply with the Part 3 duties, including the Illegal Harms duties of the Act. A GenAI tool that is made available by a service provider with the intention of allowing users to generate pornographic content may be in scope of the Part 5 duties. 914 McGlynn, C., Johnson, K., Rackley, E., Henry, N., Gavey, N., Flynn, A., and Powell, A. 2021. ‘’It’s Torture for the Soul’: The Harms of Image-Based Sexual Abuse. Social & Legal Studies, 30(4). [accessed 1 February 2024].
915 Patel, U. and Roesch, R. 2020. The Prevalence of Technology-Facilitated Sexual Violence: A Meta-Analysis and Systematic Review. Trauma, Violence and Abuse 23(2). [accessed 08 November 2024]. 916 Revenge Porn Helpline, 2024. Revenge Porn Helpline 2023 Report. [accessed 10 October 2024]. 917 McGlynn, C., Johnson, K., Rackley, E., Henry, N., Gavey, N., Flynn, A., and Powell, A. 2021. 918 Henry, N., Flynn, A. and Powell, A. 2019. Responding to ‘revenge pornography’: Prevalence, nature and impacts. [accessed 08 November 2024] 171
919 A survey by Thorn, looking at the experiences of young people who had been targets of threats to expose sexual images, found that 34% of respondents had received threats on a daily basis. In the same survey, 22% of respondents reported the threats lasting for more than six months. (Sample was recruited online and consisted of 1,631 18 to 25 year olds. Facebook and Twitter provided Thorn with grants which were used to fund the recruitment of respondents). Source: Thorn (Wolak, J. and Finkelhor, D.), 2016. Sextortion: Findings from a survey of 1,631 victims. [accessed 4 August 2023]. 920 Thorn is an international non-profit organisation working to develop new technologies to combat online child sexual abuse. 921 Sample was recruited online and consisted of 1,631 18- to 25-year-olds who had been targets of threats to expose sexual images. Facebook and Twitter provided Thorn with grants which were used to fund the recruitment of respondents. 922 Thorn (Wolak, J. and Finkelhor, D.), 2016. Sextortion: Findings from a survey of 1,631 victims. [accessed 4 August 2023]. 923 Refuge, 2020. The Naked Threat. [accessed 4 August 2023]. 924 Flynn, A., Powell, A., Scott, A. J., and Cama, E. (2022). Deepfakes and Digitally Altered Imagery Abuse: A Cross-Country Exploration of an Emerging form of Image-Based Sexual Abuse. The British Journal of Criminology, 62(6). [accessed 06 February 2024]. 925 The Cyber Civil Rights Initiative is a non-profit organisation based in the United States working to combat online abuses that threaten civil rights and civil liberties.
926 Cyber Civil Rights Initiative (Eaton, A. A., and Jacobs, H. and Ruvalcaba, Y.), 2017. 2017 Nationwide Online Study of Nonconsensual Porn Victimization and Perpetration: A Summary Report. [accessed 4 August 2023]. 927 Refuge, 2020. The Naked Threat. [accessed 4 August 2023]. 928 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 929 Davidson, J., Livingstone, S., Jenkins, S., Gekoski, A., Choak, C., Ike, T. and Phillips, K., 2019. Adult Online Hate, Harassment and Abuse: A rapid evidence assessment. [accessed 4 August 2023]. 172
Evidence of risk factors on user-to-user services¶
Risk factors: Service type¶
930 BBC News, 2016, Celebgate hack: Man to plead guilty to nude photos hack, BBC News, 15 March. [accessed 3 August 2023]; Channel 4 News, 2024, Exclusive: Hundreds of British celebrities victims of deepfake porn, Channel 4 News, 21 March [accessed 3 September 2024]. 931 The sample comprised of 714 “women-identifying” journalists. Source: Posetti, J. & Shabbir, N. 2023. The Chilling: A global study of online violence against women journalists. [accessed 3 September 2024]. 932 When intimate image abuse is leveraged against women in the public eye, often alongside other forms of abuse, they face reputational damage. 10% of women journalists surveyed in 2022 said their professional reputations or employment had been affected as a result of online abuse. Source: Posetti, J. & Shabbir, N. 2023. The Chilling: A global study of online violence against women journalists. [accessed 3 September 2024]. 933 Enock, F. E., Stevens, F., Bright, J., Cross, M., Johansson, P., Wajcman, J., Margetts, H. Z. 2024. Understanding gender differences in experiences and concerns surrounding online harms: A short report on a nationally representative survey of UK adults. Computers and Society (forthcoming). [accessed 14 November 2024].
934 Refuge, 2022. Marked as Unsafe: How online platforms are failing domestic abuse survivors. [accessed 22 August 2023]. 935 See, for example, the discussion of a broader idea of ’intimate image’ in Rackley, E., McGlynn, C., Johnson, K., Henry, N., Gavey, N, Flynn, A. and Powell, A. 2021. Seeking justice and redress for victim-survivors of image-based sexual abuse. Feminist Legal Studies, Volume 29. [accessed 14 November 2024]. 173
user-generated pornography sites936, image boards, community forums, blogging platforms, and social media services.937 In a random sample of Revenge Porn Helpline data from between 2018 and 2022, 52% of the 200 cases involved distribution to social media, with Facebook and Instagram being involved in the greatest number of cases, followed by Twitter and Snapchat. Distribution via public URLs was linked to 44% (more than 2 in 5) of cases.938
User-to-user pornography services¶
Social media services¶
936 While the analysis only applies to online user-to-user pornography services that allow users to share user-generated pornographic content, it is possible that some of our evidence also covers services that allow users to view pornographic content that has been produced by providers of pornographic content. 937 Henry, N. and Flynn, A., 2019. Image-Based Sexual Abuse: Online Distribution Channels and Illicit Communities of Support, Violence Against Women, 25(16), pp.1932-1955. [accessed 5 September 2023]. 938 Revenge Porn Helpline (Huber, A. and Ward, Z.), 2024. Non-Consensual Intimate Image Distribution: Nature, Removal, and Implications for the Online Safety Act. [accessed 10 October 2024]. 939 Revenge Porn Helpline’s response to 2021 Law Commission consultation. [accessed 3 August 2023]. 940 Vera-Gray, F., McGlynn, C., Kureshi, I. and Butterby, K., 2021. Sexual violence as a sexual script in mainstream online pornography, The British Journal of Criminology, 61(5). [accessed 22 August 2023]. 941 Henry, N. and Flynn, A., 2019. Image-Based Sexual Abuse: Online Distribution Channels and Illicit Communities of Support, Violence Against Women, 25(16), pp.1932-1955. [accessed 5 September 2023].
942 Revenge Porn Helpline, 2024. Revenge Porn Helpline 2023 Report. [accessed 10 October 2024]. 943 Australian Government (Office of the eSafety Commissioner), 2017. Image-based abuse. National survey: summary report. [accessed 22 August 2023]. 944 Revenge Porn Helpline, 2024. 945 Refuge, 2022. Marked as Unsafe: How online platforms are failing domestic abuse survivors. [accessed 22 August 2023]. 946 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 174
Messaging services¶
File-storage and file-sharing services¶
Dating services¶
947 Revenge Porn Helpline, 2024. 948 Sample was recruited online and consisted of 1,631 18- to 25-year-olds who had been targets of threats to expose sexual images. Facebook and Twitter provided Thorn with grants which were used to fund the recruitment of respondents. 949 Thorn (Wolak, J. and Finkelhor, D.), 2016. Sextortion: Findings from a survey of 1,631 victims. [accessed 4 August 2023]. 950 McLaughlin, E., 2018. Dropbox removed folder containing explicit photos of female service members, ABC News, 12 March. [accessed 22 August 2023]. 951 BBC News, 2019. Victim's warning after finding revenge porn from 'every UK city', BBC News, 17 May. [accessed 22 August 2023]. 952 Revenge Porn Helpline (Huber, A. and Ward, Z.), 2024. Non-Consensual Intimate Image Distribution: Nature, Removal, and Implications for the Online Safety Act. [accessed 10 October 2024].
953 Revenge Porn Helpline, n.d. What to do if you’ve been victim to online webcam blackmail, also known as sextortion. [accessed 12 September]. 954 Sample was recruited online and consisted of 1,631 18-25-year-olds who had been targets of threats to expose sexual images. Source: Thorn (Wolak, J. and Finkelhor, D.), 2016. Sextortion: Findings from a survey of 1,631 victims. [accessed 12 September 2023]. 175
Video-sharing services¶
Discussion forums and chatrooms¶
Risk factors: User base¶
User base size¶
955 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 956 Sample was recruited online and consisted of 1,631 18-25-year-olds who had been targets of threats to expose sexual images. Source: Thorn (Wolak, J. and Finkelhor, D.), 2016. 957 Henry, N. and Flynn, A., 2019. Image-Based Sexual Abuse: Online Distribution Channels and Illicit Communities of Support, Violence Against Women, 25(16), pp.1932-1955. [accessed 10 October 2024]. 958 Revenge Porn Helpline (Huber, A. and Ward, Z.,), 2024.
959 Tenbarge, K. (2023). The Deepfake porn industry is operating in plain sight, NBC News, 27 March. [accessed 14 August 2024]; My Image My Choice, 2024. Deepfake Landscape Analysis 2023-2024, [accessed 20 September 2024]. 960 My Image My Choice, 2024. 961 My Image My Choice, 2024. 962 My Image My Choice, 2024. 963 Graphika, 2023. A Revealing Picture. [accessed 14 August 2024]. 176
media services are considered by perpetrators as a place where their material will be seen by people whom the victims and survivors know. The primary motivation for the perpetrator is shaming their target.964 For intimate image abuse as a form of domestic abuse, the perpetrator would be looking for the largest number of users known to the person they are abusing.
User base demographics¶
6.52
964 Office of the eSafety Commissioner, 2017. Image-based abuse. National survey: summary report. [accessed 22 August 2023]. 965 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 966 Revenge Porn Helpline, 2024. Revenge Porn Helpline 2023 Report. [accessed 10 October 2024]. 967 Revenge Porn Helpline (Huber, A. and Ward, Z.), 2024. Non-Consensual Intimate Image Distribution: Nature, Removal, and Implications for the Online Safety Act. [accessed 10 October 2024].
968 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 969 Revenge Porn Helpline (2024) Revenge Porn Helpline 2023 Report. [accessed 10 October 2024]. 970 Flynn, A., Powell, A., Scott, A. J., and Cama, E. (2022). Deepfakes and Digitally Altered Imagery Abuse: A Cross-Country Exploration of an Emerging form of Image-Based Sexual Abuse. The British Journal of Criminology, 62(6). [accessed 06 February 2024]. Note: The creation of deepfake intimate imagery is not currently an offence under the OSA. 177
nudes were shared non-consensually by men and boys who were in more ‘casual’ relationships that were short-term or less serious, and the victims were women and girls.971
971 Revealing Reality, 2023. Without Consent. [accessed 15 August 2024] 972 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 973 Revenge Porn Helpline, 2024. Revenge Porn Helpline 2023 Report. [accessed 10 October 2024]. 974 Refuge, 2020. The Naked Threat. [accessed 4 August 2023]. 975 Cyber Civil Rights Initiative (Eaton, A., and Jacobs, H. and Ruvalcaba, Y.), 2017. 2017 Nationwide Online Study of Nonconsensual Porn Victimization and Perpetration: A Summary Report. [accessed 4 August 2023]. 976 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]; Revenge Porn Helpline (Ward, Z.), 2022. Revenge Porn Helpline Report. [accessed 22 August 2023]. 977 Revenge Porn Helpline (2024), Revenge Porn Helpline 2023 Report, accessed [10 October 2024]. 978 Eaton, A. A., Ramjee, D. and Saunders, J. F., 2022. The Relationships between Sextortion during COVID-19 and Pre-pandemic Intimate Partner Violence: A Large Study of Victimization among Diverse U.S Men and Women, Victims & Offenders, 18(2). [accessed 22 August 2023].
979 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 980 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 21 November 2024]. 981 Flynn, A., Powell, A., Scott, A. J., and Cama, E. (2022). Deepfakes and Digitally Altered Imagery Abuse: A Cross-Country Exploration of an Emerging form of Image-Based Sexual Abuse. The British Journal of Criminology, 62(6). [accessed 06 February 2024]. 178
982 Ofcom, 2023. Q8 (Table 234) in Online Experiences Tracker: Data Tables (Waves 1 and 2). [accessed 4 September 2023]. 983 7% of Black, Afro-Caribbean or African women and 5% of Native American or Alaskan Native women reported sextortion, while 2.4% of Latinas, 2% of Asian women, and 0.8% of white women reported sextortion. Source: Eaton, A., Ramjee, D. and Saunders, J., 2022. The Relationships between Sextortion during COVID-19 and Pre-pandemic Intimate Partner Violence: A Large Study of Victimization among Diverse U.S Men and Women, Victims & Offenders, 18 (2). [accessed 22 August 2023]. 984 7.1% of lesbian participants, 8.9% of bisexual participants and 6.3% of participants who identified as “other” sexual orientation reported sextortion, compared to 2.1% of gay participants and 2.9% of heterosexual participants. Source: Eaton, A., Ramjee, D. and Saunders, J., 2022. 985 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 21 November 2024]. 986 Flynn, A., Powell, A., Scott, A. J., and Cama, E. (2022). 987 For example, in research conducted by Revealing Reality, girls from more disadvantaged backgrounds were more likely to report that nude images they had shared had been shown and/or distributed without their permission; girls with multiple indicators of disadvantage were twice as likely to report that someone they had sent a picture to had sent it on without consent, compared to less disadvantaged girls. This research was predominantly focused on under 18s, therefore, some of the activity or experiences being discussed would be considered child sexual abuse (see the CSEA chapter), but the relationship identified between negative experiences and socio-economic indicators is stark. Source: Revealing Reality, 2022, Not just flirting: The unequal experiences and consequences of nude image-sharing by young people. [accessed 18 August 2024].
988 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 21 November 2024]. 179
Ofcom research.989 In recent research by the Revenge Porn Helpline, they found that 3 per cent of the websites featuring intimate images were sites which focused on individuals from South Asian descent.990 Additionally, women belonging to these groups may experience increased effects including inability to return to a home country.991
Risk factors: Functionalities and recommender systems¶
User identification¶
Fake user profiles¶
Anonymous user profiles¶
989 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 21 November 2024]. 990 These sites are referred to as “Desi” sites, a term used to identify individuals whose descent comes south Asian countries, including India, Pakistan, Bangladesh, Sri Lanka, the Maldives, Nepal, and Bhutan. In the pornography context, the term ‘desi porn’ has come to be used colloquially as identifying content which explicitly focuses on performers who can be identified as Indian or Pakistani. Source: Revenge Porn Helpline (Huber, A. and Ward, Z.), 2024. Non-Consensual Intimate Image Distribution: Nature, Removal, and Implications for the Online Safety Act. [accessed 10 October 2024]. 991 Revenge Porn Helpline (Huber, A. and Ward, Z.), 2024. 992 Study based on information provided by women from culturally and linguistically diverse backgrounds who have experienced technology-facilitated abuse (n=29) and 20 stakeholders who provide support services to women. 993 eSafety Commissioner, 2019. eSafety for Women from Culturally and Linguistically Diverse Backgrounds: Summary Report. [accessed 22 August 2023]. 994 Threats, Harassment and Stalking chapter includes a study from Refuge where one individual tried to block her former partner, only to find over 120 fake accounts created by him over a few weeks to continue harassing her. Source: Refuge, 2022. Marked as Unsafe: How online platforms are failing domestic abuse survivors. [accessed 22 August 2023].
995 Woods, L. and McGlynn, C., 2022. Pornography platforms, the EU Digital Services Act and Image-based sexual abuse, Media@LSE, 26 January. [accessed 22 August 2023]. 996 eSafety Commissioner, 2017. Image-based abuse. National survey: summary report. [accessed 22 August 2023]; Revenge Porn Helpline (Ward, Z.), 2021. Intimate image abuse, an evolving landscape. [accessed 4 August 2023]. 180
User networking¶
User groups¶
User communications¶
Livestreaming¶
Direct messaging¶
Encrypted messaging¶
997 For example, in one case in Australia a perpetrator filmed consensual sexual activity between himself and a victim and survivor and used a livestream to non-consensually share the image with a second perpetrator. Source: BBC News, 2013. Australia cadets online sex case: Two convicted, BBC News, 28 August. [accessed 22 August 2023].
998 Revenge Porn Helpline (Ward, Z.), 2021. 999 Sample was recruited online and consisted of 1,631 18 to 25 year olds who had been targets of threats to expose sexual images. Source: Thorn (Wolak, J. and Finkelhor, D.), 2016. Sextortion: Findings from a survey of 1,631 victims. [accessed 4 August 2023]. 1000 Refuge, 2021. Unsocial Spaces: make online spaces safer for women and girls. [accessed 22 August 2023]. 181
were often sent through messaging services which were end-to-end encrypted and not proactively searched by a moderation team.1001
Group messaging¶
Posting content (images, videos) and re-posting and forwarding content¶
1001 Revenge Porn Helpline (Ward, Z.), 2022. Revenge Porn Helpline Report. [accessed 22 August 2023]. 1002 BBC World Service Disinformation Team, 2022. Why won’t Telegram take down my naked photos?, BBC News, 20 February. [accessed 22 August 2023].
1003 eSafety Commissioner, 2019. Understanding the attitudes and motivations of adults who engage in image-based abuse. [accessed 22 August 2023]. 1004 In one case, provided by the Law Commission, a woman’s ex-partner set up a fake Facebook account in her name and uploaded intimate images of her, which were then viewed and copied to user-to-user pornography services. On one website the picture was viewed over 48,000 times. Source: Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 182
Transactions and offers¶
Content exploring¶
Content tagging¶
Content storage and capture¶
Downloading content¶
1005 Law Commission, 2021. 1006 Law Commission, 2021.
1007 Vera-Gray, F., McGlynn, C., Kureshi, I. and Butterby, K., 2021. Sexual violence as a sexual script in mainstream online pornography, The British Journal of Criminology, 61 (5). [accessed 22 August 2023]. 1008 #NotYourPorn is a UK-based movement focused on protecting non-consenting adults, sex workers and under-18s from image-based sexual abuse. 1009 Dawson, B., 2020. Revenge Porn Is Being Posted Under a Different Name, Vice, 15 December. [accessed 22 August 2023]. 183
download content facilitates intimate image abuse as it allows users to possess intimate images which they will then share further.
Screen capturing and recording¶
Content editing¶
Editing visual media¶
Recommender systems¶
1010 Law Commission, 2021. Intimate Image Abuse: A consultation paper. [accessed 3 August 2023]. 1011 Thorn (Wolak, J. and Finkelhor, D.), 2016. Sextortion: Findings from a survey of 1,631 victims. [accessed 4 August 2023]. 1012 Deepfakes are a specific type of media that involves the use of AI algorithms, particularly generative AI models, to create and/or modify videos, images or audio to create realistic synthetic content. This can be done by superimposing the face of a person onto the body of another person in a video or image as well as voice manipulation with lip syncing. Generative AI models can also create entirely new synthetic content, including sexual synthetic content of existing individuals. Deepfakes are shared as user generated content on user-to-user services but could also potentially be created using functionalities present on user-to-user services. The technology used for this purpose has many legitimate and beneficial applications, but we are concerned here only with its misuse in the generation of deepfake intimate images.
1013 Sensity AI, 2020. Automating Image Abuse: Deepfake bots on Telegram. [accessed 15 February 2024]. 1014 Glitch, 2023. AI Deepfake Roundtable 1. [accessed 17 January 2024]. 1015 Glitch, 2023. 1016 Rahman-Jones, I., Taylor Swift deepfakes spark calls in Congress for new legislation, BBC News, 27 January [accessed 19 November 2024]. 184
engagement in the form of likes, shares, and comments. Once in an open channel of communication, there is also the risk of such images being disseminated via direct messaging features.
Risk factors: Business models and commercial profile¶
Section 7 Extreme pornography offence¶
Warning: this chapter contains content that may be upsetting or distressing, including examples of sexually violent acts.
Summary analysis for the extreme pornography content offence: how harm manifests online, and risk factors Material considered ‘extreme pornography’ can include assault, rape and violence. There is limited evidence on the possession of extreme pornographic content, for several reasons, including the ethical and legal limitations on conducting research into it. However, some evidence suggests that there is a link between extreme pornographic material and CSAM, with a common pathway to CSAM online being the consumption of legal, and then increasingly problematic and potentially extreme pornography. We therefore draw similarities in risks where appropriate. Service type risk factors: User-to-user pornography services that provide user-generated pornography may be at a higher risk of showing or setting out that they offer extreme pornographic content. User base risk factors: Insights about demographic risks tend to concern perpetrators rather than victims or survivors. Crime data indicates that the creation or posting of extreme pornographic content, which suggests also the viewing of this content, is primarily committed by men, which indicates that gender – in relation to having a male-skewed user base – could be a risk factor. Functionalities and recommender systems risk factors: Extreme pornography can be facilitated by posting content, in this case, images and videos, on user-to-user (U2U) services. The ability to search for user-generated content (UGC) on U2U services may also help users find extreme pornography content. Additionally, other functionalities can be involved in the perpetration of this offence. Hyperlinks may also take a user from a U2U service with legal content to services with more extreme and potentially illegal content; this could include extreme pornographic content. Inferences from similar offences such as CSAM (see the Child sexual exploitation and abuse chapter, indicate that the ability to download content enables users to store and view extreme pornographic content and to share it with others. Anonymous profiles also give perpetrators confidence that they can avoid detection and are likely to increase the risk of extreme pornographic content being present online. 186
Content recommender systems also appear to play a role in suggesting increasingly extreme pornographic content to users. User groups can facilitate users viewing and sharing extreme pornography, with users exchanging content with like-minded individuals. It is possible that the ability to edit visual media can lead to the creation of realistic-looking deepfake extreme pornographic content. Livestreaming could allow users to broadcast extreme pornographic content in real-time. Being able to post goods and services for sale can amplify the risk posed by live streaming, as livestream sessions can be selected and purchased by users. Business model risk factors: The revenue models of some online user-to-user pornography services rely on ensuring a supply of new content to maintain and increase their user base. This applies to advertising and subscription-based revenue models, which can incentivise these services to allow content to be uploaded in the most ‘friction-free’ manner to maximise user engagement and minimise the cost of moderation. These services may, consequently, be less able to effectively detect and moderate extreme pornographic content.
Introduction¶
Relevant offences¶
• possession of extreme pornography1017
How the extreme pornography offence manifests online¶
1017 In England, Wales and Northern Ireland, this falls under section 63 of the Criminal Justice and Immigration Act 2008. 1018 McGlynn, C. and Bows., H., 2019. Possessing Extreme Pornography: Policing, Prosecutions and the Need for Reform, The Journal of Criminal Law, 83(6). [accessed 18 November 2024]. 1019 The study used data from 591 cases obtained from 33 police forces across England and Wales regarding charging and recording of this offence between 1st April 2015 and 31st March 2017. 254 recorded incidents in 2015-16 and 337 for 2016-17.
1020 McGlynn, C. and Bows., H. 2019. 1021 McGlynn, C. and Bows., H. 2019. 1022 Crown Prosecution Service, 2019. Violence against women and girls report 2018-19. [accessed 4 September 2023]. 1023 UK Government, 2020. The relationship between pornography use and harmful sexual behaviours. [accessed 05 November 2024]. 188
Risks of harm to individuals presented by the extreme pornography offence¶
1024 Upton, J., Hazell, A., Abbott, R. & Pilling, K. (The Behavioural Architects on behalf of the Government Equalities Office and Women and Equalities Unit), 2020. The relationship between pornography use and harmful sexual behaviours. [accessed 29 October 2024]. 1025 For their study the authors used the World Health Organisation definition of sexual violence, which is broader than the legal threshold for extreme pornography. They focused on four broad categories of sexual violence: sexual activity between family members; aggression and assault; image-based sexual abuse and coercive and exploitative sexual activity. Source: Vera-Gray, F., McGlynn, C., Kureshi, I., Butterby, K., 2021. Sexual violence as a sexual script in mainstream online pornography, The British Journal of Criminology, 61(5), pp.1-18. [accessed 18 November 2024].
1026 Vera-Gray, F., McGlynn, C., Kureshi, I., Butterby, K., 2021. 1027 Jones, S. and Mowlabocus, S., 2009. Hard Times and Rough Rides: The Legal and Ethical Impossibilities of Researching ‘Shock’ Pornographies, Sexualities, 12(5), pp. 613-628. [accessed 18 November 2024]. 189
normalise the acts depicted.1028 This includes depictions of gender-based violence, including sexual violence, rape and sexual assault, as well as high risk sexual behaviours such as explicit and realistic depictions of life-threatening injury.
1028 McGlynn, C. and Bows., H., 2019. Possessing Extreme Pornography: Policing, Prosecutions and the Need for Reform, The Journal of Criminal Law, 83(6). [accessed 18 November 2024]. 1029 Ethan A. Marshall, Holly A. Miller, Jeffrey A. Bouffard, PhD. 2018. Bridging the Theoretical Gap: Using Sexual Script Theory to Explain the Relationship Between Pornography Use and Sexual Coercion, Journal of Interpersonal Violence, 36(9-10). [accessed 29 October 2024]. 1030 During the development of legislation on extreme pornography, the Home Office stated that it is “possible that such material may encourage or reinforce interest in violent and aberrant sexual activity to the detriment of society as a whole”. Source: Home Office, 2005. Consultation: On the possession of extreme pornographic material. [accessed 4 September 2023]. 1031 Researchers have argued that the availability of extreme pornographic content, including rape and non-consensual sexual penetration, sustains a culture in which sexual violence is not only not taken seriously, but risks creating a culture in which it is normalised. Source: Vera-Gray, F., McGlynn, C., Kureshi, I., Butterby, K., 2021. Sexual violence as a sexual script in mainstream online pornography, The British Journal of Criminology, 61(5), pp.1-18. [accessed 29 October 2024]. 1032 McGlynn, C. and Rackley, E., 2009. Criminalising extreme pornography: a lost opportunity, Criminal law review, 4, pp. 245-260. [accessed 29 October 2024]. 1033 Drummond, A., Sauer, J. D. and Ferguson, C. J., 2020. Do longitudinal studies support long-term relationships between aggressive game play and youth aggressive behaviour? A meta-analytic examination. Royal Society Open Science, 7:200373. [accessed 29 October 2024]. 1034 Upton, J., Hazell, A., Abbott, R. & Pilling, K. (The Behavioural Architects on behalf of the Government Equalities Office and Women and Equalities Unit), 2020. The relationship between pornography use and harmful sexual behaviours. [accessed 29 October 2024]
1035 Antoniou, A. and Akrivos, D., 2017. The Rise of Extreme Porn—Legal and Criminological Perspectives on Extreme Pornography in England & Wales cited in McGlynn, C. and Bows., H., 2019. Possessing Extreme Pornography: Policing, Prosecutions and the Need for Reform, The Journal of Criminal Law, 83(6). [accessed 29 October 2024]. 1036 Gilbody-Dickerson, C., 2023. Adam Britton: What we know about man who pleaded guilty to ‘grotesque’ sexual abuse of dozens of dogs, i¸26 September. [accessed 27 September 2023] 190
suggest a link between the two offences, but the sample is too small to draw any definitive conclusions.
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
User-to-user pornography services¶
1037 McGlynn, C. and Woods, L. 2022. Pornography and Online Safety Bill. [accessed 29 October 2024] 1038 Cole, S., 2020. A new wave of reckoning is sweeping the porn industry. VICE, 10 June. [accessed 4 September 2023]. 1039 In some of these cases the violent act being filmed may not meet the legal threshold for extreme pornographic content. But it is possible to assume that if actors experience boundary violation and non-consensual acts during the filming of more mainstream pornographic content, it is likely that actors involved in the making of extreme pornographic content are also likely to be affected. 191
defined by the study.1040 The British Board of Film Classification (BBFC),1041 found that in a study of young people and pornography, most respondents said they were exposed to upsetting or disturbing videos (usually related to violent or aggressive pornography) for the first time through “videos appearing on homepages of pornography sites or as a suggested video”.1042
Messaging Services¶
Risk factors: User base¶
User base demographics¶
1040 Vera-Gray, F., McGlynn, C., Kureshi, I., Butterby, K., 2021. Sexual violence as a sexual script in mainstream online pornography, The British Journal of Criminology, 61(5), pp.1-18. See How the extreme pornography offence manifests online section for more information. 1041 The BBFC is the UK’s regulator of film and video. It is responsible for the 18 classifications for sex work, and the R18 category for legally-restricted content, primarily for explicit works of consenting sex or strong fetish material involving adults. 1042 BBFC, 2020. Young people, pornography and age-verification. [accessed 6 September 2023]. 1043 Vera-Gray, F., and McGlynn, C., 2021. Sexually violent pornography is being promoted to first-time users of top sites. [accessed 6 September 2023].
1044 News.com.au, 2023. ‘I can’t stop. I don’t want to’: Dog rapist sent disturbing Telegram messages about sordid urges, 26 September. [accessed 26 September]. 1045 Zaccaro, M. and PA Media, 2023. Met Police officer jailed over extreme pornographic image, BBC News, 17 March. [accessed 27 September 2023]. Sharman, D., 2023. Police probe ‘digital sex abuse’ after female journalists sent extreme porn, HoldtheFrontPage.co.uk, 27 September. [accessed 27 September 2023]. 192
therefore reasonable to assume that more men than women may commit offences relating to extreme pornography.1046
Risk factors: Functionalities and recommender systems¶
User identification¶
Anonymous user profiles¶
User networking¶
User groups¶
User communications¶
Direct messaging and encrypted messaging¶
Livestreaming¶
Posting content (image, video)¶
1046 McGlynn, C. and Bows., H. 2019. Possessing Extreme Pornography: Policing, Prosecutions and the Need for Reform, The Journal of Criminal Law, 83(6). [accessed 29 October 2024]. 1047 News.com.au, 2023. ‘I can’t stop. I don’t want to’: Dog rapist sent disturbing Telegram messages about sordid urges, 26 September. [accessed 26 September]. 193
the service.1048 Christian Action, Research and Education (CARE)1049 said that this action showed that “large U2U services are unsure and are unable to record the levels of illegal and extreme material hosted on their services.”1050 There is a risk that services which attract adult content, particularly where illegal content has already been found, may also be the types of services where extreme pornography is found.1051
Transactions and offers¶
Post goods or services for sale¶
Content exploring¶
User-generated content searching¶
Building lists or directories¶
Hyperlinking¶
1048 Concerns were not necessarily raised about extreme pornography material, but primarily child sexual exploitation and abuse material and intimate image abuse (IIA).
1049 CARE is a Christian public policy charity based in the United Kingdom. 1050 CARE response to 2022 Ofcom Call for Evidence: First phase of online safety regulation. 1051 This is not a specific observation about potentially illegal and extreme content currently available on Pornhub. However, Pornhub’s actions help us draw an inference that extreme pornographic content may exist on U2U services. 1052 Vera-Gray, F. and McGlynn, C., 2021. Sexual Violence in Mainstream Online Pornography. [accessed 6 September 2023]. 194
pornography and provide a relatively easy user journey to sites showing more niche or even extreme pornographic content.
Content storage and capture¶
Downloading content¶
Content editing¶
Editing visual media¶
Recommender systems¶
Content recommender systems¶
1053 Deepfakes are a specific type of media that involves the use of AI algorithms, particularly generative AI models, to modify videos, images or audio to create realistic synthetic content. This is often done by superimposing the face of a person onto the body of another person in a video or image as well as voice manipulation with lip syncing. Deepfakes are shared as user generated content on user-to-user services but could also potentially be created using functionalities present on user-to-user services. Deepfake technology is currently used to create content that can be harmful; however, we acknowledge that it may also have positive use cases.
1054 McGlynn, C., and Woods, L., 2022. Image-Based Sexual Abuse, Pornography Platforms and the Digital Services Act. [access 6 September 2023]. 195
where they found increasingly violent videos. 1055 This content may not necessarily constitute extreme pornographic content, but the evidence indicates how the design of recommender systems can lead to users being exposed to increasingly extreme content. We think it is reasonable to assume that where extreme pornographic content is present on a service and has not been detected and taken down by content moderation processes, this content may be recommended to users who are not necessarily looking for it directly.
Risk factors: Business models and commercial profiles¶
Revenue models¶
Advertisement-based revenue models¶
1055 BBFC, 2020. Young people, pornography and age-verification. [accessed 21 September 2023]. 1056 Centre to End All Sexual Exploitation, 2021. Expose Big Porn. [accessed 21 September 2023]. 1057 Mohan, M., 2020, Call for credit card freeze on porn sites, BBC News, 8 May. [accessed 21 September 2023]; Goodwin, J., 2020. Visa continues its ban on Pornhub but will allow payments on some of its parent company's sites, CNN, December 23. [accessed 21 September 2023]. 196
Section 8 Sexual exploitation of adults¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for sexual exploitation of adults offences: how harm manifests online, and risk factors This chapter analyses the risks of harm arising from offences relating to the sexual exploitation of adults – this includes adults who have been forced into sex work as well as consenting adult sex workers who are being exploited. In this chapter, we use the terms ‘adult sex worker’ or ‘victim and survivor’ rather than ‘prostitute’ and ‘prostitution’, unless referencing the legislation that uses those terms. This is to align with widely accepted terminology conventions which seek to better reflect the experiences and dynamics in this area. The International Labour Organization estimates that 6.3 million people globally are currently experiencing forced commercial sexual exploitation and that nearly four out of five are women or girls. We recognise that the same characteristics identified as risk factors in sexual exploitation of adults can at times also be safety measures for adult sex workers. For example, while social media services, marketplaces and listings services have been identified as potential risk factors there is also evidence that advertising sexual services online is generally safer than soliciting in public places. The risk of harms to individuals from sexual exploitation of adults offences include both physical and psychological effects. Victims and survivors may suffer from threats of physical abuse, rape or sexual violence. This can also have an effect on their mental health, with effects including anxiety, depression, self-harm and post-traumatic stress disorder. The risk factors identified below may lead to individuals experiencing the risks of harm from these offences. Service type risk factors: Social media services are likely to be used by potential perpetrators to recruit victims and to advertise the services of the victims and survivors they have gained control of for sexual exploitation. Marketplaces and listings services are also used to advertise services. Messaging services, particularly those with encryption, can also be used by potential perpetrators to communicate with victims and survivors. User base risk factors: Gender and age of users are risk factors - there is evidence that women and younger people are more vulnerable to exploitation. Other user base demographics can also be risk factors; individuals with intellectual disabilities, language barriers or who are homeless are more vulnerable to exploitation. 197
Functionalities and recommender systems risk factors: The ability to post goods or services for sale, such as through advertisements, also enables perpetrators to advertise and broadcast the sexual services of adults in exploitative environments. Encrypted messaging is also a functionality that can be used by buyers and abusers as a tool to arrange the transaction of services, while user profiles can also be used to identify individuals. These functionalities enable the commission of the offence ‘controlling a prostitute for gain’. Other functionalities also present risk of harm from this offence. Livestreaming can be used by perpetrators to advertise and broadcast exploitation, with these streams reaching a large global base of potential consumers. Evidence suggests that the ability to post or send location information can also be used to identify and target individuals. Direct messaging is used for communication between perpetrators, buyers, and victims and survivors. Business model risk factors Services that generate revenue through advertising can be at risk, as offenders may be able to use adverts to lure victims, who can then be coerced or controlled into sexual activities, and it has been recognised that some online services have profited from selling adverts for sexual services provided by potential victims and survivors of this kind of coercion and control.
Introduction¶
Relevant offences¶
How sexual exploitation of adults manifests online¶
1058 Section 52 of the Sexual Offences Act 2003; Article 62 of the Sexual Offences (Northern Ireland) Order 2008 (S.I. 2008/1769 (N.I. 2)). 1059 Section 53 of the Sexual Offences Act 2003; Article 63 of the Sexual Offences (Northern Ireland) Order 2008 (S.I. 2008/1769 (N.I. 2)). 199
that 6.3 million people are currently experiencing forced commercial sexual exploitation and that nearly four out of five are women or girls.1060
Risks of harm to individuals presented by offences relating to the sexual exploitation of adults¶
Causing or inciting prostitution for gain¶
1060 International Labour Organization (ILO), 2022. Global Estimates of Modern Slavery: Forced Labour and Forced Marriage. [accessed 5 July 2023]. 1061 Beyond the Gaze (Sanders, T., Scoular, J., Pitcher, J., Campbell, R. and Cunningham, S), 2018. Beyond the Gaze: Summary Briefing on Internet Sex Work. [accessed 5 July 2023].
1062 Human Trafficking Institute, 2021. Federal Human Trafficking Report 2020. [accessed 5 July 2023]. 1063 Human Trafficking Institute, 2021. 1064 McQuaid, J., 2020. Understanding the psychological effects of sex trafficking to inform service delivery, Forced Migration Review. [accessed 5 July 2023]. 1065 McQuaid, J., 2020. 200
Controlling a prostitute for gain¶
Evidence of risk factors on U2U services¶
Risk factors: Service types¶
social media services, video-sharing services, and messaging services.¶
Social media services, marketplaces and listings services¶
1066 National Police Chiefs Council, 2024. Sex Work National Police Guidance. [accessed 14 October 2024].
1067 National Police Chiefs Council, 2024. 1068 Stop the Traffik, 2021. Sex Work and Exploitation: What do you need to know?. [accessed 5 July 2023]. 1069 While our evidence only names online marketplaces, we expect similar risks of harm to arise from listings services due to similarities in the characteristics typically found on these service types. 1070 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons 2020. [accessed 5 July 2023]. 1071 United Nations Office on Drugs and Crime, 2020. 201
Messaging services¶
Video-sharing services¶
Risk factors: User base¶
User base demographics¶
8.29
1072 United Nations Office on Drugs and Crime, 2020. 1073 Thorn, 2018. Survivor Insights: The role of technology in domestic minor sex trafficking. [accessed 5 July 2023]. 1074 See Risk factors: functionalities and recommender systems section for more information. 1075 See Risk factors: functionalities and recommender systems section for more information. 202
commercial sexual exploitation and the completion of higher education, suggesting that lower academic attainment may also be a risk factor.1076
1076 Reid, J. 2014. Risk and resiliency factors influencing onset and adolescence‐limited commercial sexual exploitation of disadvantaged girls, Criminal Behaviour and Mental Health, 24 (5), p.332-344. 1077 ILO, 2022. Global Estimates of Modern Slavery: Forced Labour and Forced Marriage. [accessed 5 July 2023]. 1078 National Police Chiefs’ Council, 2023. Violence Against Women and Girls: Strategic Threat Risk Assessment. [accessed 5 July 2023]. 1079 Preventing Exploitation Toolkit, n.d. Communication difficulties. [accessed 5 July 2023]. 1080 Preventing Exploitation Toolkit, n.d. Communication difficulties. [accessed 5 July 2023].
1081 The Rights Lab/ECPAT UK (Celiksoy, E., Schwarz, K., Sawyer, L., Gorena, P. V., Ciucci, S., Yin, S. & Durán, L.) 2024. Prevention and identification of children and young adults experiencing, or at risk of, modern slavery in the UK. [accessed 28 October 2024]. 1082 International Organization for Migration (IOM), 2019. Handbook on Protection and assistance to migrants vulnerable to violence, exploitation and abuse. [accessed 13 November 2024]. 1083 Preventing Exploitation Toolkit, n.d. Communication difficulties. [accessed 5 July 2023]. 203
Risk factors: functionalities and recommender systems¶
User identification¶
User profiles and fake user profiles¶
User communication¶
Livestreaming¶
Direct messaging and encrypted messaging¶
1084 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons 2020. [accessed 5 July 2023]. 1085 Thorn, 2018. Survivor Insights: The role of technology in domestic minor sex trafficking. [accessed 5 July 2023]. 1086 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons 2020. [accessed 5 July 2023]. 204
messages with a client allows an adult sex worker to assess the safety concerns they have about the client before meeting in person.
Posting or sending location information¶
Transactions and offers¶
Posting goods or services for sale¶
1087 Beyond the Gaze, 2018. Safety and Privacy for Online Sex Workers. [accessed 5 July 2023]. 1088 United Nations Office on Drugs and Crime, 2020. 1089 L’Hoiry, X., Moretti, A. & Antonopoulos, G. A. 2021. Identifying sex trafficking in Adult Services Websites: an exploratory study with a British police force, Trends in Organised Crime. Vol. 27. [accessed 13 November 2024]. 1090 These include: the use of third- or first-person plural pronouns; the same phone number used in more than one advertisement; a high degree of similarity between sex workers’ advertisements; sex workers offering risky or violent sexual services; advertisements promoting inexpensive sex services; sex workers moving frequently between several locations; sex workers moving to a different location along with other sex workers; sex workers offering in-call services only; advertisements using words alluding to the youthful characteristics of the sex workers; and stating a dress size typical of underage women.
1091 Most advertisements in the study are said to provide ‘demographic information about the sex worker (e.g. town where they are active, nationality, age, etc.), information on their physical appearance (e.g. height, hair and eye colour, etc.), sexual orientation (e.g. bi-sexual, heterosexual, etc.), sexual services provided, and pricing. Moreover, most 205
this includes posted content (as opposed to paid advertising) which, in turn, has the effect of advertising a service by offering it for sale.
Risk factors: Business models and commercial profiles¶
Revenue models¶
Advertising-based models¶
advertisements have free text spaces that the sex workers use to introduce themselves, a public and private gallery to post pictures and videos, and an ‘interview’ section where sex workers provide more details about themselves and their services.’ 1092 Giommoni, L. and Ikwu, R., 2021. Identifying human trafficking indicators in the UK online sex market, Trends in Organized Crime. [accessed 19 September 2024].
1093 National Police Chiefs Council, 2024. Sex Work National Police Guidance. [accessed 4 March 2024]. 1094 Crown Prosecution Office, 2019. Prostitution and Exploitation of Prostitution. [accessed 20 September 2023]. 1095 See also the Register of Risks chapter ‘Human trafficking’ for information on related offences. 1096 Blunt, D. & Wolfe, A. 2020. Erased: The Impact of FOSTA-SESTA. [accessed 11 September 2024]. 1097 See also Albert, K., et al. 2021. FOSTA in Legal Context. [accessed 26 November 2024]. Chamberlain, L. 2019. FOSTA: A Hostile Law with a Human Cost. [accessed 26 November 2024]. 206
opportunity to advertise can be used by perpetrators, including sexual traffickers, to entice victims into a situation where they are captured, controlled and coerced into sexual activities.
1098 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons 2020. [accessed 5 July 2023]. 1099 “Examples of advertisements used to attract victims often include wording that describes the possibility of living a luxurious life or promising jobs in industries such as modelling or entertainment”. Source: United Nations Office Drugs and Crime, 2020. 1100 Thorn, 2018. Survivor Insights: The role of technology in domestic minor sex trafficking, p38 [accessed 5 July 2023]. 207
Section 9 Human trafficking¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for human trafficking offence: how harm manifests online, and risk factors Human trafficking encompasses a wide range of harmful activities. It can involve modern slavery, and victims and survivors include adults and children. It is estimated that there were 122,000 people living in modern slavery in the UK in 2021. Notable forms of human trafficking where harm can manifest online include sexual exploitation and abuse, forced labour, and criminal exploitation such as county lines exportation of illegal drugs. An individual’s experience of harms from human trafficking offence are unique to their situation, but victims and survivors of these offences can be considered one of the most vulnerable groups at risk of complex mental health difficulties, as well as long-lasting physical health problems. Service type risk factors: Social media services and messaging services are shown to be risk factors for human trafficking offence. Among other elements, they are used to target potential victims and advertise services. Perpetrators can identify their victim on social media before transitioning to private messaging services to further the facilitation of the offence. Marketplace and listing services, particularly adult services websites are recognised as risky in the context of the human trafficking, sexual exploitation and abuse in particular. This is because these types of websites are often used to advertise services provided by victims who have been trafficked and are being coerced. Functionalities and recommender systems risk factors: User profiles, including fake profiles used to hide the perpetrator’s real identity and manipulate a victim/survivor, can be exploited by a perpetrator looking to build trust with their victim. Posting content is also used to lure victims and promote illegal services in human trafficking offence. Direct and encrypted messaging are a risk factor as they are used by traffickers to recruit and communicate with victims, as well as for communication between those involved in exploitation. Business model risk factors: 208
Services providing classified listings or other advertising opportunities may increase risk, as the services are incentivised to maximise advertising revenues. The opportunity to advertise can be used by traffickers to reach and attract potential victims, who can then be lured into a situation where they are captured, controlled and coerced. Where sex workers are victims of human trafficking, online listings that advertise sex workers’ services may also present a risk of facilitating human trafficking.
Introduction¶
• the use of these services for the commission and/or facilitation of this offence (collectively, the ‘risks of harm’).
Relevant offences¶
1101 ‘People smuggling’ and ‘people trafficking’ are different concepts in law. Offences relating to ‘people smuggling’ will generally relate to the Immigration Act offences, whereas ‘people trafficking’ will generally be offences under the Modern Slavery Act, Crown Prosecution Service, Updated 6 July 2022. Source: Crown Prosecution Service, 2022. Modern Slavery, Human Trafficking and Smuggling. [accessed 25 September 2023]. 1102 See the Register of Risks chapters ‘Unlawful immigration’ and ‘Child sexual exploitation and abuse (CSEA)’. 209
a) arranging or facilitating the travel of another person, or taking a relevant action, with a view to them being exploited (human trafficking)1103
How human trafficking offence manifest online¶
1103 Section 2 of the Modern Slavery Act 2015; section 1 of the Human Trafficking and Exploitation (Scotland) Act 2015 (asp 12); section 2 of the Human Trafficking and Exploitation (Criminal Justice and Support for Victims) Act (Northern Ireland) 2015 (c. 2 (N.I.)).
1104 Some of the evidence in this chapter draws on evidence from outside of the UK context, where this is relevant to build understanding of the risks of harm people face, as well as the role of online services in facilitating offences. 1105 Lovett, S., 2023. Foreign diplomats trap and abuse domestic workers in private households across London. The Telegraph, 2 February. [accessed 14th August 2024]. 210
harvesting.1106 The concept of modern slavery is used to describe many kinds of human trafficking, indicating the harmful as well as often-hidden nature of these offences.
1106 Reported cases of organ harvesting in the UK are rarer, but examples include 3 people found guilty of exploiting a vulnerable victim for illegal organ harvesting in 2023. Source: Crown Prosecution Service, 2023. Updated with sentence: Senior Nigerian politician jailed over illegal UK organ-harvesting plot. [accessed 23 September 2024] 1107 ONS, 2020. Modern slavery in the UK: March 2020. [accessed 12th August 2024]. 1108 International Labour Organisation, 2022. Global Estimates of Modern Slavery: Forced Labour and Forced Marriage. [accessed 13 August 2024]. 1109 Save the Children, n.d. The Fight Against Child Trafficking. [accessed 25 September 2023]. 1110 UNODC, 2019. Children on the move, smuggling and trafficking. [accessed 25 September 2023]. 1111 Potential victims of modern slavery in the UK who come to the attention of authorised first responder organisations are referred to the National Referral Mechanism (NRM). Authorised first responder organisations include local authorities, specified non-governmental organisations (NGOs), police forces and specified government agencies. Adults (aged 18 or above) must consent to being referred to the NRM, whilst children under the age of 18 need not consent to being referred. Home Office, 2024. Modern Slavery: National Referral Mechanism and Duty to Notify statistics UK, end of year summary 2023. [accessed 12th August 2024].
1112 Walk Free, 2023. Global Slavery Index: United Kingdom. [accessed 17 October 2024]. 1113 For example, government guidance about county lines notes that in cases of criminal exploitation, victims sometimes have pre-existing connections with perpetrators. Source: Home Office, 2023. Criminal exploitation of children, young people and vulnerable adults: county lines. [accessed 17 October 2024]. 1114 Council of Europe, 2022. Online and technology-facilitated trafficking in human beings. [accessed 17 October 2024]. 211
someone looking for an employment opportunity into thinking they are providing non-exploitative work for them.1115
Risks of harm to individuals presented by the human trafficking offence¶
1115 UNODC, 2020. Global Report on Trafficking in Persons 2020: Chapter 5. [accessed 17 October 2024]. 1116 Brown, E., 2022. Dozens of teenage girls trafficked in UK modern slavery first, Unilad, 10 February. [accessed 17 October 2024]. 1117 Children are trafficked for many of the same reasons as adults including, but not limited to, forced marriage, domestic servitude, forced labour, organ harvesting, criminal exploitation and sexual exploitation and can experience one of or a multitude of forms of abuse and exploitation. Source: NSPCC, n.d. Child Trafficking. [accessed 25 September 2023]. 1118 The CPS discusses exploitation and talks about extensive examples in its guidance here. Source: CPS, 2022. Modern slavery, human trafficking and smuggling. [accessed 25 September 2023]. 1119 National Crime Agency, n.d. Modern slavery and human trafficking. [accessed 17 October 2024]. 1120 Unseen UK, n.d. Frank’s story. [accessed 17 October 2024]. 1121 For example, the majority of referrals under the National Referral Mechanism for potential sexual exploitation are for women and girls. Source: Home Office, 2024. Modern Slavery: National Referral Mechanism and Duty to Notify statistics UK, quarter 2 2024 (April to June). [accessed 17 October 2024]. In addition, there is evidence of links between experiences of violence among sex workers and having experiences of being trafficked. Source: Deering, K.N., Amin, A., Shoveller, J., Nesbitt, A., Garcia-Moreno, C., Duff, P., Argento, E. and Shannon, K., 2014. A systematic review of the correlates of violence against sex workers. American Journal of Public Health, 104(5). [accessed 17 October 2024].
1122 Unseen (Garbers, K., Malpass, A., Saunders, L., Horwood, J., McLeod, H., Anderson, E. and Farr., M.), 2021. Impact of mobile technology for survivors of modern slavery and human trafficking. [accessed 25 September 2023]; Hestia, 2023. Underground lives: Forgotten children - The intergenerational impact of modern slavery. [accessed 17 October 2024]. 212
Sexual exploitation and abuse¶
1123 NSPCC, 2023. Protecting children from trafficking and modern slavery. [accessed 25 September 2023]. 1124 It is worth noting that many victims may experience multiple types of trafficking. This is captured in National Referral Mechanism data about potential modern slavery victims. 1125 All-Party Parliamentary Group on Commercial Sexual Exploitation, 2023. Pornography Regulation: the case for Parliamentary Reform. [accessed 17 October 2024]. All-Party Parliamentary Group on Commercial Sexual Exploitation, 2021. Bust the Business Model: How to stop sex trafficking and sexual exploitation in the UK. [accessed 17 October 2024]. Home Affairs Committee, 2023. Oral Evidence: Human Trafficking. [accessed 17 October 2024]. 1126 International Labour Organization, 2022. Global Estimates of Modern Slavery: Forced Labour and Forced Marriage. [accessed 17 October 2024]. 1127 Human Trafficking Institute, 2021. Federal Human Trafficking Report 2020. [accessed 25 September 2023]. 1128 Human Trafficking Institute, 2021. 1129 In one case a human trafficking survivor met their exploiter via a dating app. Source: Ashcraft, E., 2019. Human trafficking survivor shares how she fell victim after Tinder date, KSL News, 23 January. [accessed 17 October 2024].
1130 An example of a human trafficking survivor who was subjected to sexual abuse and violence is discussed in this news report. Source: Deas, A. and Mortimer, H., 2022. I was trafficked, raped, and left for my abusers to find, BBC News, 27 September. [accessed 17 October 2024]. 1131 McQuaid, J. 2020. Understanding the psychological effects of sex trafficking to inform service delivery. [accessed 25 September 2023]. 1132 McQuaid, J. 2020. 213
Criminal exploitation including county lines¶
1133 Department for Education, 2017. Child sexual exploitation. [accessed 25 September 2023]. 1134 Thorn, 2019. Survivor Insights: The Role of Technology in Domestic Minor Sex Trafficking. [accessed 17 October 2024]; Thorn, 2024. Survivor Survey. [accessed 17 October 2024]; Reuters, 2022. Former UK army officer jailed for online child sex abuse. [accessed 17 October 2024].; Baird, K., Connolly, J., 2021. Recruitment and Entrapment Pathways of Minors into Sex Trafficking in Canada and the United States: A Systematic Review. Trauma, Violence & Abuse, 24(1). [accessed 28 October 2024]. 1135 See for example, Pearson, J., Cavener, J. 2024. Professionals’ understanding of the County Lines phenomenon: Insights from a study exploring the perceptions of young peoples’ supported accommodation staff. Children and Youth Services Review, vol 156. [accessed 28 October 2024]. 1136 Baird, K., Connolly, J., 2021. Recruitment and Entrapment Pathways of Minors into Sex Trafficking in Canada and the United States: A Systematic Review. Trauma, Violence & Abuse, 24(1). [accessed 28 October 2024]
1137 Home Office, 2024. Modern Slavery: National Referral Mechanism and Duty to Notify statistics UK, end of year summary 2023. [accessed 12 August 2024]. These figures include 447 potential victims where more than one type of exploitation was highlighted in the referral. 1138 Independent Anti-Slavery Commissioner, 2021. Child trafficking in the UK 2021: a snapshot. [accessed 25 September 2023]. 214
of illegal items. Victims can be coerced or groomed into involvement, or not be aware they are involved in criminal activity.1139
1139 Hestia (Papadaki, H.), 2020. Underground Lives: Criminal Exploitation of Adult Victims. [accessed 18 October 2024]. 1140 NCA, 2023. NCA targets crime group suspected of operating slave labour cannabis farms. 25 January. [accessed 25 September 2023]. 1141 National Crime Agency, n.d. County Lines. [Accessed 12 May 2023]. 1142 Maxwell, N. and Wallace, C., 2021. Child Criminal Exploitation in Wales. [accessed 17 October 2024]. 1143 Home Office, 2024. Modern Slavery: National Referral Mechanism and Duty to Notify statistics UK, end of year summary 2023. [accessed 12th August 2024]. 1144 Children and Young People’s Centre for Justice, 2023. Understanding Child Criminal Exploitation in Scotland: A Scoping Review. [accessed 1st November 2024]. 1145 Whittaker et. Al, 2020. No two gangs are alike: The digital divide in street gangs’ differential adaptations to social media. [accessed 1st November 2024].
1146 Maxwell, N. and Wallace, C., 2021. 1147 Barnardo’s, 2023. Invisible Children: Understanding the risk of the cost-of-living crisis and school holidays on child sexual and criminal exploitation. [accessed 17 October 2024]. 1148 Campbell, C., 2021. County lines gang ‘recruited teen in 80 minutes via Snapchat’, BBC News, 14 April. [accessed 17 October 2024]. 215
of victims is made, and where online ‘collateral’ (for example, incriminating messages) can be used by perpetrators to ensure compliance.1149
Forced labour and labour exploitation¶
1149 Children’s Society response to the Protection of Children May 2024 Consultation. 1150 Public Health England, 2021. County Lines exploitation: applying All Our Health. [accessed 12 May 2023]. 1151 Rescue and response, 2019. Rescue and response county lines project. [accessed 12 May 2023]. 1152 Public Health England, 2021. 1153 Hestia (Papadaki, H.), 2020. Underground Lives: Criminal Exploitation of Adult Victims. [accessed 18 October 2024]. 1154 Children and Young People’s Centre for Justice, 2023. Understanding Child Criminal Exploitation in Scotland: A Scoping Review. [accessed 1st November 2024]. 1155 Home Office, 2024. Modern Slavery: National Referral Mechanism and Duty to Notify statistics UK, end of year summary 2023. [accessed 12 August 2024]. 1156 A relevant example in this news report involved a gang who forced 16 victims to work at a branch of McDonald’s and a factory. The gang took nearly all of the victims’ pay and housed them in poor living conditions. Source: McLennan, W., Shepka, P. and Ironmonger, J., 2024. McDonald’s and supermarkets failed to spot slavery, BBC News, 30 September. [accessed 18 October 2024].
1157 UK Parliament, 2023. Written evidence submitted by the NCA. [accessed 17 October 2024]. 1158 GLAA, 2023. Nepalese recruitment agency directors handed slavery order. [accessed 17 October 2024]. 1159 Europol, 2024. 51 persons arrested in crackdown on labour exploitation. [accessed 17 October 2024]. 216
Trafficking offences and migrant victims¶
Evidence of risk factors on user-to-user services¶
1160 Metropolitan Police, n.d. Modern Slavery. [accessed 25 September 2023]. 1161 Unseen, 2023. Who Cares? A Review of Reports of Exploitation in the Care Sector. [accessed 17 October 2024]. 1162 IOM, 2022. Migrants and their vulnerability. [accessed 17 October 2024]. 1163 Europol, 2024. 13 victims of human trafficking safeguarded in Spain. [accessed 17 October 2024]. 1164 Council of Europe, 2022. Online and technology-facilitated trafficking in human beings: Full report. [accessed 17 October 2024]. 1165 EU Monitor, 2023. Explanatory Memorandum. [accessed 12 August 2023].
1166 Europol, 2024. Tackling threats, addressing challenges - Europol’s response to migrant smuggling and trafficking in human beings in 2023 and onwards. [accessed 17 October 2024]. 1167 Home Office, 2024. Irregular migration to the UK, year ending June 2024. [accessed 17 October 2024]. 1168 Justice and Care, 2023. Modern Slavery Issue Brief: Addressing vulnerability to modern slavery in a growing tide of migration. [accessed 17 October 2024]. 1169 Programme Challenger, 2024. Greater Manchester guide to exploitation in the care sector. [accessed 17 October 2024]. 217
Risk factors: Service types¶
Social media services and messaging services¶
1170 UNODC, 2020. Global Report on Trafficking in Persons 2020: Chapter 5. [accessed 17 October 2024]. 1171 Administration for children and families (Contreras, J. and Chon, K.), 2022. Technology's complicated relationship with human trafficking. [accessed 25 September 2023]. 1172 Administration for children and families (Contreras, J. and Chon, K.), 2022. 1173 “In 2020, researchers identified a 125% year-on-year increase in the number of reports of trafficking recruitment on Facebook, and a 95% increase in similar reports on Instagram. Individuals often share posts, updates and content that describe their hobbies and interests and express their frustrations and hardships. Traffickers leverage this information to exploit people’s vulnerabilities and develop tactics to escalate manipulation, grooming individuals by offering empathy and support, forming emotional connections, and building trust and confidence. In cases of labour exploitation, traffickers will use social media to scout job seekers or those experiencing financial hardships and then use online job boards and employment websites to recruit them through false advertisements”. Source: Administration for children and families (Contreras, J. and Chon, K.), 2022.
1174 Centre for Social Justice, 2024. Criminal exploitation. [accessed 17 October 2024]. 1175 Crest (Caluori, J, Mooney, B. and Kirk, E.), 2022. Running out of credit: Mobile phone tech and the birth of county lines. [accessed 25 September 2023]. 1176 Alliance to Counter Crime Online, n.d. Human trafficking: How Social media Fuels Modern Day Slavery. [accessed 25 September 2023]. 1177 Rescue and response, 2019. Rescue and response county lines project. [accessed 12 May 2023]. 218
Marketplaces and listings services¶
User-to-user pornography services¶
1178 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons 2020. [accessed 5 July 2023]. 1179 Council of Europe, 2022. Online and technology-facilitated trafficking in human beings: Full report. [accessed 17 October 2024]. 1180 Children’s Society response to the Protection of Children May 2024 Consultation. pp. 8-9. 1181 All-Party Parliamentary Group on Commercial Sexual Exploitation, 2023. Pornography Regulation: the case for Parliamentary Reform; and All-Party Parliamentary Group on Commercial Sexual Exploitation, 2021. Bust the Business Model: How to stop sex trafficking and sexual exploitation in the UK. [both accessed 17 October 2024]. 1182 As noted in the Illegal Content Judgement Guidance, ASWs need not necessarily be a website, it can also be an app, forum or another service type.
1183 Modern Slavery & Human Rights, 2023. The role of adult service websites in addressing modern slavery. [accessed 13 August 2024]. 1184 Cross-Party Group on Commercial Sexual Exploitation, 2021. Online Pimping: An inquiry into Sexual Exploitation Advertising Websites. [accessed 17 October 2024]. 1185 NCA, n.d. Modern slavery and human trafficking. [accessed 25 September 2023]. 219
feature victims of trafficking.1186 There are also examples of CSAM being hosted on user-to-user pornography services featuring trafficking victims.1187
Gaming services¶
Risk factors: User base¶
User base size¶
User base demographics¶
9.64
1186 All-Party Parliamentary Group on Commercial Sexual Exploitation, 2023. Pornography Regulation: the case for Parliamentary Reform [accessed 17 October 2024]. 1187 BBC News, 2021. Pornhub owner settles with Girls Do Porn victims over videos, BBC News 19 October. [accessed 17 October 2024]. 1188 All-Party Parliamentary Group on Commercial Sexual Exploitation, 2021. Bust the Business Model: How to stop sex trafficking and sexual exploitation in the UK. [accessed 17 October 2024]; and UNODC, 2020. Global Report on Trafficking in Persons 2020: Chapter 5. [accessed 17 October 2024]. 1189 Children and Young People’s Centre for Justice, 2023. Understanding Child Criminal Exploitation in Scotland: A Scoping Review. [accessed 1st November 2024]. Council of Europe, 2022. Online and technology-facilitated trafficking in human beings: Full report. [accessed 17 October 2024].
1190 UK Parliament, 2023. Oral evidence submitted by the NCA. [accessed 17 October 2024]; Children’s Society response to the Protection of Children May 2024 Consultation.; Mararike, S., (2021) Dealers are using Fortnite treats to groom children as drug mules, The Sunday Times, 21 March. [accessed 17 October 2024]. 1191 Administration for children and families (Contreras, J. and Chon, K.), 2022. Technology's complicated relationship with human trafficking. [accessed 25 September 2023]. 220
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles and fake user profiles¶
1192 Unseen is a UK charity which provides safehouses and support in the community for survivors of trafficking and modern slavery. 1193 Unseen, n.d. Modern Slavery Facts and Figures. [accessed 25 September 2023]. 1194 Alliance to Counter Crime Online, n.d. Human trafficking: How Social media Fuels Modern Day Slavery. [accessed 25 September 2023]. 1195 The Rights Lab and ECPAT UK (Every Child Protected Against Trafficking), 2024. Prevention and identification of children and young adults experiencing, or at risk of, modern slavery in the UK. [accessed 17 October 2024].
1196 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons. [accessed 25 September 2023]. 1197 Di Nicola, A., Baratto, G., and Martini, E. 2017. Surf and Sound: The role of the Internet in People Smuggling and Human Trafficking. [accessed 13 November 2024]. 1198 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons. [accessed 25 September 2023]. 221
User communications¶
Ephemeral messaging¶
Posting content (images, videos, hashtags, emojis)1201 and livestreaming¶
Direct messaging and encrypted messaging¶
1199 Crest (Caluori, J, Mooney, B. and Kirk, E.), 2022. Running out of credit: Mobile phone tech and the birth of county lines. [accessed 25 September 2023]. Children and Young People’s Centre for Justice, 2023. Understanding Child Criminal Exploitation in Scotland: A Scoping Review. [accessed 1st November 2024]. 1200 Children’s Society response to the Protection of Children May 2024 Consultation. 1201 Where the evidence below makes reference to ‘advertising,’ this is typically used to refer to content which has the effect of marketing or promoting goods and services, rather than paid advertising, which is detailed in the Revenue Model section below. 1202 The Children’s Society, 2019. Counting lives: Responding to children who are criminally exploited. [accessed 13 November 2024]. 1203 Human Trafficking Front, 2023. Social media and child sex trafficking. [accessed 17 October 2024]. 1204 Alliance to Counter Crime Online, n.d. Human trafficking: How Social media Fuels Modern Day Slavery. [accessed 25 September 2023]. 1205 Although one of the hashtags involved was subsequently banned by the service provider, the potential for using coded language in hashtags means we consider the risk related to this feature remains. Source: BBC News, 2019. Technology and human trafficking. [accessed 17 October 2024].
1206 Council of Europe, 2022. Online and technology-facilitated trafficking in human beings: Full report. [accessed 17 October 2024]. 1207 Children and Young People’s Centre for Justice, 2023. Understanding Child Criminal Exploitation in Scotland: A Scoping Review. [accessed 1st November 2024]. 222
exploitation for communication between traffickers and buyers of sexual services carried out by the person that is being trafficked. After initially advertising an opportunity, direct and/or encrypted messaging is often used to continue communication between a perpetrator and their target as a way of facilitating exploitation, such as providing information about and discussing a supposed job opportunity. One report shares that traffickers have in some cases continued communicating with victims/survivors via encrypted communication after exploitation has ended, to try and intimidate and dissuade them from seeking justice.1208
Posting or sending location information¶
User networking¶
User groups and user connections¶
1208 Council of Europe, 2022. Online and technology-facilitated trafficking in human beings: Full report. [accessed 17 October 2024]. 1209 Children’s Society response to the Protection of Children May 2024 Consultation. 1210 The Children’s Society, 2019. Counting lives: Responding to children who are criminally exploited. [accessed 13 November 2024]. 1211 Polaris Project, 2018. A roadmap for systems and industries to prevent and disrupt human trafficking: Social media. [accessed 17 October 2024]. 1212 This report also notes communication between traffickers and victims within closed online groups, but it is unclear if these refer to groups with more than two people or not. Source: Council of Europe, 2022. Online and technology-facilitated trafficking in human beings: Full report. [accessed 17 October 2024].
1213 Robins-Early, N. (The Guardian). 2024. The anonymous, anything-goes forum at the heart of the Pelicot rape case, 12 October. [accessed 28 October 2024]. 1214 For example, government guidance about county lines notes that in cases of criminal exploitation victims sometimes have pre-existing connections with perpetrators. Source: Home Office, 2023. Criminal exploitation of children, young people and vulnerable adults: county lines. [accessed 17 October 2024]. 223
Transactions and offers¶
Posting goods or services for sale¶
Risk factors: Business model and commercial profile¶
Revenue models¶
1215 Rescue and Response, 2020. Year 2 Strategic Assessment. [accessed 1st November 2024].
1216 Modern Slavery & Human Rights, n.d. The role of adult service websites in addressing modern slavery. [accessed 25 September 2023]. 1217 United Nations Office on Drugs and Crime, 2020. Global report on trafficking in persons. [accessed 25 September 2023]. 1218 United Nations Office on Drugs and Crime, 2020.. 1219 Thorn, 2018. Survivor Insights: The role of technology in domestic minor sex trafficking, p38 [accessed 5 July 2023]. 224
1220 Children’s Society response to the Protection of Children May 2024 Consultation.; Mararike, S., 2021. Dealers are using Fortnite treats to groom children as drug mules, The Sunday Times, 21 March. [accessed 17 October 2024]. 225
Section 10 Unlawful immigration¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for unlawful immigration offences: how harm manifests online, and risk factors This chapter covers offences relating to unlawful immigration. People entering the UK without permission may do this for a range of reasons, which can include fleeing another country owing to fear of physical harm or forms of illegitimate persecution. This chapter talks about the risks of harm from unlawful immigration, which often result from actions taken by people smugglers. These can be widespread, including trauma, financial hardship, injury or even death for those undertaking potentially dangerous routes of entry. Evidence also shows that some people smugglers also commit human trafficking offences, meaning that some irregular migrants face risks of harm related to those offences. Service type risk factors: Social media services and messaging services are shown to be risk factors for unlawful immigration offences. Among other elements, they are used to target people and to advertise services. Often, perpetrators will identify someone on social media before transitioning to an encrypted private messaging service to further the facilitation of the offence. Functionalities and recommender systems risk factors: User profiles can be exploited by a perpetrator looking to build trust with their targets, by providing a sense of legitimacy to their enterprise. Closed user groups were found to be a risk factor, with smugglers using these groups to share information which could help facilitate the offences. Posting content is also used to promote illegal services. Posting content could also be used by a smuggler to build trust in the journey’s safety. Direct messaging and encrypted messaging can allow smugglers to communicate with people using their services and others facilitating unlawful migration.
Introduction¶
Relevant offences¶
1221 ‘People smuggling’ and ‘people trafficking’ are different concepts in law. Offences relating to ‘people smuggling’ will generally relate to the Immigration Act offences, whereas ‘people trafficking’ will generally be offences under the Modern Slavery Act, Crown Prosecution Service, Updated 6 July 2022. Source: Crown Prosecution Service, 2022. Modern Slavery, Human Trafficking and Smuggling. [accessed 25 September 2023].
1222 Section 24(A1), (B1), (C1) or (D1) of the Immigration Act 1971. 1223 Section 25 of the Immigration Act 1971. 1224 Generally referred to as ‘leave to enter or remain’. Source: Home Office, 2023. Immigration Rules. [accessed 25 September 2023]. 1225 See the Fraud and financial services chapter for information about other offences regarding the sale of counterfeit products or other fraud involving mis-selling. 227
How unlawful immigration offences manifest online¶
1226 There are many reasons for this, including the sometimes-clandestine nature of irregular migration. On the use of ‘irregular migration’, terminology around unlawful immigration is highly contentious. The UK government reports statistics based on ‘irregular migration’ rather than ‘unlawful immigration’, and we have adopted that wording in this chapter. 1227 “Irregular” is the term used by the UK government to refer to migration that occurs outside of the law. “Undocumented” migration is also used. 1228 Statistics on small boats include individuals who were detected on arrival to the UK or detected in the Channel and subsequently brought to the UK. It is also worth noting the latest figures show a 26% decrease in irregular arrivals compared to the previous year. Source: Home Office, 2024. Irregular migration to the UK, year ending June 2024. [accessed 5 September 2024]. 1229 Between January 2018 and the end of June 2024, 35,396 asylum applicants who had arrived on a small boat had been granted refugee status or leave; this was the most common outcome of applications during this period. Source: Home Office, 2024. Irregular migration to the UK, year ending June 2024. [accessed 5 September 2024].
1230 The Migration Observatory, 2024. UK policies to deter people from claiming asylum. [accessed 8 August 2024]. 1231 Refugee Council (Crawley, H.), 2010. Chance or choice? Understanding why asylum seekers come to the UK. [accessed 8 August 2024]. 1232 EU Monitor, 2023. Explanatory Memorandum. [accessed 12th August 2023]. 1233 National Crime Agency (NCA) (Arnold, P.), 2024. Written evidence to the Modern Slavery Act 2015 Committee. [accessed 14 November 2024]. 228
Risks of harm to individuals presented by unlawful immigration offences¶
1234 NCA, 2024. 1235 NCA, 2024. 12,000 takedowns as NCA leads blitz on people smugglers' social media accounts. 24 July 2024. [accessed 8 August 2024]. 1236 Home Office, 2023. New tech partnerships to stop the boats. [accessed 25 September 2023]. 1237 NCA, 2024. 1238 NCA, 2024. 1239 The report states that these services are “frequently used for various purposes such as advertising, recruitment, communication, coordination, guidance, money transfer or monitoring law enforcement activities” Europol, 2022. European Migrant Smuggling Centre - 6th Annual Report. [accessed 25 September 2023].
1240 NCA, 2024. Pair jailed for attempt to smuggle migrants out of the UK. [accessed 12 August 2024]. 1241 For example, see World Health Organization, 2023. Mental health of refugees and migrants: risk and protective factors and access to care. [accessed 12 August 2023]. 1242 Hymas, C, 2024. Migrant smugglers quadruple price of passage across the Channel, The Telegraph, 20 May [accessed 8 August 2024]. 229
to relevant authorities.1243 Financial costs can leave people vulnerable to exploitation (see below) and debt bondage.1244
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services and messaging services¶
1243 This report provides evidence of online posts that expose fraudulent smugglers. Diba, P., Papanicolaou, G. & Antonopoulos, G.A., 2019. The digital routes of human smuggling? Evidence from the UK. [accessed 25 September 2023]. 1244 NCA, 2024. 1245 Data shows that of the 263 who died or went missing travelling the English Channel to the UK during this time, 248 had died due to reasons including hazardous transport and drowning. Missing Migrants Project, 2024. Migration Within Europe, Latest Incidents. [accessed 25th October 2024]. 1246 Europol, 2023. Criminal Networks in Migrant Smuggling. [accessed 12th August 2024]. 1247 For more information about these kinds of illegal harm see the relevant register of Risks chapters (Human trafficking, Sexual exploitation of adults and Child sexual abuse and exploitation). 1248 Europol, 2023.
1249 Europol, 2024. 21 arrested in hit against migrant smuggling across the EU-Russian border. [accessed 10 October 2024]. 1250 Infomigrants is co-financed by the EU and in partnership with three major European media sources: France Médias Monde (French), Deutsche Welle (German) and ANSA (Italian). Source: Infomigrants (Alboz, D.), 2016. Social media networks, the best friend of smugglers. [accessed 25 September 2023]. 1251 NCA, 2024. 12,000 takedowns as NCA leads blitz on people smugglers' social media accounts. 24 July 2024. [accessed 8 August 2024]. 230
media services, particularly those with encryption, provide a “really good, dynamic, agile way for people to move migrants between them, and for groups to communicate”.1252
Services enabling users to build online communities¶
1252 Gentleman, A., 2020. Social media refuse to pull people-smuggling pages, MPs told, The Guardian, 3 September. [accessed 25 September 2023]. 1253 Europol, 2024. Tackling threats, addressing challenges: Europol’s response to migrant smuggling and trafficking in human beings in 2023 and onwards. [accessed 12th August 2024]. 1254Diba, P., Papanicolaou, G. and Antonopoulos, G.A., 2019. The digital routes of human smuggling? Evidence from the UK. [accessed 25 September 2023]. 1255 Europol, 2023. Criminal Networks in Migrant Smuggling. [accessed 12th August 2024]. 1256 NCA, 2024. 1257 Europol, 2024. Tackling threats, addressing challenges: Europol’s response to migrant smuggling and trafficking in human beings in 2023 and onwards. [accessed 12th August 2024]. 1258 Europol, n.d. Europol coordinates referral action targeting migrant smuggling from Belarus. [accessed 25 September 2023]. Whilst this particular illegal immigration trend may no longer be taking place, it is an example of how these online functionalities can be used to facilitate similar activities in other contexts.
1259 FATF, 2022. Money Laundering and Terrorist Financing Risks Arising from Migrant Smuggling. [accessed 10 October 2023]. 1260 Infomigrants (Alboz, D.), 2016. Social media networks, the best friend of smugglers. [accessed 25 September 2023]. 1261 Diba, P., Papanicolaou, G. and Antonopoulos, G.A., 2019. 231
Risk factors: User base¶
User base size¶
10.31
User base demographics¶
10.32
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles, fake user profiles, and anonymous user profiles¶
1262 NCA, 2024. 1263 NCA, 2024. 1264 NCA, 2024. 232
crossings are then contacted on social media to ask details about which routes to take and how to approach an illegal crossing.1265
User communications¶
Posting content (videos, emojis, hashtags)¶
Direct messaging and encrypted messaging¶
1265 Rest of World (Joles, B.), 2022. Inside the risky world of “Migrant TikTok”. [accessed 25 September 2023]. 1266 Burgess, S., 2021. Channel deaths: People smugglers touting openly on Facebook, Sky News, 25 November. [accessed 10 October 2024]. 1267 Where the evidence below makes reference to ‘advertising,’ this is typically used to refer to content which has the effect of marketing or promoting goods and services, rather than paid advertising, which is detailed in the Revenue Model section below. 1268 Diba, P., Papanicolaou, G. and Antonopoulos, G.A., 2019. The digital routes of human smuggling? Evidence from the UK [accessed 25 September 2023]. 1269 Sinoruka, F., 2022. Rise in TikTok Ads Among Albanians Selling Smuggling Operations to UK. Balkan Insight, 8 August. [accessed 25 September 2023].
1270 Rest of World (Joles, B.), 2022. Inside the risk world of “Migrant TikTok”. [accessed 25 September 2023]. 1271 Kansara, R., Fatima, S. and Dyer, J., 2023. Going undercover to reveal people smugglers' sales tactics, BBC News, 28 October. [accessed 10 October 2024]. 1272 NCA, 2024. 1273 NCA, 2024. 233
activities.1274 1275 Messaging services are also used for the advertisement of services that smugglers provide.1276
User networking¶
User groups¶
Transactions and offers¶
Posting goods or services for sale¶
1274 Europol, 2024. Tackling threats, addressing challenges: Europol’s response to migrant smuggling and trafficking in human beings in 2023 and onwards. [accessed 12th August 2024]. 1275 Gentleman, A, 2020. Social media refuse to pull people-smuggling pages, MPs told, The Guardian, 3 September. [accessed 25 September 2023]. 1276 Europol, n.d. Europol coordinates referral action targeting migrant smuggling from Belarus. [accessed 25 September 2023]. Whilst this particular illegal immigration trend may no longer be taking place, it is an example of how these online functionalities can be used to facilitate similar activities in other contexts. 1277 Infomigrants (Alboz, D.), 2016. Social media networks, the best friend of smugglers. [accessed 25 September 2023]. 1278Diba, P., Papanicolaou, G. & Antonopoulos, G.A., 2019. The digital routes of human smuggling? Evidence from the UK [accessed 25 September 2023].
1279Diba, P., Papanicolaou, G. & Antonopoulos, G.A., 2019. 1280 Tech Transparency Project, 2022. Facebook Marketplace, WhatsApp Storefronts, and TikTok Videos: How Coyotes Get Creative. [accessed 10 October 2024]. 1281 Europol, 2024. Tackling threats, addressing challenges: Europol’s response to migrant smuggling and trafficking in human beings in 2023 and onwards. [accessed 10 October 2024]. 234
Section 11 Fraud and financial services offences¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for fraud and financial offences: how harm manifests online, and risk factors This chapter covers offences linked to fraud, fraudulent activity and financial services, including the use of misleading statements intended to entice users to engage in relevant investment activity. Fraud is the most frequently experienced crime in the UK and the risks of harm to individuals from fraud and financial services offences are broad. The 2023 UK Government Fraud Strategy estimated the total economic and social cost of fraud to individuals between 2019 and 2020 to be £6.8bn. Financial loss to victims is not the only result of fraud; the harm can be multi-faceted and can have serious consequences on both mental and physical health. We have identified the following characteristics of online services that are relevant to risks of harm in relation to these offences. Service type risk factors: Our evidence points to fraud taking place on a wide range of services. This can include social media services, messaging services, marketplaces and listings services, and dating services. Victims of fraud can be targeted through one type of service and then potentially be moved to other types of service for further communications and transactions. Different types of fraud may be more common on different types of service; for example, investment or purchase fraud activities are more likely to occur on social media services. User base risk factors: Online services with a large user base can help fraudsters reach large numbers of potential victims at low cost with minimal effort. In addition, a large user base risks amplifying the initial reach of fraudulent content to an even bigger potential audience via a higher volume of content reactions, posts and re-posts. Anyone online, of any age, can be a potential victim of fraud; including being targeted for a specific type. Those who are likely to have the least financial resilience (for instance, those less able to withstand financial shocks) may fall victim more easily to some specific scams where the supposed gains are promised quickly. Scams include purchase scams offering cheap goods, and loan-fee scams which appear to peak at periods of financial difficulty. In contrast, investment 235
scams, for example, tend to target older age groups with greater disposable income, as well as low-capital individuals attracted by the promise of large returns. People who have experienced mental health challenges are also more likely to have been a victim of an online scam. Low levels of media literacy may also be a significant factor when assessing the risks of harm. Research suggests that users do not always have the critical skills to recognise fake propositions. Functionalities and recommender systems risk factors: Fraud and financial services priority offences can be enabled using a range of functionalities on user-to-user (U2U) services; these are common across most online services and therefore, in principle, almost any service can be attractive to fraudsters. Creating fake user profiles on a U2U service enables fraudsters to commit or facilitate fraud, allowing them to conceal their identity and impersonate legitimate entities such as banks, insurance providers or financial advisors to add legitimacy to false claims. Fraudsters will also make use of people who have many user connections to achieve their aims. While user groups are used by fraudsters to share knowledge to successfully carry out scams. Functionalities that allow communications between users can be abused by fraudsters. For example, the ability to communicate via direct messaging, group messaging or private messaging may help fraudsters create the appearance of a legitimate organisation when engaging with a user or may help to promote a relationship in a romance scam. Fraudsters can move conversations onto services with encrypted messaging to avoid moderation or intervention disrupting their activity, as well as evidence of illegality. The functionality of posting goods and services for sale can enable fraudsters to trick users into paying for goods and services that do not exist or are less valuable than described. Searching for user-generated content (UGC) can enable fraudsters or potential fraudsters to find posts offering to supply information, advice and articles (such as stolen bank details) which support the commission of fraud. These functionalities (posting goods and services for sale and searching for UGC) are included in the risk profiles for their role in propagating fraud (see also the ‘Proceeds of crime’ chapter). The functionality of hyperlinking enables fraudsters to redirect victims to webpages outside of the original service which can then facilitate scams such as purchase scams, advance-fee scams and impersonation scams if the victim shares their personal information or have malicious programmes downloaded onto their device. The information on user profiles can also be used by fraudsters to identify potential victims, such as high net-worth individuals or those who are looking to make 236
connections, for instance on online dating services. Fraudsters may also join user groups to identify potential targets. Users can also encounter fraud in the comments on posts, so commenting on content can also be considered a risk factor.
Introduction¶
Relevant offences¶
Priority offences for fraud¶
•
• Fraud by abuse of position:1283 It is an offence to commit fraud by way of a person dishonestly abusing their position
• assist in the commission of, fraud. In Scotland, this is covered by a separate but similar offence1285 •
Priority offences for financial services¶
How fraud and financial services offences manifest online¶
Fraud offences: Examples¶
1284 Section 7 of the Fraud Act 2006. 1285 Section 49(3) of Criminal Justice and Licensing (Scotland) Act 2010.
1286 Section 9 of the Fraud Act 2006. 1287 These offences are set out in the Financial Services and Markets Act (FMSA) 2000 and the Financial Services Act 2012. 1288 Section 23 of the FMSA 2000 Act. 1289 Section 24 of the FMSA 2000 Act. 1290 Section 25 of the FMSA 2000 Act. 1291 Section 89 misleading statements or Section 90 misleading impressions of the Financial Services Act 2012. 238
officials or police officers) and request a payment or information from an individual, potentially via phishing.1292
• Purchase scams, where a product purchased by a user is not provided, or where provided, is different from the product advertised on the U2U service. For example, sale of fake holidays or counterfeit goods described as genuine, and ‘ghost broking’1293 which involves the sale of fake insurance policies. Fraudsters may pose as known brands. • Investment scams, where fraudsters persuade users to invest in a financial product which does not exist and so a victims’ money is stolen. Fraudsters may present themselves as a trustworthy institution, advisor, or someone known to the victim; use pressurising tactics; or promise returns generally not available through mainstream products, for example, by offering cryptocurrency. Pension scams are considered a type of investment scam in this chapter.1294 • Romance scams, where fraudsters exploit the trust of the victim, who is under the impression that the perpetrator is genuinely interested in building a relationship or friendship. The fraudster typically asks for money or financial information. • Employment scams, in which fake job opportunities are promoted by fraudsters. The fraudsters may ask for payments for processes they say are needed for the victim to secure the role, or to gain personal information from victims. • Mule herders1295 seeking to recruit money mules1296 may also commit fraud by false representation via user-generated content to trick people into becoming a mule. For example, mule herders may create fake jobs that involve moving money between accounts, including asking the victim to use their own account to help move the money, or to hand over control of their account. Other tactics used to hook potential victims could include a romance scam, where the mule herder exploits the victim’s trust to ask them to transfer money or hand over their account details. Users may respond to opportunities to make money, shared via user-generated content, which involve becoming a money mule and earning a commission.
1292 “Phishing is when attackers attempt to trick users into doing 'the wrong thing', such as clicking a bad link that will download malware, or direct them to a dodgy website.” National Cyber Security Centre (NCSC), 2018. Phishing attacks: defending your organisation. [accessed 2 October 2023]. 1293 A ‘ghost broker’ is a term used to describe a fraudster who pretends to be a genuine insurance broker in order to sell fraudulent insurance. 1294 “The word ‘investment’ is used in connection with a wide range of schemes offering income, interest or profit in return for a financial investment. ‘Investment’ is often used loosely, and sometimes misleadingly, in order to disguise the true nature of a fraud, for example, pyramid schemes, chain letters or other types of scheme where a return depends on persuading others to join. The term ‘investment’ is commonly used in connection with the purchase of something - such as high value or rare goods, stocks and shares, property, in the expectation that what is purchased will increase in value, and even provide an exceptional return compared to other forms of investment….An investment seminar will hook individuals by offering a return which is more attractive than a conventional investment, and so the return on the outlay is always likely to be exaggerated or unrealistic. It follows that the essential message which applies to other scams applies equally to investments. If it looks too good to be true, it probably is.” Source: Home Office, 2023. Counting rules for recorded crime. [accessed 1 August 2023].
1295 Mule herders are people who recruit money mules. A money mule is someone who lets criminals use their bank account to move money. See also chapter 12: Proceeds of Crime offences for more information on money mules. 1296 A money mule is someone who receives money from a third party in their bank account and transfers it somewhere else, or who withdraws it as cash and gives it to someone else, obtaining a commission for it or payments in kind. 239
• User-generated content can also be used to supply stolen identity or banking credentials,1297 such as stolen personal information, credit card details, or fraud ‘how-to’ guides and fake passports.
1297 Research carried out by Which? reflects that user profiles, ‘pages’ and user groups on social media services are being used by criminals to provide stolen credentials, enabling the perpetration of further frauds through identity theft. “They advertised a mixture of stolen identities, credit card details, compromised Netflix and Uber Eats accounts, as well as fraud 'how to' guides and fake passports made to order.” Source: Which? (Lipson, F.), 2020. Your life for sale: stolen bank details and fake passports advertised on social media. [accessed 1 August 2023]. 1298 BBC One Rip Off Britain, 2023. What is a ‘pig-butchering’ scam – and why is it on the rise?. [accessed 18 October 2024]. 1299 National Crime Agency (NCA), 2024. Fraud. [accessed 9 September 2024]. 1300 Office of National Statistics (ONS), 2023. Crime in England and Wales: year ending June 2023. [accessed 31 July 2024]. 1301 UK Parliament The Parliamentary Office of Science and Technology (POST) (Low, N. and Lally, C.), 2024. Social and psychological implications of fraud. [accessed 31 July 2024].
1302 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1303 Section 2 of the Fraud Act 2006 [accessed 2 August 2023]. 1304 The research asked participants about the different types of online fraud or scams they experienced if they self- reported to have personally been drawn into engaging with fraud or scams that began online. Participants may not have lost money but they may have, for example, clicked on an advertisement, followed specific instructions, or replied to a message. Source: Ofcom, 2023, Online Scams & Fraud Research. [accessed 2 August 2023]. 240
‘scams, fraud and phishing’ is an exception, with relatively high levels of concern (79%) and experience (33%).1305
1305 Ofcom, 2024. Online Experiences Tracker - Wave 6. [accessed 18 October 2024]. 1306 Action Fraud, n.d. Fraud Crime Trends. [accessed 5 September 2023]. 1307 Note that the overall cost to the UK is considered to be significantly greater than just the value of the money reported lost directly to frauds. Source: Home Office, 2023. Fraud Strategy. [accessed 10 October 2024]. 1308 In comparison, UK Finance estimated to total money lost to fraud was £1.17 billion in 2023. Source: UK Finance, 2024. Annual Fraud Report 2024. [accessed 10 October 2024].
1309 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1310 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1311 The Cyber Resilience Centre (Duckett, S.), 2023. What is Brand Impersonation? How can I Protect my Business?. [accessed 31 July 2024]. 1312 DNS Research Federation (Taylor, E. and Taylor, L.), 2023. Anatomy of a scam. What makes consumers click?. [accessed 31 July 2024]. 241
Financial services offences: Examples¶
Risks of harm presented by fraud and financial services offences¶
1313 DRCF, 2023. 2023/24 Workplan. [accessed 1 August 2023]. 1314 An alert warns people of unauthorised firms and individuals who are conducting unregulated activity. A list of these firms and individuals can be viewed on the FCA’s warning list. Financial Conduct Authority (FCA), 2023. FCA Warning List of unauthorised firms. [accessed 1 August 2023].
1315 FCA, 2022. Financial promotions data 2022. [accessed 1 August 2023]. 1316 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1317 Action Fraud, 2021. Fraud Crime Trends. [accessed 1 August 2023]. 1318 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 242
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services¶
1319 Age UK, 2023. Age UK- written evidence (DCL0049). [accessed 1 August 2023]. 1320 “This can include bank account closure, limited access to loans or credit cards, difficulty obtaining a phone contract, and/or a prison sentence of up to 14 years.” House of Lords, 2022. Fighting Fraud: Breaking the Chain. [accessed 1 August 2023]. 1321 VICE (via YouTube), 2022. The Rise of Money Launderers on Snapchat and Instagram, 25 October. [accessed 2 August 2023]. 1322 Ofcom, 2023. Online Scams & Fraud Research: Data Tables. [accessed 2 August 2023]. 1323 Action Fraud, 2021. New figures reveal victims lost over £63m to investment fraud scams on social media. [accessed 1 August 2023].
1324 National Fraud Intelligence Bureau (NFIB), 2022. Annual Assessment. 1325 “A romantic partner showers you with attention, money, and gifts in order to gain control in a relationship.” Reader’s Digest (Nelson, B.), 2022. Is Love Bombing the Newest Scam to Avoid?. [accessed 1 August 2023]. 1326 The Guardian (Clark, J. and Wood, Z.), 2023. Victims speak out over ‘tsunami’ of fraud on Instagram, Facebook and Whatsapp. [accessed 1 August 2023]. 1327 NFIB, 2022. Annual Assessment. 243
Romance scammers seek to make direct contact with their victims and may look to move conversations to a private messaging service with encryption. They may ‘love bomb’ victims with frequent messaging and wait for many months before executing the scam.
Messaging services¶
1328 FCA, 2021. FCA issues warning over ‘clone firm’ investment scams. [accessed 2 October 2023]. 1329 Investment fraud can relate to financial services offences. 1330 City of London Police response to 2022 Call for Evidence: First phase of online safety regulation. 1331 The process of improving your site to increase its visibility when people search for products or services related to your business in Google, Bing, and other search engines. Search Engine Land, n.d.. What Is SEO – Search Engine Optimization?. [accessed 1 August 2023]. 1332 City of London Police response to 2022 Call for Evidence: First phase of online safety regulation. “Social media services are used in multiple fraud offences, there were 138,375 Action Fraud reports that featured a social media or communication service in 2021/22. A total of £555m of financial losses related to these reports. In 2020/21 the figure was 75,769 - this increase of 83% indicates the accelerating trend of offenders using social media services to target UK victims for fraud.”; Publication of consultation on standalone code on fraudulent advertising to follow. 1333 Identity theft relates to fraud by false representation. 1334 Advocating Against Romance Scammers (Denny, B. and Waters, K.), 2021. Community Substandards: Capturing the Empty Promises of Big Tech’s Safety against Online Romance Scams. [accessed 1 August 2023]. 1335 Which?, 2020. Your life for sale: stolen bank details and fake passports advertised on social media. [accessed 13 September 2023].
1336 NFIB, 2022. Annual Assessment. 1337 An NFIB profile into romance fraud in 2019 found that of the reports analysed, in 47% the conversation had been moved to a secondary encrypted service after the initial contract on a public primary service. City of London Police, 2019. 2019 Romance Fraud Profile, document owned by NFIB. 1338 This may be relevant to offences for fraud and also for financial services. 244
that, according to survey respondents, just under half (46%) of fraudsters used a targeted message to make initial contact with their victim, and typically this is done through direct messaging (41%).1339 The NFIB also found that fraudsters may also use group messaging to store information and to communicate with victims in a group.1340
Marketplaces and listings services¶
Dating services¶
Risk factors: User base¶
User base size¶
Services with a large user base¶
Services with a small user base¶
1339 Ofcom, 2023. Online Scams & Fraud Research: Data Tables. [accessed 2 August 2023]. 1340 NFIB, 2022. Annual Assessment. 1341 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1342 UK Finance, 2022. Annual Fraud Report. [accessed 1 August 2023].
1343 Kaspersky, n.d. Online dating scams and how to avoid them. [accessed 1 August 2023]. Note that the source is a company specialising in cybersecurity. 1344 Consumers International, 2019. Social Media Scams: Understanding the Consumer Experience to Create a Safer Digital World. [accessed 1 August 2023]. 1345 Federal Trade Commission (Fletcher, E.), 2022. Social media a gold mine for scammers in 2021. [accessed 1 August 2023]. 245
look for more niche services in the UK, identifying services that are widely used by communities or professions which they can target.1346
User base demographics¶
Age¶
Financial resilience¶
1346 NFIB, 2022. Annual Assessment. 1347 Which?, 2023. The Psychology of Scams. [accessed 30 August 2024]. 1348 FCA, 2016. Over 55s at heightened risk of fraud. [accessed 1 August 2023]. 1349 City of London Police response to 2022 Call for Evidence: First phase of online safety regulation. 246
turning to social media to target younger generations”.1350 Younger individuals (those aged 18 to 54) are the most likely to have low financial resilience (29% compared with the UK average of 24%).1351 This may also mean they are susceptible to the types of fraud that appeal most strongly to those who are financially disadvantaged; where the supposed gains are promised quickly; purchase scams offering cheap goods; and loan-fee fraud scams which appear to peak at periods of financial difficulty.
Mental health¶
Media literacy¶
1350 Phoenix, 2021. Three in ten 18-34 year olds fell victim to scams in the last year, with scammers turning to social media to target younger generations. [accessed 20 September 2023]. 1351 “Adults are described as having low financial resilience if they have little capacity to withstand financial shocks, because, for example, they do not think they would be able to withstand losing their main source of household income for even a week or are finding it to be a heavy burden keeping up with their domestic bills or credit commitments, or because they have already missed paying these bills in 3 or more of the last 6 months. So, our definition includes both those adults who are already in financial difficulty (because they are missing bills – so this is an objective measure) and those who could quickly find themselves in difficulty if they suffer a financial shock (by more subjective measures)”. FCA, 2022. Financial Lives 2022 survey: insights on vulnerability and financial resilience relevant to the rising cost of living. [accessed 1 August 2023]. 1352 NFIB, 2022. Annual Assessment. 1353 This is compared with the UK average (Ethnicity: Black & Black British 44%, Mixed/Multiple 39%, UK average 24%). FCA, 2022. Financial Lives 2022 survey: 3. Low financial resilience. [accessed 1 August 2023].
1354 The Money and Mental Health Policy Institute (Holkar, M. and Lees, C.), 2020. Caught in the Web. [accessed 1 August 2023]. 1355 Ofcom, 2022. Adults’ Media Use and Attitudes report. [accessed 2 August 2023]. Note: The research relates to paid advertising content (out of scope of this assessment), where four in ten claimed they would not be able to tell if an advert was fake or not. 247
of respondents to Ofcom research stated that they always, mostly, or sometimes checked for verification symbols when deciding to follow or interact with an account.1356
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles¶
1356 Ofcom, 2023. Open Data. [accessed 5 September 2023]. 1357 NFIB, 2022. Annual Assessment. 1358 Which? (Lipson, F.), 2020. Your life for sale: stolen bank details and fake passports advertised on social media. [accessed 13 September 2023]. 1359 Youngs, I., 2024. Facebook 'did nothing about Taylor Swift ticket hack scam', BBC News, 8 May. [accessed 18 October 2024]. 1360 Morris, M. R., Counts, S., Roseway, A., Hoff, A. and Schwarz, J., 2012. Tweeting is believing? Understanding microblog credibility perceptions, Computer Supported Cooperative Work: Proceedings of the ACM 2012 Conference. [accessed 31 July 2024].
1361 Agarwal, S., 2022. The black market for stolen verified accounts from Twitter and Instagram, The Verge, 18 October. [accessed 31 July 2024]. 1362 Francisco, E., 2020. July 15 Twitter hack: A list of every hacked verified account, Inverse, 20 February. [accessed 31 July 2024]. 1363 Binder, M., 2023. Scammers hack verified Facebook pages to impersonate Meta and Google, Mashable, 5 May. [accessed 1 August 2024]. 248
Fake user profiles¶
1364 King, S., 2021. Martin Lewis, Sir Richard Branson, Deborah Meaden and other public figures issue plea to the PM to put scam ads in the Online Safety Bill, MoneySavingExpert News, 16 November. [accessed 1 August 2023]. 1365 Cifas response to 2022 Call for Evidence: First phase of online safety regulation. 1366 Clean Up the Internet is an independent, UK-based organisation concerned about the degradation in online discourse and its implications for society and democracy. 1367 Babbs, D, 2023. New report on fraud, fake accounts, and the User Verification Duty, Clean Up the Internet, 25 April. [accessed 1 August 2023]. 1368 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1369 LoveSaid response to November 2023 Illegal Harms Consultation, LoveSaid (Ofcom.org.uk); College of Policing (Cumming, L.), 2021. Romance fraud: Five things you need to know. [accessed 31 July 2024].
1370 City of London Police response to 2022 Call for Evidence: First phase of online safety regulation. 1371 Home Office, 2023. Fraud Strategy: Stopping Scams and Protecting the Public. [accessed 8 August 2024]. 1372 Burgess, M., 2022. Elon Musk's Twitter Is a Scammer’s Paradise, Wired, 10 November. [accessed 1 August 2024]. 1373 Clean up the Internet (Kinsella, S.), 2022. What do Elon Musk’s “Blue Tick” experiments mean for the UK’s Online Safety Bill?. [accessed 1 August 2024]. 249
User networking¶
User connections¶
User groups¶
1374 NFIB, 2022. Annual Assessment; A study into fake animal rescues noted that a large following is potentially lucrative for fraudsters who seek to monetise on ‘fake’ rescue content or build legitimacy for donations or other rewards. Social Media Animal Cruelty Coalition, 2024. Spot the Scam: Unmasking Fake Animal Rescues. [accessed 29 October 2024]. 1375 Influencers gather large followings of enthusiastic, engaged people who pay close attention to their views. 1376 NFIB, 2021. The Role of Social Media in Investment Fraud. Examples of ‘edited advertisements’ include but not limited to content where images of notable entities or ‘influencers’ were extracted from existing media and added to fraudulent advertisements made by fraudsters: Sproson, K. and Slater, B., 2024. Martin Lewis scam adverts, MoneySavingExpert, 7 October. [accessed 18 October 2024]; King, S., 2021. Martin Lewis, Sir Richard Branson, Deborah Meaden and other public figures issue plea to the PM to put scam ads in the Online Safety Bill, MoneySavingExpert News, 16 November. [accessed 1 August 2023]. 1377 NFIB, 2022. Annual Assessment. 1378 FCA, 2024. FG24/1: Finalised guidance on financial promotions on social media. [accessed 10 September 2024]. 1379 Advocating Against Romance Scammers (Denny, B. and Waters, K.), 2021. Community Substandards: Capturing the Empty Promises of Big Tech’s Safety against Online Romance Scams. [accessed 1 August 2023].
1380 For example, fraudsters used social media groups to identify targets trying to buy Taylor Swift tickets, who they could then try and scam. BBC News (Smith, E. and Horsburgh, L.), 2024. Taylor Swift ticket scammers 'feed off fans' desperation'. [accessed 24 September 2024]. 1381 Which? (Lipson, F.), 2020. Your life for sale: stolen bank details and fake passports advertised on social media. [accessed 13 September 2023]. 250
encourage others to engage in personal chats outside the platform where they sell financial advice or financial products”.1382
User communications¶
Livestreaming¶
Direct messaging¶
1382 FCA, 2024. FG24/1: Finalised guidance on financial promotions on social media. [accessed 10 September 2024]. 1383 UK Finance response to November 2023 Illegal Harms Consultation. 1384 BBC Radio 4 You and Yours (Vahl, S. and Smith, E.), 2024. Facebook Ticket Scam, Business Nimbys and Smart Meter Update. [accessed 24 October 2024]. 1385 'Financially Motivated Sexual Extortion' is a form of blackmail that involves threatening to publish sexual information, photos or videos about someone.
1386 Please note that where the victim of financially motivated sexual extortion is a child then the activity and content would also likely constitute one or more grooming or CSAM offences. For details, see ‘Financially motivated sexual extortion’ in the CSEA chapter. 1387 National Crime Agency, n.d. Kidnap and Extortion. [accessed 1 August 2023]. 1388 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1389 Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 251
Group messaging¶
Private messaging and encrypted messaging¶
Commenting on content¶
1390 Okpattah, K., 2021. Social media fraud: The influencers promoting criminal scams, BBC News, 16 August. [accessed 13 September 2023].
1391 NFIB, 2022. Annual Assessment. 1392 NFIB, 2022. Annual Assessment. 1393 City of London Police response to 2022 Call for Evidence: First phase of online safety regulation. 1394 City of London Police response to 2022 Call for Evidence: First phase of online safety regulation. 1395 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024]. 252
percent (just under 1 in 5) of those who had encountered scams or frauds said they were suspicious because of comments from other users voicing concerns.1396
Hyperlinking¶
Transactions and offers¶
Posting goods or services for sale¶
1396 Ofcom, 2023. Online Scams & Fraud Research 2022. [accessed 2 August 2023]. 1397 Cifas and Forensic Pathways, 2018. Wolves of the Internet. [Accessed: 13 September 2023].
1398 UK Finance response to November 2023 Illegal Harms Consultation. UK Finance (ofcom.org.uk). 1399 McAfee (Dhaliwal, J), 2023. What Are the Risks of Clicking on Malicious Links. [accessed 10 October 2024] 1400 UK Finance response to November 2023 Illegal Harms Consultation. UK Finance (ofcom.org.uk). 1401 Which? response to November 2023 Illegal Harms Consultation. Which? (pfcom.org.uk). 1402 UK Finance, 2022. Annual Fraud Report. [accessed 1 August 2023]. 1403 UK Finance, 2022. 253
computers and smartphones, or advertising for sale fake holiday rentals, concert tickets1404, or exam papers. 1405
Content exploring¶
User-generated content searching¶
1404 BBC News (Smith, E. and Horsburgh, L.), 2024. Taylor Swift ticket scammers 'feed off fans' desperation'. [accessed 24 September 2024]. 1405 Johnson, K., 2024. GCSE pupils targeted by 'manipulative' exam scams, BBC News, 7 May. [accessed 24 October 2024]. 1406 PDSA, 2023. PDSA Animal Wellbeing (PAW) Report. [accessed 2 August 2024]; Lloyds Banking Group, 2023. Fraudsters go unleashed online as pet scams rise. [access 2 August 2024]. 1407 Cats Protection, 2023. CATS Report. [accessed 2 August 2024]. 1408 Dogs Trust, n.d. The Puppy Smuggling Scandal. [accessed 2 August 2024]; Sky News (Jones, T.), 2022. Dogs Trust warns people not to risk buying smuggled puppies this Christmas. [accessed 2 August 2024]. 1409 Action Fraud, 2021. Ruff time for animal lovers as scale of pandemic pet fraud unleashed. [accessed 2 August 2024]; BBC News (O'Donoghue, D. and Hesketh, S.), 2024. Missing pets: 'Heartless' scammers targeting desperate owners. [accessed 2 August 2024]. 1410 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024]. 1411 Ofcom, 2024. Online Experiences Tracker – Wave 6. [accessed 22 November 2024].
1412 Ofcom, 2023. Prevalence of Potentially Prohibited Items on Search Services. [accessed 21 September 2023]. 1413 Okpattah, K., 2021. Social media fraud: The influencers promoting criminal scams, BBC News, 16 August. [accessed 23 August 2023]. 1414 Ofcom, 2023. Prevalence of Potentially Prohibited Items on Search Services. [accessed 21 September 2023]. 1415 Okpattah, K., 2021. 1416 SEON, n.d. What Are Fullz. [accessed 4 September 2023]; Fraud.net, n.d.. What is Fullz?. [accessed 4 September]. 254
accidentally by internet users, this type of content is often very discoverable by criminals and likely to be prevalent on the open web and dark web – often on online forums.1417 Once criminals have acquired access to a package of stolen financial credentials and related personal information, this access will then typically be used to undertake a wide range of secondary fraud activities. These include card-related fraud (for example, the fraudulent purchase of goods, services, or subscriptions, making payments to ‘money mule’ accounts to launder the proceeds of crime), or impersonation or identity fraud (stealing someone’s identity to take over or set up new bank accounts, email accounts or social media profiles to support fraudulent loan applications.).1418
Risk factors: Business models and commercial profiles¶
Revenue models¶
1417 Bodker, A., Connolly, P., Sing, O., Hutchins, B., Townsley, M. and Drew J., 2022. Card-not-present fraud: using crime scripts to inform crime prevention initiatives, Security Journal, 36 (693-711). [accessed 23 August 2023]. 1418 SEON, n.d.; Data Dome, 2023. What are fullz? How do fullz work?. [accessed 4 September 2023].
1419 Cifas and Forensic Pathways, 2018. Wolves of the Internet. [accessed: 28 September 2023]. 1420 “Counterfeit goods were described as fake designer brand clothes, accessories, perfumes, pirated copies of DVDs and computer games, often found at auctions and web marketplaces, where you can’t check if the products are genuine until the item has been delivered”. Ofcom, 2023. Online Scams & Fraud Research. [accessed 2 August 2023]. 1421 We note that content that users pay to promote is within the scope of this risk assessment and the wider regime. There are separate duties for ‘fraudulent advertising’ that apply to non-user-generated content. 255
Section 12 Proceeds of Crime¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for proceeds of crime priority offences: how harm manifests online, and risk factors ‘Proceeds of crime’ is the term used for money or assets gained by criminals during their criminal activity and money laundering. Examples of activities which involve the proceeds of crime online include people being recruited as money mules to transfer illegally obtained money between bank accounts, discussion between criminals to arrange money laundering, and stolen personal information (via other criminal activity) offered for sale which can be used to commit or facilitate other types of fraud.1422 Further information on fraudulent activity can be found in the chapter ‘Fraud and financial services offences’. ‘Proceeds of crime offences’ is taken to mean offences relating to the concealment, arrangement of, acquisition, possession and use of criminal property in the Proceeds of Crime Act 2002. The risks of harm to individuals that could arise from proceeds of crime offences include losing livelihoods, losing access to financial services and consequential effects on mental health. Service type risk factors: Our evidence shows that proceeds of crime offences committed or facilitated online most commonly rely on social media services, messaging services and gaming services that have a broad userbase and wide reach. User base risk factors: Online services with large user bases are particularly attractive to fraudsters (including money mule recruiters) as they make it easy for them to reach large numbers of people at low cost, with minimal effort. In addition, a large user base can make it more likely that the initial reach of fraudulent content will be shared with a larger audience through likes, shares and re-shares. Services which make use of large, open groups of users will also be attractive to mule recruiters as they tend to add legitimacy to criminals. Services with a large user base are more likely to provide such groups.
1422 A money mule, or simply ‘mule’, ‘is someone who receives money from a third party in their bank account and transfers it somewhere else, or who withdraws it as cash and gives it to someone else, obtaining a commission for it or payments in kind.’ These individuals are targeted by ‘money mule recruiters’, sometimes referred to as ‘mule herders’, who recruit money mules, often using social media or online gaming services. House of Lords, 2022. Fighting Fraud: Breaking the Chain. [accessed 25 September 2023]. 256
Our evidence also shows that individuals from lower-income households and from certain minority groups may be more at risk of harm from this offence. Young adults were also seen to be more at risk as they sometimes have lower financial resilience and are more likely to have “clean” accounts. This could lead them to fall victims to potential money mule recruiters with the promise of money. Functionalities and recommender systems risk factors: The ability to create fake user profiles can make it harder to trace money mule recruiters and individuals posting fake job opportunities. Recruiters of money mules will use user-to-user (U2U) services to contact potential victims easily and directly. Direct messaging can be used by recruiters to directly contact potential money mules, often using specific phrases to attract people. The functionality of user-generated content (UGC) searching can also enable victims to initiate contact. Potential victims can respond to a post offering the chance to make money after searching for relevant content or seeing a misleading job opportunity. Criminals intending to commit proceeds of crime offences are likely to prefer encrypted messaging to communicate between themselves. The ability to post content and comment on posts can also enable perpetrators to find and contact at-risk individuals. User profiles, and the information displayed on them, can be used by perpetrators to gather information surrounding a potential victim.
Introduction¶
Relevant Offences¶
a) Concealing etc criminal property1423 b) Arrangements related to criminal property1424 c) Acquisition, use and possession of criminal property1425
How proceeds of crime offences manifest online¶
1423 Section 327 of Proceeds of Crime Act 2002. 1424 Section 328 of Proceeds of Crime Act 2002. 1425 Section 329 of Proceeds of Crime Act 2002. 1426 Evidence provided by City of London Police to the House of Lords ‘Fraud Act 2006 and Digital Fraud Committee’. Source: House of Lords, 2022. Fighting Fraud: Breaking the Chain. [accessed 22 September 2023]. 1427 Interpol, n.d. Money mules – what are the risks? [accessed 22 September 2023]. 258
money mule recruiter exploits the victim’s trust to ask them to transfer money or share their account details.1428
• In other circumstances, mules may be aware of or partially complicit in potentially engaging in illegal activity – actively responding to opportunities to make money. These money mules may make transfers or agree to surrender control of their accounts in return for earning a commission. • Alternatively, potential perpetrators can communicate with each other through UGC. This may include criminals coordinating or arranging money laundering via private messaging or using UGC posts to promote the sale of stolen financial or personal credentials1429 which can be used to launder funds.
1428 Triodos Bank, n.d. What is money muling? [accessed 22 September 2024]; UK Finance response to November 2023 Illegal Harms Consultation, Appendix 1. 1429 Note that the sale of, or offering for sale, stolen personal details itself may constitute the offence of making or supplying articles for use in frauds. See the chapter ‘Fraud and financial services offences’ for further information. 1430 Sanction Scanner, n.d. The Change of Money Laundering in The Digital Age. [accessed 22 September 2023].
1431 FBI, n.d. Money Mules. [accessed 22 September 2023]. 1432 Cifas, 2023. Fraudscape 2023. [accessed 22 September 2023]. 1433 Which?, 2020. Your life for sale: stolen bank details and fake passports advertised on social media [accessed 22 September 2023]. 1434 Crown Prosecution Service, 2019. Cybercrime - prosecution guidance. [accessed 22 September 2023]. 1435 Cifas, 2023. Identity fraud cases reach all-time high as cost-of-living crisis bites. [accessed 22 September 2023]. 259
Risks of harm to individuals presented by online proceeds of crime offences¶
1436 West Yorkshire Police, Money Mules (also known as Squaring). [accessed 5 August 2024]. 1437 FBI Omaha, Money Mule Scheme Targets Teenagers and Young Adults. [accessed 5 August 2024]. 1438 Children’s Society response to May 2024 Protection of Children Consultation, p. 11. 1439 This can include bank account closure, limited access to loans or credit cards, difficulty obtaining a phone contract, and/or a prison sentence of up to 14 years. Source: House of Lords, 2022. Fighting Fraud: Breaking the chain. [accessed 22 September 2023]. 1440 VICE (via YouTube), 2022. The rise of money launderers on Snapchat and Instagram: Crimewave, VICE, 25 October. [accessed 22 September 2023]. 1441 UK Finance, Cifas, n.d. Criminals may ask you to receive money into your bank account and transfer it into another account, keeping some of the cash for yourself. If you let this happen, you’re a money mule. You’re involved in money laundering, which is a crime. [accessed 22 September 2023]. 1442 Cifas, 2021. Money mule recruiters use fake online job adverts to target ‘Generation Covid’. [accessed 22 September 2023].
1443 Cifas, 2023. Fraudscape 2023. [accessed 22 September 2023]. 1444 The latest research from Cifas reported 17,157 cases of suspected money muling activity involving 21-30-year-olds in 2020, 5% up on the previous year. This age group accounted for 42% of money mule activity in 2020, up from 38% three years ago. Source: Cifas, 2021. Money mule recruiters use fake online job adverts to target ‘Generation Covid’. [accessed 22 September 2023]. 260
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services¶
Messaging services¶
1445 Lloyds Bank, 2024. Stubborn as a mule-hunter: Lloyds Bank cracks down on money mules. [accessed 18 October 2024]. 1446 VICE (via YouTube), 2022. The rise of money launderers on Snapchat and Instagram: Crimewave, VICE, 25 October. [accessed 22 September 2023]. 1447 Cifas, 2021. Money mule recruiters use fake online job adverts to target ‘Generation Covid’. [accessed 22 September 2023]. 1448 Cifas, 2021. 1449 Cifas, 2023. Fraudscape 2023. [accessed 22 September 2023].
1450 Cifas, 2021. 1451 Keyworth, M., 2018. I was a teenage ‘money mule’, BBC News, 26 April. [accessed 22 September 2023]. See Risk factors: functionalities and recommender systems section for more information. 1452 Barclays, n.d. Money Mules: Don’t be tricked into committing a crime. [accessed 22 September 2023]. 1453 Cifas, 2021. 1454 See risks of harm to individuals presented by online proceeds of crime offences for more information. 261
Gaming services¶
Risk factors: User base¶
User base size¶
User base demographics¶
1455 House of Lords, 2022. Fighting Fraud: Breaking the Chain. [accessed 25 September 2023]. 1456 Children’s Society Response to Protection of Children Consultation, p. 11. 1457 FBI Omaha, Money Mule Scheme Targets Teenagers and Young Adults, [accessed 5 August 2024]. 1458 National Crime Agency, 2021. National Strategic Assessment of Serious and Organised Crime. [accessed 22 September 2023]. 1459 Consumers International, 2019. Social Media Scams: Understanding the Consumer Experience to Create a Safer Digital World. [accessed 22 September 2023]. 1460 Federal Trade Commission (Fletcher, E.), 2022. Social Media a gold mine for scammers in 2021. [accessed 22 September 2023].
1461 Cifas, 2023. Fraudscape 2023. [accessed 22 September 2023]. 1462 The latest research from Cifas reported 17,157 cases of suspected money muling activity involving 21-30-year-olds in 2020, 5% up on the previous year. This age group accounted for 42% of money mule activity in 2020, up from 38% three years ago. Source: Cifas, 2021. Money mule recruiters use fake online job adverts to target ’Generation Covid‘. [accessed 22 September 2023]. 262
low financial resilience.1463 1464 Lloyds Bank also found that “almost one in 10 (9%) of those aged between 18 and 24 years old said they would agree to move money through their bank account in return for a fee or a percentage of the funds”; 1465 this may suggest that young people are less aware of the consequences of engaging in money muling and so are more likely to be herded, or are struggling to find legal alternatives during a difficult economic situation.
Risk factors: Functionalities and recommender systems¶
User identification¶
Fake user profiles¶
1463 Lloyds Bank, 2022. Money mules are getting older - with serious penalties for those caught moving scam cash. [accessed 22 September 2023]. 1464 Financial Conduct Authority, 2022. Financial lives 2022 survey: insights on vulnerability and financial resilience relevant to the cost of living. [accessed 22 September 2023]. 1465 Lloyds Bank, 2022. 1466 Most mules are recruited between the ages of 17 and 24. Source: National Crime Agency, n.d. Young People. [accessed 22 September 2023]. 1467 Hickey, S., 2023. Older people hired as ‘money mules’ by gangs as cost of living crisis bites, The Guardian, 12 June. [accessed 22 September 2023].
1468 Lloyds Bank, 2022. Money mules are getting older - with serious penalties for those caught moving scam cash. [accessed 22 September 2023]. 1469 Financial Conduct Authority, 2022. Financial lives 2022 survey: insights on vulnerability and financial resilience relevant to the cost of living. [accessed 22 September 2023]. 1470 VICE (via YouTube), 2022. The rise of money launderers on Snapchat and Instagram: Crimewave, VICE, 25 October. [accessed 22 September 2023]. 263
accessible to individuals choosing to make contact and partake in fraudulent activity under the false promise of ‘fast money’.1471
User profiles¶
User Groups¶
User communication¶
Direct messaging, commenting on content, and re-posting or forwarding¶
Transactions and offers¶
Post goods or services for sale¶
1471 Bekkers, L.M.J. and Leukfeldt, E.R., 2022. Recruiting money mules on Instagram: a qualitative examination of the online involvement mechanisms of cybercrime, Deviant Behaviour, 44(4). [accessed 22 September 2023].
1472 Cifas, 2021. Money mule recruiters use fake online job adverts to target ’Generation Covid‘. [accessed 22 September 2023]. 1473 Barclays, n.d. Money Mules: Don’t be tricked into committing a crime. [accessed 24 August 2023]. 1474 UK Finance Illegal Harms Consultation Response page 4. 1475 Keyworth, M., 2018. “I was a teenage ‘money mule’”. BBC News, 26 April. [accessed 22 September 2023]. 1476 Crown Prosecution Service, 2019. Cybercrime - prosecution guidance. [accessed 22 September 2023].f 264
Finance says that this tactic is particularly targeted towards young people whose job prospects have been damaged by the pandemic.1477
Content exploring¶
User-generated content (UGC) searching¶
Risk factors: Business models and commercial profile¶
1477 UK Finance, n.d. Money Mule Recruiters Use Fake Online Job Adverts to Target ”Generation Covid?. [accessed 22 September 2023]. 1478 Action Fraud, n.d. Money Muling. [accessed 22 September 2023]. 265
Section 13 Drugs and psychoactive substances¶
Summary analysis for drugs and psychoactive substances offences: how harm manifests online and risk factors This chapter summarises the risks of harm from the supply, or offer to supply, drugs and psychoactive substances. The harm to individuals resulting from the supply and use of drugs and psychoactive substances is substantial, and online services play an important role in facilitating the supply and distribution of illegal drugs in the UK. In 2022 there were 4,907 deaths related to drug poisoning registered in England and Wales, the highest number since records began in 1993, and 1.0% higher than in 2021. Under-18-year-olds may be particularly at risk of the harms brought about by illicit substances, and young people are most likely to be exposed to the related risks online. Surveys show that a wide range of services are being used to access drugs, with social media services and some video-sharing services being the most used. Law enforcement reported in 2024 that 20 to 24% of teenagers report having seen drugs for sale on social media. The impact of the trade and use of illicit substances is felt keenly on society, not just through associated criminality but also the burden it places on health and other public services. Service type risk factors: From our research, there is significant evidence supporting the role of social media and video sharing services in facilitating or committing drugs and other psychoactive substance offences. Suppliers use these services to advertise the sale of these illicit substances and then use messaging services to negotiate transactions. Our research identifies three additional service types that can also be linked to the supply of drugs and other psychoactive substances; however, the known use of these services is much more limited: discussion forums and chat room services, dating services, marketplace and listing services. User base risk factors: Under-18s tend to receive more content promoting drugs on certain social media services than over-18s and therefore our research has identified age and potentially gender as a risk factor. Gender is not necessarily a defining factor for users who sell and purchase drugs and psychoactive substances online. However, men tended to see more Class A drugs promoted online than women. Functionalities and recommender systems risk factors: The ability to post content, in particular, images and emojis, as well as tag content have been identified as important functionalities in the supply of drugs and 266
psychoactive substances online, as these can be used to promote drugs and signpost potential buyers. This can all be supported by the ability to create anonymous user profiles which provides privacy to users conducting illegal activity. Similarly, direct messaging allows suppliers to talk with their customers away from larger user groups. Although it is often encrypted messaging that is favoured over messaging without automatically enabled encryption, due to its added security. Ephemeral messaging can also encourage perpetration by limiting digital traces of purchases. Network recommender systems can recommend other dealers to users and allow them to increase their exposure. This risk can be amplified depending on the design of the recommender system used by the service. Likewise, hyperlinking and user-generated content (UGC) searching can be popular methodologies for linking users to additional illicit content. Users often have to connect with potential dealers through user connections before viewing their user profiles and associated content. Menus or images depicting the products for sale can be posted on services, or in closed user groups. It is also possible to link posting goods or services for sale, reacting to content and commenting on content to the buying and selling offenses connected to drugs and psychoactive substances. However, there is limited research in this chapter to identify their role as a key facilitator of this harm.
Introduction¶
1479 Commission On Combating Synthetic Opioid Trafficking, 2022. Commission on Combating Synthetic Opioid Trafficking: Technical Annexes [Accessed 17 October 2022]. 1480 European Monitoring Centre for Drugs and Drug Addiction (EMCDDA (Demant, J. and Bakken, S.A.), 2019. Technology- facilitated drug dealing via social media in the Nordic countries. [accessed 17 October 2022], p. 15. [Accessed 17 October 2022]. 267
Relevant offences¶
How drugs and psychoactive substances offences manifest online¶
13.8
1481 As per the Misuse of Drugs Act 1971, ‘Supplying’ includes distributing, and ‘Controlled drugs' refers to the definition listed in Schedule 2 and categorised as Class A (e.g., cocaine, ecstasy), Class B (e.g. cannabis, codeine) and Class C Drugs (e.g. benzodiazepines, diazepam). A ‘psychoactive substance’ is defined as a substance which is capable of producing a psychoactive effect on a person who consumes it. 1482 Section 4(3) of the Misuse of Drugs Act 1971. 1483 Section 9A of the Misuse of Drugs Act 1971. There is very limited evidence linked to this particular offence; for further information on this offence, please refer to the Illegal Content Judgements Guidance. Throughout this chapter, we expect the risk factors associated with this offence to be largely similar to the offence of unlawful supply, or offer to supply, of controlled drugs.
1484 Section 19 of the Misuse of Drugs Act 1971. 1485 Section 5 of the Psychoactive Substances Act 2016. 1486 Dewey, M. & Buzzetti, A. 2024. Easier, faster and safer: The social organization of drug dealing through encrypted messaging apps. Sociology Compass, 18(2). [accessed 22 October 2024]. 1487 National Crime Agency (NCA), 2023. 268
sale on social media sites or apps.1488 A 2019 Volteface poll of 16–24-year-olds found that one in four young people had seen illicit drugs advertised for sale on social media. Out of those who reported seeing illicit drugs for sale on social media, Cannabis was identified as the most seen drug, with nearly two-thirds (63%) of respondents saying they had seen it offered for sale on social media services, followed by cocaine (26%) and MDMA (24%).1489 Among those aged under 18 the figures cannabis was still the most common drug seen (70%), followed by cocaine (28%) and MDMA (35%).1490
Risks of harm presented by supply or offer to supply controlled drugs and psychoactive substances¶
1488 Please note the survey sample of 1,919 young people was self-selecting so it is not possible to determine whether it was representative of the whole population of 13–18-year-olds in the UK. Source: Daniel Spargo-Mabbs Foundation, 2021. Young people, drugs and social media – a survey for 13-18 year olds. [accessed 31 October 2024]. 1489 Volteface, 2019. DM for details: Selling drugs in the age of social media [Accessed 17 October 2022]. 1490 Volteface., 2019. 1491 Office for National Statistics, 2022. Drug misuse in England and Wales: year ending June 2022. [accessed 7 August 2023]. 1492 In addition, a significant proportion of young people reported a problem with alcohol (41%) and nicotine (12%): Office for Health Improvement and Disparities, 2022. Young people's substance misuse treatment statistics 2020 to 2021: report. [accessed 8 May 2023].
1493 Office for National Statistics, 2022. Deaths related to drug poisoning in England and Wales: 2021 registrations. [accessed 8 May 2023]. 1494 UK Parliament, 2024. Reducing the harm from illegal drugs [accessed 30 October 2024]. 1495 World Health Organisation, 2022. Drugs (psychoactive). [accessed 24 October 2022]. 269
Service.1496 1497 The risks associated with traditional street dealing also continue to be prevalent as a result of the options for in-person collections.1498
1496 Between 2019/2020 there were ”7,027 hospital admissions for drug-related mental and behavioural disorders”, ”16,994 hospital admissions for poisoning by drug misuse” and “99,782 admission with a primary or secondary diagnosis of drug-related mental and behavioural disorders”: NHS England, 2021. Statistics on Drug Misuse, England 2020. [accessed 18 November 2024]. 1497 In 2022 the Government published statistics relating to the Children in Need Census; in the year up to the 31 March 2022, Drug misuse concerns relating to the child and a parent were a factor in 92,250 cases. ‘Children in Need’ are a legally defined group of children (under the Children Act 1989), assessed as needing help and protection as a result of risks to their development or health. The latest data for ‘Children in Need’ is available: UK Department of Education, 2024. Reporting Year 2024. [accessed 18 November 2024]. 1498 Debt bondage, a real or perceived debt used as a method to exert control over individuals to carry out tasks including drug dealing, is a common risk associated with street-level drug dealing. Such debt can be incurred through accepting drugs as a ‘gift’, which recipients are then expected to repay: Crown Prosecution Service, 2022. County Lines Offending [accessed 18 November 2024]; Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023]; Volteface, 2019. 1499 Gobbi, G., Atkin, T. and Zytynski, T. 2019. Association of Cannabis Use in Adolescence and Risk of Depression, Anxiety, and Suicidality in Young Adulthood AMA Psychiatry: A Systematic Review and Meta-analysis. JAMA Psychiatry, 76(4):426-43. [accessed 24 October 2024]. 1500 Kiburi, S. K., Molebatsi, K., Ntlantsana, V. and Lynskey, M. T. 2021. Cannabis use in adolescence and risk of psychosis: Are there factors that moderate this relationship? A systematic review and meta-analysis. Substance Abuse, 2021;42(4):527-542. [accessed 24 October 2024]. 1501 Barnardos, 2023. Child exploitation: a hidden crisis. [accessed 28 October 2024]. 1502 A Nitazene is a Class A synthetic opioid associated with mimicking the effects of natural opioids. These drugs have a high potency and are often cut with additional drugs increasing the risk of overdose. 1503 An example is Alprazolam, a medicine in the benzodiazepine family of drugs. This is ten times stronger than diazepam and is not prescribed by the National Health Service. The UK Government has recognised that Alprazolam, in the form of counterfeit Xanax tablets, is a commodity available in street-level drug dealing markets and on illegal website and social media service: UK Health Security Agency, 2024. Alprazolam (Xanax): What are the facts? [accessed 24 October 2024].
1504 One in 2022 and six in 2021. Source: NCA, 2024. 1505 NCA, 2024. 1506 NCA, 2024. 270
(WHO) recognises counterfeit drugs as one of the urgent health challenges for the next generation.1507
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services and video-sharing services¶
1507 World Health Organization, n.d. Substandard and falsified medical products. [accessed 15 October 2024]. 1508 Chemsex-context illicit substances are drugs that are associated with sexual activity. The relationship between these drugs and sexual behaviour, creates a link with harms of a sexual nature. 1509 NCA response to the November 2023 Illegal Harms Consultation. 1510 For instance, researchers in Europe highlighted the relative ease with which they identified suspected drug dealers on Snapchat compared to Facebook and Instagram using the same search queries and slang terminology European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) (Demant, J. and Aagesen, K.M.B.), 2022. An analysis of drug dealing via social media. [accessed 22 October 2024].
1511 The Independent. 2021. One in five 13-14-year-olds have seen drugs being sold on social media [accessed 25 October 2024]. 1512 Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023] [accessed 17 October 2022]. 271
1513 Petersen et al. found that of the 152,308 pieces of online content identified specifically relating to study drugs on Instagram, 27.3% related to illicit drugs supply. The majority of this content also emphasised the benefits of using these drugs. Petersen et al. 2021. #studydrugs-Persuasive posting on Instagram [accessed 17 October 2022]. 1514 Mackey and Kalyanam found that of the 28,711 Fentanyl-related posts identified on Twitter during 2015, a period when the Fentanyl crisis was escalating, only 771 (<1% of total) were determined to be promoting the marketing and sale of Fentanyl and other controlled substances online after isolating posts relating to news reports. Mackey and Kalyanam. 2017. Detection of illicit online sales of fentanyls via Twitter [accessed 17 October 2022]. 1515 Fuller, A., Vasek, M., Mariconti, E., and Johnson, S.D., 2023. Understanding and preventing the advertisement and sale of illicit drugs to young people through social media: A multidisciplinary scoping review. [accessed 21 June 2024]. 1516 A 2018 study collected a total of 12,857 posts from Instagram relating to Xanax, OxyContin, LSD and MDMA. They found a total of 1,228 posts by those likely to be drug dealers, comprising 267 unique users. Of the 1,228 detected posts analysed, 232 explicitly included an offer for supply or an offer to purchase. These included posts, or comments within posts, from users offering to supply drug(s) (with contact information), and comments from other users asking for more information or requesting to buy the drug. Li, J., Zu, Q., Shah, N. and Mackey, T. 2018. A Machine Learning Approach for the Detection and Characterization of Illicit Drug Dealers on Instagram: Model Evaluation Study [accessed 17 October 2022]
1517 A precursor is a chemical needed to synthesise a drug. 1518 Commission On Combating Synthetic Opioid Trafficking, 2022. Commission on Combating Synthetic Opioid Trafficking: Technical Annexes E-14. [accessed 17 October 2022]. 1519 Volteface, 2019. DM for details: Selling drugs in the age of social media. [accessed 17 October 2022]. See functionalities section for more information. 1520 Volteface, 2019. 272
Messaging services¶
Discussion forums and chat room services¶
Dating services¶
1521 Volteface, 2019; Moyle, L., Childs, A., Coomer, R., and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63 101-110 [accessed 3rd June 2023]. 1521 European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) (Demant, J. and Bakken, S.A.), 2019. Technology-facilitated drug dealing via social media in the Nordic countries. [accessed 17 October 2022]. 1522 European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) (Demant, J. and Aagesen, K.M.B.), 2022. An analysis of drug dealing via social media. [accessed 22 October 2024].; EMCDDA (Demant, J. and Bakken, S.A.), 2019: Volteface, 2019. 1523 Dewey, M. & Buzzetti, A. 2024. Easier, faster and safer: The social organization of drug dealing through encrypted messaging apps. Sociology Compass, 18(2). [accessed 22 October 2024]. 1524 Volteface, 2019. DM for details: Selling drugs in the age of social media [Accessed 17 October 2022]; Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023]; C4ADS. 2020. Lethal Exchange: Synthetic Drug Networks in the Digital Era [accessed 17 October 2022].
1525 European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) (Demant, J. and Aagesen, K.M.B.), 2022. An analysis of drug dealing via social media. [accessed 22 October 2024]. 1526 Interest sites refers to websites dedicated to a particular subject such as gaming, fashion and motoring. 1527 NCA response to the November 2023 Illegal Harms Consultation. 273
of knowledge of drugs purchase options.1528 High-risk high-harm substances, such as methamphetamine, GHB/GBL, and synthetics, are often advertised for sale as ‘bundles’ of multiple substances on dating apps. This is of particular concern to law enforcement as the increase in poly-drug use is believed to be a factor linked to the increase in drug-related deaths across the UK.1529
Risk factors: User base¶
User base demographics¶
1528 NCA, 2023. 1529 NCA response to the November 2023 Illegal Harms Consultation.
1530 NCA, 2023. 1531 International Narcotics Control Board, 2023. Annual Report 2023. [accessed 15 October 2024]. 1532 Crest Advisory (Caluori, J., Mooney, B. and Kirk, E.), 2023. Running out of credit: Mobile phone tech and the birth of county lines. [accessed 12 May 2023]. 1533 Volteface, 2019. DM for details: Selling drugs in the age of social media. [accessed 17 October 2022]. 1534 NCA response to the November 2023 Illegal Harms Consultation. 274
Risk factors: functionalities and recommender systems¶
User identification¶
User profiles¶
Anonymous User profiles¶
1535 Crack cocaine seen advertised online for sale on social media by 16 to 24 year olds: 16.3% male vs 7.7% female. Heroin seen advertised online for sale on social media by 16 to 24 year olds: 10.3% male vs 4.7% female. Source: Volteface, 2019. 1536 Volteface, 2019. 1537 Volteface, 2019. 1538 Volteface, 2019. 1539 Examples include users including emojis in their user profiles that other users could look for to identify a potential dealer. Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023]. 1540 Volteface, 2019. 1541 “For example, one account had two profiles where the only difference was the surname: ‘Green’ and ‘Greenn’”. Source: Volteface, 2019.
1542 Hammond, A.S., Paul, M.J., Hobelmann, J., Koratana, A.R., Dredze, M. and Chisolm, M. 2018. Perceived Attitudes About Substance Use in Anonymous Social Media Posts Near College Campuses: Observational Study. JMIR Mental Health 5(3). [accessed 17 October 2022]. 1543 Sanden, R.v.d., Wilkins, C. Rychert, M. and Barratt, M.J., 2022. The Use of Discord Servers to Buy and Sell Drugs, Contemporary Drug Problems, 49(4), 453-477. [accessed 18 November 2024]. 275
User networking¶
User connections¶
13.40
User groups¶
progress from public groups to private groups via invite links.1546 Likewise, closed user groups on some social media services, where an invitation is required, allow suppliers to promote their services, providing a contact for potential buyers.
User communication¶
Direct messaging¶
13.45
13.46
1544 Volteface, 2019.
1545 Volteface, 2019. 1546 NCA, 2024. 1547 C4ADS, 2020. LETHAL EXCHANGE: SYNTHETIC DRUG NETWORKS IN THE DIGITAL ERA. [accessed 17 October 2022]. 1548 European Monitoring Centre for Drugs and Drug Addiction (Demant, J. and Bakken, S.A.), 2019. Technology-facilitated drug dealing via social media in the Nordic countries. [accessed 17 October 2022]. 1549 European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) (Demant, J. and Bakken, S.A.), 2019. 276
13.47 popular for purchasing and/or selling drugs and other psychoactive substances. Messages
Encrypted messaging¶
13.49
Ephemeral messaging¶
13.51
1550 Volteface, 2019. DM for details: Selling drugs in the age of social media. [accessed 17 October 2022]. 1551 Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023].
1552 NCA, 2024. 1553 Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs. [accessed 17 October 2022]. 1554 Volteface, 2019. DM for details: Selling drugs in the age of social media [Accessed 17 October 2022]. 1555 Volteface, 2019. 1556 C4ADS, 2020. Lethal Exchange: Synthetic Drug Networks in the Digital Era [Accessed 17 October 2022]. 277
offer to supply illicit drugs.1557 While stories are not messages and are more closely related
13.52 suggested that a video-sharing service’s use of ‘ephemeral messaging’ is a common external method of contact linked by dealers in their adverts on services.1559
Reacting and commenting on content¶
13.53
Posting content (images, videos, text, emojis)¶
13.54
1557 Volteface, 2019.. 1558 Moyle et al. 2019. 1559 NCA response to the November 2023 Illegal Harms Consultation. 1560 Moyle et al. 2019. 1561 Volteface, 2019. DM for details: Selling drugs in the age of social media [Accessed 17 October 2022]
1562 “For example, to indicate the strain of cannabis, the account would describe it as ‘Lemon Haze’”. Source: Volteface, 2019. 1563 For example: “200 vals £90 200 lorazepam £120”. Source: Volteface, 2019. 1564 European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) (Demant, J. and Bakken, S.A.), 2019. Technology- facilitated drug dealing via social media in the Nordic countries [Accessed 17 October 2022]. 1565 Volteface, 2019. 278
13.57 Although images of potentially harmful content can be recognised by content moderation tools, the multiple formats these drugs can take make it more difficult to identify illegal substances.1567
Transactions and offers¶
Posting goods or services for sale¶
Content exploring¶
User-generated content searching¶
13.60
Content tagging¶
13.61
1566 Volteface, 2019. 1567 National Crime Agency response to the November 2023 Illegal Harms Consultation 1568 Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023]. 1569 Volteface, 2019.
1570 The other options were asking a friend, asking a family member, and asking a stranger. Source: Volteface, 2019. 1571 Volteface, 2019. 1572 For example, #buy, #sell, #buypainmeds, #drugsforsale, #opioids, #painmeds, and #controlled. Source: Mackey, T., Kalyanam, J., Klugman, J., Kuzmenko, E and Gupta, R. 2018. Solution to Detect, Classify, and Report Illicit Online Marketing and Sales of Controlled Substances via Twitter: Using Machine Learning and Web Forensics to Combat Digital Opioid Access. Journal of Medical Internet Research, 20(4). [accessed 18 November 2024] 279
Hyperlinking¶
Recommender systems¶
Network recommender systems¶
1573 Commission On Combating Synthetic Opioid Trafficking. 2022. Commission on Combating Synthetic Opioid Trafficking: Technical Annexes [accessed 17 October 2022]. 1574 Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023]. 1575 Mackey et al., 2017. Twitter-Based Detection of Illegal Online Sale of Prescription Opioid. [accessed 17 October 2022].
1576 Petersen et al. 2021. #studydrugs-Persuasive posting on Instagram. [accessed 17 October 2022]. 1577 For example, Metropolitan Police warned Londoners in August 2024 not to scan QR codes promoting illicit cannabis products, with the QR codes taking users to fully operational online marketplaces selling a variety of likely illegal drugs products. Source: Thurston, A. 2024. Londoners warned not to scan mysterious QR codes linking to slick website selling cannabis, MyLondon, 24 August. [accessed 31 October 2024]. 1578 Center for Countering Digital Hate, 2023. TikTok’s Toxic Trade. [accessed 25 August 2024]. 280
users who were suspected of drug dealing, they were suggested ‘mutual friends’ of dealers who were also suspected of dealing. Additionally, researchers saw more user profiles suspected of drug dealing appear in the search bar because of mutual user connections.1579
Risk factors: Business model¶
1579 Volteface, 2019. DM for details: Selling drugs in the age of social media. [accessed 17 October 2022]. This study found that, “once a few drug dealer accounts had been followed, the platforms would soon start ‘suggesting’ other drug dealer accounts to follow.” 281
Section 14 Firearms, knives and other weapons¶
Summary analysis for firearms, knives and other weapons offences: how harm manifests online and risk factors This chapter summarises the risks of harm to individuals that could happen due to several offences linked to the buying and selling of firearms, knives and other weapons online. The risks of harm to individuals from these offences are broad. Publication of material in connection with the marketing of knives and the sale or offering to supply firearms, knives and other weapons may result in violent crime, with the most extreme consequence being loss of life. In the year ending March 2023, 19,555 cautions and convictions were made for possession of a knife or offensive weapon; 18% (nearly 1 in 5) of the cases involved juveniles aged between 10 and 17. While it is difficult to report how many of these knives were originally bought or marketed online, the glamourisation of weapons to young people is of particular concern. Research has identified that more than half of teens have said they had seen real-life acts of violence on social media in the past 12 months and that young people are displaying positive attitudes towards the accessibility and ‘coolness’ of knives.Offenses in relation to firearms are less prevalent online and appear to operate in closed networks, such as encrypted messaging services. The online sale and subsequent importing of prohibited front-venting blank firing firearms continues to be a focus for Service type risk factors: Our evidence points to user-to-user (U2U) services as significant facilitators of these priority offences., However, the role of these service types varies across different offences. For facilitating the supply of illegal knives and sharp weapons our evidence suggests social media and direct messaging services are particularly important. Online marketplaces and listing services and discussion forums and chat rooms are prominent service types in the facilitation or commission of weapons offences related to various weapons. User base group: The involvement of young people inadvertently and accidentally in knife and weapon crime signifies that age is a potential risk factor. Likewise, crime data for knife and weapons offenses shows that gender could also be an important influencing feature, with most offenders and victims of weapons offences being men and boys. Functionalities and recommender systems risk factors: 282
Unlike the service types the role of functionalities and recommender systems varies less across different offence types. The ability to generate anonymous user profiles and use direct and encrypted messaging are key functionalities that facilitate the marketing, sale and purchasing of firearms, knives and other weapons by allowing a direct – privacy preserving – channel of communication between purchaser and seller. Posting goods or services for sale particularly supports the ability to sell and purchase knives as there is greater complexity around their legality. By association with other buying and selling offences (the supply of drugs and psychoactive substances), it may be possible to infer that, user-generated content searching and commenting on content and tagging users may be risk factors associated with the sale, hire, purchase and marketing of firearms, knives and other weapons.
Introduction¶
Relevant offences¶
completely illegal to possess, sell and supply those weapons. These are often classified as “offensive weapons” and include items such as Butterfly Knives, Zombie Knives and Swords. However, there are other instances, where the Act includes items that can be legal to possess, sell and supply, but only under certain circumstances. For example, some items are commonly prohibited to under 18-year-olds (for example, kitchen knives), whilst others may also require certification to own (for example, a firearm).
1580 Section 1(1) and Section 2(1) of the Firearms Act 1968; section 2 of the Air Weapons and Licensing (Scotland) Act 2015 (asp 10). 1581 Section 3(1) of the Firearms Act 1968; Article 24 of the Firearms (Northern Ireland) Order 2004 (S.I. 2004/702 (N.I. 3); section 24 of the Air Weapons and Licensing (Scotland) Act 2015 (asp 10). 1582 Section 3(2) of the Firearms Act 1968; section 24 of the Air Weapons and Licensing (Scotland) Act 2015 (asp 10); Article 37(1) of the Firearms (Northern Ireland) Order 2004 (S.I. 2004/702 (N.I. 3). 1583 Section 22(1) of the Firearms Act 1968. 1584 Section 24 of the Firearms Act 1968. 1585 Section 21(5) Firearms Act 1968; Article 63(8) of the Firearms (Northern Ireland) Order 2004 (S.I. 2004/702 (N.I. 3)). 1586 Section 24(A) of the Firearms Act 1968; Article 66A of the Firearms (Northern Ireland) Order 2004 (S.I. 2004/702 (N.I. 3). 1587 Section 24(A) of the Firearms Act 1968; Article 66A of the Firearms (Northern Ireland) Order 2004 (S.I. 2004/702 (N.I. 3).
1588 Section 36(1)(c) of the Violent Crime Reduction Act 2006 (sale etc of realistic imitation firearms). 1589 Section 1 and Section 2 of the Knives Act 1997. 1590 Section 5(1), (1A) or (2A) of the Firearms Act 1968; Article 45(1) or (2) of the Firearms (Northern Ireland) Order 2004 (S.I. 2004/702 (N.I. 3)). 1591 Section 1 of the Crossbows Act 1987. 284
• Purchase or hire of a crossbow by a person under the age of 181592 • Manufacture, sale, hire, offer for sale, expose, possess for sale or hire, lend or give to another person banned knives or offensive weapons1593 • Sale of knives or other articles with blade or point to underage persons1594 • Importation of banned knives, offensive weapons or realistic imitation firearms1595
How firearms, knives and other weapons offences manifest online¶
1592 Section 2 of the Crossbows Act 1987. 1593 Section 1(1) of the Restriction of Offensive Weapons Act 1959; Section 141(1) of the Criminal Justice Act 1988; Article 53 of the Criminal Justice (Northern Ireland) Order 1996 (S.I. 1996/3160 (N.I. 24)). 1594 Section 141A of the Criminal Justice Act 1988; Article 54 of the Criminal Justice (Northern Ireland) Order 1996 (S.I. 1996/3160 (N.I. 24)).
1595 Section 1(2) of the Restriction of Offensive Weapons Act 1959; section 141(4) of the Criminal Justice Act 1988; section 36(1)(d) of the Violent Crime Reduction Act 2006 (sale etc of realistic imitation firearms). 1596 Crown Prosecution Service, 2022. Offensive Weapons, Knife Crime Practical Guidance, Marketing of Knives section. [accessed 18 November 2024]. 1597 Publicly accessible internet. Often referred to as the ’Surface Web’. 285
between the supplier and the buyer.1598 Contrastingly, knives and other weapons are more accessible on the clear web because there is a larger variety of legal products. However, regarding the sale of knives and other weapons, investigative reporting from Which?, a consumer protection organisation, highlighted that third-party sellers on large online marketplaces have advertised for sale various prohibited weapons, including various kinds of knives, to UK consumers.1599
Risks of harm to individuals presented by firearms, knives and other weapons offences online¶
1598 National Crime Agency (NCA), 2024. Illegal firearms. [accessed 22 October 2024]. 1599 Which?, 2022. Illegal weapons for sale on AliExpress, Amazon, eBay and Wish, Which? warns. [accessed 22 October 2024]. 1600 NCA response to the November 2023 Illegal Harms Consultation. 1601 Commonly referred to as a ‘Stun Gun’ and by the manufacturer brand ‘TASER’. 1602 Self-loading hybrid firearms consisting of approximately 80% 3D-printed components combined with easily accessible metal non-firearms parts. Source: Home Office, 2023. Consultation document (accessible). [accessed 20 September 2023]. 1603 Ofcom / Law Enforcement event / meeting 6 June 2024. 1604 Crown Prosecution Service (CPS), 2023. Offensive Weapons, Knife Crime Practical Guidance, Marketing of Knives section. [accessed 22 October 2024]. 1605 NCA, 2024. Criminals still want to acquire and use original lethal purpose weapons but they are finding them more difficult to obtain. [accessed 22 October 2024]. For information on 3D Printed Firearms, please see the Terrorism chapter of the Register of Risks. 1606 There is no overarching consensus on defining a USG. According to the Centre for Social Justice’s (CSJ) 2009 report ‘Dying to Belong’, an Urban Street Gang is defined as “a relatively durable, predominantly street-based group of young people, who see themselves (and are seen by others) as a discernible group; engage in criminal activity and violence; lay claim over territory (not necessarily geographical but can include an illegal economy territory); have some form of identifying structural feature; and are in conflict with other, similar, gangs”. Source: Centre for Social Justice, 2009. Dying to Belong. [accessed 20 September 2023].
1607 Organised crime groups are defined as a group of “members who plan, coordinate and carry out serious crime on a continuing basis. Their motivation is often, but not always, financial gain. Many OCGs are loose networks of criminals who come together for a specific criminal activity, acting in different roles depending on their skills and expertise”. Source: CPS, 2021. Gang related offences – Decision making in. [accessed 20 September 2023]. 286
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Messaging services¶
1608 NCA response to the November 2023 Illegal Harms Consultation. 1609 Collectively, front-venting and top-venting blank firers are known as ‘convertible blank firers’. Source: NCA, 2024. Firearms threat assessment 2024. [accessed 14 October 2024]. 1610 NCA, 2024. Illegal firearms. [accessed 11 September 2024].
1611 House of Commons Library, 2023. Knife Crime in England and Wales: Statistics, p.5. [accessed 18 November 2024]. 1612 Cogan, N., Chin-Van Chau, Y., Russell, K., Linden, W., Swinson, N., Eckler, P., Knifton, L., Jordan, V., Williams, D., Coleman, C., and Hunter, S., 2021. Are images of seized knives an effective crime deterrent? A comparative thematic analysis of young people’s views within the Scottish context, PsyArXiv Preprints. [accessed 18 November 2024]. 30 Youth Endowment Fund, 2022. Children, violence and vulnerability 2022. [accessed 20 September 2023]. 1614 Ofcom / Law Enforcement meeting, 6 June 2024. 287
to move forward with a purchase.1615 It has been suggested that these private messaging platforms can be complex and time consuming to investigate, making them popular within criminal networks.1616 Encrypted messaging software has also been recognised as a methodology for facilitating the sale of weapons in support of international conflicts.1617
Social media services¶
Marketplaces and listing services, discussion forums and chat rooms¶
1615 “Aman described seeing a range of things for sale, most commonly weapons or drugs, posted by other Snapchat users to their Stories. He said he sees the items with price tags attached to them, often laid out in someone’s house. “They just lay it down on a bed or something. A lot of knives or knuckledusters, with the money sign and the amount it is. And then just says, ‘Text me if you wanna buy it.’ They ask you to text them on Snapchat. After that, they’d get in contact in real life and sell to each other.” Source: Revealing Reality, 2023. Anti-social Media. [accessed 11 September 2024]. 1616 Home Office, 2023. Consultation on new knife legislation proposals to tackle the use of machetes and other bladed articles in crime. [accessed 11 September 2024]. 1617 The Economist, 2024. How encrypted messaging apps conquered the world. [accessed 11 September 2024]. 1618 Research by Revealing Reality that explored the content vulnerable children were seeing on Snapchat highlighted that many of the children in the sample had seen content involving firearms, knives and other weapons, including people brandishing these weapons, displaying or using knives or weapons in fights, or advertising weapons for sale. Source: Revealing Reality, 2023. Anti-social Media. [accessed 11 September 2024]. 1619 Sky News, 2024. Teens buying knives illegally online as criminals 'move with digital age'. [accessed 11 September 2024]. 1620 Spring, M, 2022. A social media murder: Olly’s story, BBC News, 20 June. [accessed 11 September 2024].
1621 The Crown Prosecution Service, 2021. Gang related offences - Decision making in. [accessed 11 September 2024]. 1622 Mayor of London, 2023. Study shows impact of knife imagery not universal, but is more profound for some young people affected by violence. [accessed 11 September 2024]. 1623 NCA, 2024. Illegal firearms. [accessed 22 October 2024]. 1624 Which?, 2022. Illegal weapons for sale on AliExpress, Amazon, eBay and Wish, Which? warns. [accessed 22 October 2024]. 288
Risk factors: User base¶
User base demographics¶
Risk factors: Functionalities and recommender systems¶
User identification¶
Anonymous user profiles¶
Transactions and offers¶
Posting goods or services for sale¶
1625 MOJ, 2024. Knife and Offensive Weapon Sentencing Statistics: July to September 2023. [accessed 11 September 2024]. 1626 Youth Endowment Fund, 2022. Children, violence and vulnerability 2022., p. 91. [accessed 17 October 2024].
1627 ONS, 2024. Homicide in England and Wales. [accessed 11 September 2024]. 1628 ONS, 2024. Knife and Offensive Weapon Sentencing Statistics: January to March 2024. [accessed 11 September 2024]. 1629 Bakken, S.A. and Demant, J.J., 2019. Sellers’ risk perceptions in public and private social media drug markets. [accessed 22 October 2024]. 1630 Van der Sanden, R., Wilkins, C., Rychert, M. and Barratt, M. J. 2022. ‘Choice’ of social media platform or encrypted messaging app to buy and sell illegal drugs, Internal journal of drug policy 108. [accessed 22 October 2024]. 289
starting from as little as 49p.”1631 Often prohibited items sold online in the UK are made available through international services with users who operate under different legislation.1632
Direct messaging¶
Encrypted messaging¶
Commenting on content and user tagging¶
Content exploring¶
User-generated content searching¶
1631 Which?, 2022. Illegal weapons for sale on AliExpress, Amazon, eBay and Wish, Which? warns. 1632 Petrakos, K, 2024. Warning as machetes, knives and swords for sale online for as low as £1.17, i News, 25 August . [accessed 17 October 2024]. 1633 Revealing Reality, 2023. Anti-social Media. [accessed 11 September 2024]. 1634 Van der Sanden, R., Wilkins, C., Rychert, M. and Barratt, M. J., 2022. 1635 Home Office, 2023. Consultation on new knife legislation proposals to tackle the use of machetes and other bladed articles in crime. [accessed 11 September 2024]. 1636 Campbell, D. 2022. Encrypted messaging system used to procure gun for murder, court hears. The Guardian, 8 February. [accessed 17 October 2024].
1637 The Economist, 2024. How encrypted messaging apps conquered the world. [accessed 11 September 2024]. 1638 Sherlock, G, 2024. Man who used secret chat to sell guns and drugs jailed, BBC News, 20 August. [accessed 17 October 2024]. 1639 Volteface, 2019. DM for Details: Selling Drugs in the Age of Social Media. [accessed 22 October 2024] 1640 See ‘Social media services’, ‘Marketplaces and listing services, discussion forums and chat rooms’ and File-storage and file sharing services’ earlier in this chapter 290
A Which? investigation found that it was easy to conduct simple searches for banned offensive weapons on popular online marketplaces and that specific characters were often used within the item’s title to avoid detection.1641 Research has also identified that the absence of age verification on certain services has enabled under-18s to purchase weapons prohibited for that age group which they have found and bought through a search for user-generated content in which knives and weapons are posted for sale.1642 1643
Risk factors: Business models and commercial profiles¶
1641 Which?, 2022. Illegal weapons for sale on AliExpress, Amazon, eBay and Wish, Which? warns. [accessed 22 October 2024]. 1642 Busby, M. 2019. Knives being sold via Facebook without any age check. The Guardian, 9 August. [accessed 22 October 2024]. 1643 Which?, 2023. Illegal weapons and age-restricted items sold without checks on Temu. [accessed 22 October 2024]. 291
Section 15 Encouraging or assisting suicide (or attempted suicide), and serious self-harm¶
Warning: This chapter contains discussion of suicide, self-harm and eating disorders.1644 Summary analysis for encouraging or assisting suicide (or attempted suicide), and serious self-harm: how harm manifests online, and risk factors Suicide and self-harm are two different offences under the Online Safety Act. However, we have included them together in this chapter because it is often difficult to draw a clear distinction between the two types of content, and much of the research in this area that contributes to our understanding does not focus solely on one or another type of content. The two offences carry risk of harm from the same service types and functionalities. The suicide offence takes place when an individual intentionally encourages or assists a person to end their life, or attempt to end their life. Ofcom’s 2024 Online Experiences Tracker found that 4% of UK internet users reported seeing or experiencing content ‘promoting suicide’ in the past four weeks.1645 Younger respondents were more likely to see or experience this content, with 6% of 13-to-24-year-olds, 8% of 18-to-24-year-olds and 5% of 25-to-34-year-olds, compared to 3% of those in age groups 35 or older.1646 The physical and psychological harms that can arise from these offences are severe and can include long-term mental health concerns, eating disorders, physical harm to oneself, and death. Harm from these offences can affect both viewers of the content and the user posting the content themselves. The role of online content in encouraging or assisting suicide must also be understood in the context of increasing rates of suicide in the UK. The Office for National Statistics (ONS) estimates that the age-standardised suicide rate in England and Wales increased by 15% between 2010 and 2022.1647
1644 If you need support, please check the following websites: NHS, 2023. Help for suicidal thoughts.; NHS, 2023. Where to get help for self-harm; Beat, 2023. Helplines for eating disorder support. 1645 This may include content that could be deemed illegal. 1646 Ofcom, 2024. Online Experiences Tracker 2024. [accessed 18 November 2024]. Fieldwork was carried out in a four- week period in May and June 2024. 1647 Office for National Statistics, 2024. Suicides in England and Wales: 2023 registrations. [accessed 15 August 2024]. 292
1648 Ofcom, 2024. Online Experiences Tracker. [accessed 22 November 2024]. Fieldwork was carried out in a four-week period in May and June 2024. Content users reported seeing may include content that could be deemed illegal. 1649 For example, using the same hashtags, posted to the same accounts, or in the same user groups. 293
facilitate discussions on niche or specialised content among smaller groups of users, which could include suicide or self-harm content. However, discussion forums and chat rooms may also be used by individuals experiencing mental health difficulties to connect with other users for support and guidance. Social media services can allow users who may intentionally wish to hurt others to disseminate illegal or harmful suicide or self-harm related content. Users can view suicide or self-harm content on these types of services, particularly through the creation of user groups on social media services. Services that allow users to build online communities are also a risk factor, as online communities can act as spaces where suicide, self-harm or eating disorders are promoted or encouraged. User base risk factors: Small and large user base sizes can pose risks for different reasons. With a larger user base, more people risk encountering this content, while smaller user bases can encourage the sharing of specialised and extreme content relating to suicide, self-harm or eating disorders. Users who are in vulnerable circumstances such as such as those struggling with their mental health and who might be experiencing thoughts of suicide or self-harm are more likely than other users to be at risk from the effects of this type of content. Age is also a potential risk factor. Our evidence suggests that young people are more likely to encounter this content, to use the internet for suicide or self-harm related purposes, and to be susceptible to copycat behaviour. Functionalities and recommender systems risk factors: Commenting on content is a risk factor, and there is evidence of people using comments to encourage the suicide or self-harm of the person that distributed the content. Commenting on content intersects with other risk factors such as livestreaming and posting content to create high-risk context. For example, livestreaming is a risk factor that has been used to share real-time acts of suicide or self-harm. While livestreaming these activities is not in itself illegal, the social functionality attached to the livestream, including commenting on the livestream or in user groups connected to the livestream can be used to encourage a potential suicide attempt, or encourage users to self-harm on a livestream. Similarly, comments on posts related to suicide or self-harm can encourage the user, who may be experiencing thoughts of suicide or self-harm, to attempt to take their own life, or self-harm. Anonymous user profiles appear to be a risk factor. Some users may feel more confident in sharing content depicting or discussing harmful themes if they cannot be identified, including assisting or encouraging others in acts of suicide or serious self-harm. However, users may also feel that anonymity allows them to talk more 294
openly about their own thoughts of suicide and self-harm, and to connect with others with similar experiences. Therefore, anonymity can both pose risks and confer potential benefits to those seeking help. The ability to post content and re-post or forward content can enable and benefit users to connect with others who are experiencing similar thoughts or behaviours, but it can also be used to disseminate harmful suicide and self-harm content. Content recommender systems can also be a risk factor for this type of content. The way in which recommender systems are designed can influence the extent to which harmful (and potentially illegal) content is recommended to users. Recommender systems are commonly designed to optimise for user engagement, and learn about users’ preferences through implicit (e.g., viewing multiple times) and explicit (e.g., liking, sharing, and commenting) user feedback. Research suggests that where there are vulnerable users who are engaging with harmful content, such as self-harm or suicide content, recommender systems are more likely to create a ‘filter bubble’ or ‘rabbit hole.’ This may lead to users discovering more content that is harmful or distressing, as well as potentially illegal. If a user is primarily engaging with harmful content, then this is likely to create a filter bubble where the user is recommended more harmful content, while other content is deprioritised. Other functionalities risk propagating these offences are content tagging such as hashtags as they can help evade content moderation techniques on suicide or self-harm content, because groups of users create hashtags that differ from those that may be blocked as harmful. Group messaging can also enable users to contact one another and can encourage harmful behaviour in a group setting.
Introduction¶
encountered by an individual, or where content of a particular kind is encountered in combination with content of a different kind.1650
Relevant offences¶
1650 As with other chapters, we have considered evidence of suicide and self-harm content from a variety of sources, including information provided by services, academic literature, third-party research and civil society in general. Some of this evidence relates to content which may not necessarily mirror, or is broader than, the criminal definitions of these offences.
1651 Brennan, C., Saraiva, S., Mitchell, E, Melia, R., Campbell, L., King, N. and House, A., 2022. Self-harm and suicidal content online, harmful or helpful? A systematic review of the recent evidence, Journal of Public Mental Health, 21 (1). [accessed 10 July 2023]. 296
How encouraging or assisting suicide or serious self-harm manifests online¶
1652 Section 2 of the Suicide Act 1961 and section 13 of the Criminal Justice Act (Northern Ireland) 1966 (c. 20 (N.I.)). (c. 20 (N.I.)). 1653 Section 184 of the Online Safety Act 2023. 1654 Samaritans, 2020. Understanding self-harm and suicide content online. [accessed 24 May 2023]. 1655 The Illegal Content Judgements Guidance indicates that a specific kind of eating disorder content could be considered illegal under the spirit of the offence. To that end, we have included evidence on the risks of harm related to types of eating disorder content in this chapter. We note that many risk factors are used in a similar way as suicide and self-harm content. 1656 While some of the content referred to in this chapter may cause harm/distress, it may not necessarily meet the criminal threshold. Please refer to the Illegal Content Judgements Guidance to assess whether content amounts to illegal content.
1657 Ofcom, 2024. Online Experiences Tracker. [accessed 22 November 2024]. Fieldwork was carried out in a four-week period in May and June 2024. Content users reported seeing may include content that could be deemed illegal. 1658 This concern is not limited to the impact of this content on children, but also on adults. Four in five (83%) UK adults agree that “harmful suicide or self-harm content” can have a damaging effect on adults as well as children. Source: Samaritans, 2023. Government is failing the public with online safety bill, says Samaritans. [accessed 19 January 2023]. 297
identify whether exposure to suicide and self-harm related content online causes increased risk of suicide or self-harm related outcomes.1659
Risks of harm to individuals presented by the offences of encouraging or assisting suicide and serious self-harm online¶
1659 Arendt, F., Scherr, S. and Romer, D., 2019. Effects of exposure to self-harm on social media: Evidence from a two-wave panel study among young adults, New Media & Society, 21 (11-12). [accessed 10 July 2023]. 1660 Samaritans, 2022. Towards a suicide-safer internet. [accessed 24 May 2023]. 1661 These examples may not necessarily be illegal in all cases. For information on what could be considered illegal, please refer to the ICJG guidance. 1662 The full list of examples includes: detailed and instructive information about suicide or self-harm methods; posts encouraging, glamourising or celebrating suicide or self-harm; posts from people seeking or encouraging suicide or self-harm pacts; posts relating to suicide or self-harm challenges; graphic images relating to self-harm or suicide; and livestreams or recorded videos of suicidal or self-harming behaviour. 1663 Costs are mainly driven by healthcare services used, including intensive care and psychosocial assessment. Therefore estimated cost of each episode varies substantially across types of self-harm. Source: Tsiachristas, A., Geulayov, G., Casey, D., Ness, J., Waters, K., Clements, C., Kapur, N., McDaid, D., Brand, F., & Hawton, K. (2020). Incidence and general hospital costs of self-harm across England: estimates based on the multicentre study of self-harm. Epidemiology and psychiatric sciences, 29. [accessed 10 October 2024] 1664 Office for National Statistics, 2024. Suicides in England and Wales: 2023 registrations. [accessed 15 August 2024].
1665 Conversely, in Northern Ireland, the age-standardised rate of probable suicides reduced by 36% between 2010 and 2022 - Northern Ireland Statistics and Research Agency, 2023. Suicide statistics 2022 [accessed 12 October 2024]; The age- standardised rate of probable suicides in Scotland in 2022 was 14 per 100,000 people, which was not significantly different from the rate in 2010 (14.7), nor was there any clear positive or negative trend over the years within that range - National Records of Scotland, 2024. Probable Suicides [accessed 12 October 2024]. 1666 Samaritans, 2022. Towards a suicide-safer internet. [accessed 24 May 2023]. 298
harm, and graphic images relating to serious self-harm or suicide.1667 Not all of this content will be illegal, although all has the potential to be harmful. For information on what could be considered illegal, please refer to the ICJG guidance.
1667 The full list of examples includes: detailed and instructive information about suicide or self-harm methods; posts encouraging, glamourising or celebrating suicide or self-harm; posts from people seeking or encouraging suicide or self-harm pacts; posts relating to suicide or self-harm challenges; graphic images relating to self-harm or suicide; and livestreams or recorded videos of suicidal or self-harming behaviour. 1668 The study found that exposure to depictions of self-harm on Instagram resulted in an ‘emotional disturbance’ in some users, with this exposure statistically positively associated with psychometric predictors of “(possibly harmful) self-harm and suicidality-related outcomes”. Source: Arendt, F., Scherr, S. and Romer, D., 2019. Effects of exposure to self-harm on social media: Evidence from a two-wave panel study among young adults, New Media & Society, 21 (11-12). [accessed 10 July 2023]. 1669 The sample included 5,294 individuals aged 16-84 years. Many of the participants in the study were females aged under 25, and so does not represent any population as a whole. Source: Samaritans and Swansea University, 2022. How social media users experience self-harm and suicide content. [accessed 10 July 2023].
1670 Samaritans and Swansea University, 2022. 1671 Susi, K., Glover-Ford, F., Stewart, A., Knowles Bevis, R. and Hawton, K., 2023. Research Review: Viewing self-harm images on the internet and social media platforms: systematic review of the impact and associated psychological mechanisms, Journal of Child Psychology and Psychiatry, 64 (8). [accessed 10 July 2023]. 1672 Susi, K., Glover-Ford, F., Stewart, A., Knowles Bevis, R. and Hawton, K., 2023. 299
the same time desensitised her to the act of self-harm and reinforced self-harm objectives.1673
1673 Livingstone, S., Stoilova, M., Stoilova, M., Stänicke, L. I., Jessen, R. S., Graham, R., Staksrud, E., Jensen, T. 2022. Young people experiencing internet-related mental health difficulties: the benefits and risks of digital skills, ySKILLS. [accessed 10 October 2024]. 1674 Susi, K., Glover-Ford, F., Stewart, A., Knowles Bevis, R. and Hawton, K., 2023. 1675 Royal College of Psychiatrists, 2020. Technology use and the mental health of children and young people. [accessed 10th July 2023].
1676 Domaradzki, J. 2021. The Werther Effect, the Papageno Effect or No Effect? A Literature Review. International Journal of Environmental Research and Public Health, 18(5). [accessed 10 October 2024] 1677 Media portrayals also increase the rate of suicides by the same suicide method as that portrayed. Domaradzki, J. 2021. The Werther Effect, the Papageno Effect or No Effect? A Literature Review. International Journal of Environmental Research and Public Health, 18(5). [accessed 10 October 2024]. 1678 Samaritans, 2022. Towards a suicide-safer internet. [accessed 24 May 2023] 300
take one’s life had risked exacerbating their suicidal thoughts at a time when the men interviewed in the study were feeling distressed, isolated and confused.1679
1679 NatCen (McManus, S., Lubian, K., Bennett, C., Turley, C., Porter, L., Gill, V., Gunnell, D. and Weich, S.), 2019. Suicide and self-harm in Britain – researching risk and resilience. [accessed 10 July 2023]. 1680 Please note that the figure for 2022/23 is much lower at 319/100,000, but the Nuffield Trust attributes this to “(…) the change in NHS England’s reporting methodology, which reclassified Same Day Emergency Care (SDEC) cases, leading to fewer admissions being recorded under the Admitted Patient Care data set.” Source: Nuffield Trust, 2024. Hospital admissions as a result of self-harm in children and young people. [accessed 1 October 2024] 1681 NSSI admissions of patients aged 10-14-years-old increased 148% to 307.1 per 100,000 in 2021/22 from 123.8 in 2012/13. NSSI admissions of patients aged 15-19-years-old increased 37% to 641.7 per 100,000 in 2021/22 from 469.2 in 2012/13: Nuffield Trust, 2024. 1682 Feeding and eating disorders, as defined by the ICD-11, include anorexia nervosa, bulimia nervosa, binge eating disorder, other specified food intake disorder (OSFED), avoidant restrictive food intake disorder (ARFID), rumination disorder and pica. Source: ICD, 2023. Feeding or eating disorders. [accessed 10 July 2023]. 1683 Smith, A., Zuromski, K. and Dodd, R., 2018. Eating disorders and suicidality: what we know, what we don't know, and suggestions for future research, Current Opinion in Psychology, 22. [accessed 10 July 2023]. 1684 Beat, 2023. Online Safety and Eating Disorders, [accessed 10 July 2023]. 1685 Mento, C., Silvestri, M C., Muscatello, M R A., Rizzo, A., Celebre, L., Praticò, M, Zoccali, R A. and Bruno, A., 2021. Psychological Impact of Pro-Anorexia and Pro-Eating Disorder Websites on Adolescent Females: A Systematic Review. International journal of environmental research and public health, 18 (4). [accessed 10 July 2023].
1687 Beat, 2022. Best practice in ensuring early intervention for eating disorders. [accessed 10 July 2023]. 1688 Other research demonstrates that the first three years of an eating disorder are a critical window after which symptoms can become more entrenched. Source: Treasure, J., Stein, D. and Maguire, S., 2015. Has the time come for a staging model to map the course of eating disorders from high risk to severe enduring illness? An examination of the evidence, Early Intervention in Psychiatry, 9 (3). [accessed 10 July 2023]. 301
disorders can affect anyone, of any gender or age.1689 1690 Indeed, it has been estimated that 25% of people with an eating disorder are male, and a recent report by the Eating Disorder Genetics Initiative found that eating disorders are just as likely to start in adulthood as in childhood.1691 1692 1693
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Discussion forums and chat rooms, information sharing services¶
1689 Ofcom’s risk assessment for content harmful for children considers content that encourages, promotes or provides instructions for an eating disorder or behaviours associated with an eating disorder. 1690 Mento, C., Silvestri, M C., Muscatello, M R A., Rizzo, A., Celebre, L., Praticò, M, Zoccali, R A. and Bruno, A., 2021. Psychological Impact of Pro-Anorexia and Pro-Eating Disorder Websites on Adolescent Females: A Systematic Review. International journal of environmental research and public health, 18 (4). [accessed 10 July 2023]. 1691 Sweeting, H., Walker, L., MacLean, A., Patterson, C., Räisänen, U and Hunt, K., 2015. Prevalence of eating disorders in males: a review of rates reported in academic research and UK mass media, International Journal of Mens Health, 14 (2). [accessed 11 July 2023]. 1692 Wooldridge, T., Mok, C. and Chiu, S., 2014. Content analysis of male participation in pro-eating disorder web sites, Eating Disorders, 22 (2). [accessed 11 July 2023].
1693 King’s College London and Beat (Davies, H L., Kelly, J., Ayton, A., Hübel, C., Bryant-Waugh, R., Treasure, J. and Breen, G)., 2022. When Do Eating Disorders Start? An Investigation into Two Large UK Samples. [accessed 11 July 2023]. 1694 Wales Online, 2022. Coaches are training children to be anorexic with vile comments online. [accessed 10 July 2023]. 1695 Achilles, L., Mandl, T. and Womser-Hacker, C., 2022. “Meanspo Please, I Want to Lose Weight”: A Characterization Study of Meanspiration Content on Tumblr Based on Images and Texts. [accessed 10 July 2023]. 302
which can escalate into encouragement of suicidal behaviours, including sharing content that can be harmful or distressing to users.1696 Users may also specifically post to prompt other users to provide them with information detailing suicide methods or regarding how to carry out acts of serious self-harm.
1696 There are services such as some discussion forums that are dedicated to suicide or self-harm content. However, a recent inquest has revealed that this content also exists across services that actively prohibit suicide or self-harm content. Source: The Coroner’s Service, 2022. Prevention of Future Deaths. [accessed 28 October 2022]. 1697 In the UK between 2001 and 2008, there were at least 17 deaths involving chatrooms or sites that provide advice on suicide methods. Source: Cohen-Almagor, R, and Lehman-Wilzig, S., 2022. Digital Promotion of Suicide: A Platform-Level Ethical Analysis, Journal of Media Ethics, 32 (2). [accessed 10 July 2023]. One forum in particular was linked to 50 deaths in the UK, with coroners and police investigations highlighting the role it played and issuing warnings about the site. BBC, 2023. 'Failure to act' on suicide website linked to 50 UK deaths. [accessed 26 September 2024]. 1698 Royal College of Psychiatrists, 2020. Technology use and the mental health of children and young people. [accessed 10 July 2023]. For more information, see the section, ‘Risks of harm to individuals presented by the offence of encouraging or assisting suicide online’. 1699 Bell, J., Mok, K., Gardiner, E. & Pirkis, J., 2017. Suicide-related internet use among suicidal young people in the UK: Characteristics of users, effects of use, and barriers to offline help-seeking, Archives of suicide research: official journal of the International Academy for Suicide Research, 22 (4). [accessed 10 July 2023].
1700 Dunlop, S M., More, E. and Romer, D., 2011. Where do youth learn about suicides on the internet, and what influence does this have on suicidal ideation? [accessed 5 July 2023]. 1701 Biddle, L., Derges, J., Goldsmith, C., Donovan, J L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, PLoS ONE, 13 (5). [accessed 10 July 2023]. 303
Social media services¶
1702 [DISTRESSING CONTENT WARNING] Network Contagion Research Institute, 2022. Online Communities of Adolescents and Young Adults Celebrating, Glorifying, and Encouraging Self-Harm and Suicide are Growing Rapidly on Twitter. [accessed 10 October 2024] 1703 This is sometimes referred to as ‘algospeak’. 1704 Feldhege, J., Moessner, M. and Bauer, S., 2021. Detrimental Effects of Online Pro-Eating Disorder Communities on Weight Loss and Desired Weight: Longitudinal Observational Study, Journal of Medical Internet Research, 23 (10). [accessed 11 July 2023]. 1705 In a US study, adult females without history of an ED who were exposed to pro-eating disorder content for 90 minutes showed a significant decrease in their calorific intake from pre- to post-exposure, had strong emotional responses to the content and reported changes in their current eating behaviour three weeks after the study. Source: Jett, S., LaPorte, D J. and Wanchisn, J., 2010. Impact of exposure to pro-eating disorder websites on eating behaviour in college women, European Eating Disorders Review, 18 (5). [accessed 11 July 2023]. 1706 Feldhege, J., Moessner, M. and Bauer, S., 2021. Detrimental Effects of Online Pro-Eating Disorder Communities on Weight Loss and Desired Weight: Longitudinal Observational Study, Journal of Medical Internet Research, 23 (10). [accessed 11 July 2023]. 1707 Ging, D. and Garvey, S., 2018. ‘Written in these scars are the stories I can’t explain’: A content analysis of pro-ana and thinspiration image sharing on Instagram, New Media and Society, 20 (3). [accessed 11 July 2023]. 1708 Osborne,K D., 2023. Competing for perfection: a scoping review evaluating relationships between competitiveness and eating disorders or disordered eating behaviours. [accessed 8 September 2023].
1709 In particular, Instagram, as the focus of this study at the time. 1710 The study found that exposure to this content resulted in an ‘emotional disturbance’ in some users, with this exposure statistically related to ‘(possibly harmful) self-harm and suicidality-related outcomes’. Source: Arendt, F., Scherr, S. and Romer, D., 2019. Effects of exposure to self-harm on social media: Evidence from a two-wave panel study among young adults, p.3, New Media & Society, 21 (11-12). [accessed 10 July 2023]. 304
occasionally set up by users on social media services, offering users a chance to discuss topics with other users.1711
Video-sharing services¶
Risk factors: User base¶
User base size¶
1711 Marchant, A., Hawton, K., Stewart, A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K. and John, A., 2017. A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown, PLoS ONE, 12 (8). [accessed 10 July 2023]. For more information, see the section, ‘Risk factors: functionalities and recommender systems’. 1712 Biddle, L., Derges, J., Goldsmith, C., Donovan, J L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, PLoS ONE, 13 (5). [accessed 10 July 2023]. 1713 Harriger, J A., Evans, J A., Thompson, J K. and Tylka, T L., 2022. The dangers of the rabbit hole: Reflections on social media as a portal into a distorted world of edited bodies and eating disorder risk and the role of algorithms, Body Image, 41. [accessed 11 July 2023].
1714 For an explanation, see ‘livestreaming’ in Risk factors: functionalities and recommender systems section. 1715 Ekō, 2023. Suicide, Incels, and Drugs: How TikTok’s deadly algorithm harms kids. [accessed 11 July 2023]. 1716 Cooper, G., Lewis, C., 2023. 'Failure to act' on suicide website linked to 50 UK deaths, BBC, 24 October. [accessed 28 July 2024]. 305
of the forum showed that 30% of the discussions were about suicide methods and that the site has been linked to increased incidents of previously infrequent methods of suicide.1717
User base demographics¶
15.47
1717 Sartori, E. 2022. Analyzing Sanctioned Suicide: a case study on pro-choice sites. Università di Padova. [accessed 9 September 2024]. 1718 Feldhege, J., Moessner, M. and Bauer, S., 2021. Detrimental Effects of Online Pro-Eating Disorder Communities on Weight Loss and Desired Weight: Longitudinal Observational Study, Journal of Medical Internet Research, 23 (10). [accessed 11 July 2023]. 1719 Smahelova, M., Drtilova, H., Smahel, D., Cevelicek, M, 2020. Internet Usage by Women with Eating Disorders during Illness and Recovery, Health Communication 35 (5). [accessed 24 July 2023].
1720 Gale L, Channon S, Larner M, James D. 2016 Experiences of using pro-eating disorder websites: a qualitative study with service users in NHS eating disorder services. Eating and Weight Disorders, 21(3). [accessed 24 July 2023]. 1721 Rodgers, R.F., Melioli, T, 2016. The Relationship Between Body Image Concerns, Eating Disorders and Internet Use, Part I: A Review of Empirical Support. Adolescent Res Rev 1, 95–119 [accessed 24 July 2023]. 1722 Ofcom, 2024. Online Experiences Tracker 2024. 1723 Instagram, 2021, Bad Experiences and Encounters Framework (BEEF) Survey. [accessed 6 November 2024]. 306
1724 Domaradzki, J. 2021. The Werther Effect, the Papageno Effect or No Effect? A Literature Review. International Journal of Environmental Research and Public Health, 18(5). [accessed 10 October 2024]. 1725 In 2011-2018, a national confidential inquiry into suicide and homicide by people with mental illness found that 15% of under-25s (aged 10+) had reported using the internet for suicide-related purposes (e.g. visiting pro-suicide websites) during this time, which was significantly higher than for patients aged 25+ (7%). Source: University of Manchester, 2021. The National Confidential Inquiry into Suicide and Safety in Mental Health. [accessed 11 July 2023]. 1726 Samaritans and Swansea University, 2022. How social media users experience self-harm and suicide content. [accessed 10 July 2023]. 1727 Samaritans response to 2023 Ofcom Call for Evidence: Second phase of online safety regulation: Protection of Children 1728 Data from Bristol Royal Infirmary (BRI) and Bristol Royal Hospital for Children (BRHC) was analysed. Suicide and Self-Harm Related Internet Use. 1729 In this study, suicide related internet use (SRIU) is composed of the following recorded activities, percentages are of those that engaged in any SRIU activity: obtaining information on suicide methods (68.9%), visited pro-suicide websites (32.9%), communicated suicidality online (15.9%), other SRIU including purchasing means of suicide online (9.2%). Percentages do not sum to 100% because many individuals engaged in more than one activity. 1730 Padmanathan, P., Biddle, L., Carroll, R., Derges, J., Potokar, J., & Gunnell, D. (2018). Suicide and Self-Harm Related Internet Use. Crisis, 39(6), 469–478. [accessed 10 October 2024].
1731 Bojanić, L., Turnbull, P., Ibrahim, S., Flynn, S., Kapur, N., Appleby, L., Hunt, I. M. 2024. Suicide-related internet use among mental health patients who died by suicide in the UK: a national clinical survey with case-control analysis. The Lancet Regional Health – Europe. Volume 44, September. [accessed 10 October 2024]. 1732 Rodway, C., Tham, S. G., Richards, N., Ibrahim, S., Turnbull, P., Kapur, N., Appleby, L. 2023. Online harms? Suicide- related online experience: a UK-wide case series study of young people who die by suicide. Psychol Med. 2023 Jul;53(10):4434-4445. [accessed 10 October 2024]. 307
1733 Office for National Statistics, Suicides in England and Wales: 2023 registrations. [accessed 11 August 2024]. 1734 Commensurate figures for Scotland and Northern Ireland were not available. 1735 From 3.1/100,000 to 5.1/100,000. Office for National Statistics, Suicides in England and Wales: 2023 registrations. [accessed 11 August 2024]. 1736 Wilson-Lemoine et al, 2024. Bullying Victimization and Self-Harm Among Adolescents from Diverse Inner-City Schools. [accessed 01 June 2025]. 1737 Winstone et al, 2024. Cyberbullying Perpetration and Victimization as Risk Factors for Self-Harm: Results From a Longitudinal Cohort Study of 13–14-Year-Olds in England - Journal of Adolescent Health. [accessed 01 June 2025]. 1738 Thériault-Couture et al, 2025. Cybervictimisation and mental health conditions in young people: findings from a nationally representative longitudinal cohort. [accessed 02 June 2026].
1739 Ofcom’s OET refers to mental health as ‘Anxiety, depression or trauma-related conditions, for example.’ Experiences of using online services - Ofcom.
1740 Ofcom research into how people are harmed online included a case study on a male aged 26-30. He had struggled with his mental health over the lockdown period and as a result had sought information on suicide methods via a search engine, until he reached forums that discussed suicide methods. His poor mental health increased the likelihood that he would experience harm from the content. Ofcom, 2022. How people are harmed online: Testing a model from a user perspective. [accessed 11 July 2023]. 308
likelihood of suicide related internet use was higher among those diagnosed with affective, anxiety, and autism spectrum disorders, and those receiving any psychological treatment.1741 A higher proportion (21%) of autistic patients had used the internet for suicide-related purposes compared to just 7% of non-autistic patients.1742
1741 Lana B., Pauline T., Saied I., Sandra F., Navneet K., Louis A., Isabelle M. H., 2024, Suicide-related internet use among mental health patients who died by suicide in the UK: a national clinical survey with case-control analysis. [accessed 21 June 2024]. 1742 The National Confidential Inquiry into Suicide and Safety in Mental Health, 2024. Annual Report: UK patient and general population data, 2011-2021. [accessed 10 October 2024] 1743 Ofcom research into how people are harmed online included a case study on a male aged 26-30. He had struggled with his mental health over the lockdown period and as a result had sought information on suicide methods via a search engine, until he reached forums that discussed suicide methods. His poor mental health increased the likelihood that he would experience harm from the content. Ofcom, 2022. How people are harmed online: Testing a model from a user perspective. [accessed 11 July 2023].
1744 Molly Rose Foundation, 2025. MRF Report Updated 2309 PDF. [accessed 01 June 2026]. 1745 MEL Research, 2026. SPARK Report: Suicide Attempts & Suicidal Thoughts Statistics & Tracking Research. [accessed 02 June 2026]. 1746 Ofcom, 2024. Online Experiences Tracker. [accessed 22 November 2024]. 1747 Ofcom, 2024. Online Experiences Tracker 2024. [accessed 22 November 2024]. 309
data on suicides by young people (aged 10 to 19) between 2014 and 2016 found that those identifying as LGBT were 2.3 times more likely to have suicide related online experience 1748 reported as an antecedent to their suicide than non-LGBT individuals.1749 Similarly, content promoting self-harm is experienced significantly more by adult internet users who identify with ‘other’ sexuality (10%) and bisexual (10%) versus heterosexual (3%). As with suicide content, those who identified as non-binary were significantly more likely to report experiencing content promoting self-harm (17%) than males (4%) and females (3%).1750
Risk factors: Functionalities and recommender systems¶
User identification¶
Anonymous profiles¶
1748 Suicide related online experience includes: searching the internet for information on suicide method, visiting websites that may have encouraged suicide behaviour, communicating suicidal ideation or intent online and being bullied online. Rodway, C., Tham, S. G., Richards, N., Ibrahim, S., Turnbull, P., Kapur, N., Appleby, L. 2023. Online harms? Suicide-related online experience: a UK-wide case series study of young people who die by suicide. Psychol Med. 2023 Jul;53(10):4434-4445. [accessed 10 October 2024]. 1749 Female versus male (OR 1.87, 95% CI 1.23-2.85, p = 0.003), LGBT versus non-LGBT (OR 2.35, 95% CI 1.10-5.05, p = 0.028). 1750 Ofcom, 2024. 1751 Please note that these data do not include admissions to A&E. Self-harm admission rates also sharply declined, for male and female patients, in 2022/23, but this may be due to a change in NHS England’s reporting methodology, which reclassified same day emergency cases, leading to fewer admissions being recorded. Source: Nuffield Trust, 2024. Hospital admissions as a result of self-harm in children and young people. [accessed 01 October 2024] 1752 Between 2012/13 and 2021/22 the rate for females rose from 508 admissions per 100,000 to 711. For the same period the inverse trend has been observed for males as the rate of admissions decreased by 22% from 193 per 100,000 to 154. Source: Nuffield Trust, 2024. 1753 Averaging over age-standardised suicide rates for the years between 2010 and 2022. Office for National Statistics, Suicides in England and Wales: 2023 registrations. [accessed 11 August 2024].
1754 Suicide-related online experience includes: searching the internet for information on suicide method, visiting websites that may have encouraged suicide behaviour, communicating suicidal ideation or intent online and being bullied online. Rodway, C. et al. 2023. 1755 Anonymity can have benefits in helping some individuals feel more able to express themselves online, particularly users who may be experiencing thoughts of suicide or self-harm. 310
including ‘suicide baiters’ who wish to encourage the suicide of individuals expressing suicidal ideation – to commit offences online,1756 or intentionally encourage acts of self-injury with less risk of being identified and facing consequences. Analysis of German language posts depicting non-suicidal self-injury (NSSI) on Instagram in 2016 found that around 80% of the posts were posted by anonymous accounts, those displaying no personal information. 1757 Likewise, an analysis of self-harm communities on X (formerly Twitter) revealed that some accounts, which claimed to be children, demonstrated the sophisticated operational security of 'internet-savvy adults'. These accounts employed advanced techniques to avoid automatic risk-scoring and flagging mechanisms,1758 suggesting that predators are exploiting self-harm communities for the purposes of exploiting vulnerable children. This is particularly concerning given the existence of networks of accelerationist terrorist groups extorting children online to self-harm.1759
Fake user profiles¶
1756 Phillips, J G., Diesfeld, K. and Mann, L., 2019. Instances of online suicide, the law and potential solutions, Psychiatry, Psychology and Law, 26 (3). [accessed 27 January 2023]. 1757 Brown, R. C., Fisher, T., Goldwich, A. D., Keller, F., Young, R., Plener, P. L. 2018. #cutting: Non-suicidal self-injury (NSSI) on Instagram. Psychological Medicine, 48, 337-346. [accessed 10 October 2024]. 1758 [DISTRESSING CONTENT WARNING] Network Contagion Research Institute, 2022. Online Communities of Adolescents and Young Adults Celebrating, Glorifying, and Encouraging Self-Harm and Suicide are Growing Rapidly on Twitter. [accessed 10 October 2024] 1759 Winston, A. (2024). There Are Dark Corners of the Internet. Ten There’s 764. WIRED. [accessed 10 October 2024]; Argentino, M.-A., Barrett, G., Tyler, M. B., (2024). 764: The Intersection of Terrorism, Violent Extremism, and Child Sexual Exploitation. [accessed 10 October 2024]
1760 Balt, E., Mérelle, S., Robinson, J., Popma, A., Creemers, D., Brand, IVD., Bergen, DV., Rasing, S., Mulder, W. and Gilissen, R., 2023. Social media use of adolescents who died by suicide: lessons from a psychological autopsy study, Child and Adolescent Psychiatry and Mental Health, 17 (48). [accessed 11 July 2023]. 1761 Biddle, L., Derges, J., Goldsmith, C., Donovan, J L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, p.12, PLoS ONE, 13 (5). [accessed 10 July 2023]. 311
for using fake identities to encourage individuals to join bogus suicide pacts and to assist with suicides by suggesting methods for victims and survivors to use.1762
User networking¶
User groups¶
User communication¶
Posting content (text, images, videos)¶
1762 Phillips, J G., Diesfeld, K. and Mann, L., 2019. Instances of online suicide, the law and potential solutions, Psychiatry, Psychology and Law, 26 (3). [accessed 27 January 2023]. 1763 Marchant, A., Hawton, K., Stewart, A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K. and John, A., 2017. A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown, p.16, PLoS ONE, 12 (8). [accessed 10 July 2023]. 1764 Biddle, L., Derges, J., Goldsmith, C., Donovan, J L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, p.12, PLoS ONE, 13 (5). [accessed 10 July 2023].
1765 Bell, J., Mok, K., Gardiner, E. and Pirkis, J., 2017. Suicide-related internet use among suicidal young people in the UK: Characteristics of users, effects of use, and barriers to offline help-seeking, pp.11-12, Archives of suicide research: official journal of the International Academy for Suicide Research, 22 (4). [accessed 10 July 2023]. 1766 Susi, K., Glover-Ford, F., Stewart, A., Knowles Bevis, R. and Hawton, K., 2023. Research Review: Viewing self-harm images on the internet and social media platforms: systematic review of the impact and associated psychological mechanisms p.17, Journal of Child Psychology and Psychiatry, 64 (8). [accessed 10 July 2023]. 312
images of wounds and scars, self-harm memes, videos with NSSI (non-suicidal self-injury) content, suicide images from a first-person and third-person perspective, and content containing images of self-harm coupled with negative words (such as ‘suicide’ and ‘death’).1767
Commenting on content¶
1767 Susi, K., Glover-Ford, F., Stewart, A., Knowles Bevis, R. and Hawton, K., 2023. Research Review: Viewing self-harm images on the internet and social media platforms: systematic review of the impact and associated psychological mechanisms, pp.4-11, Journal of Child Psychology and Psychiatry, 64 (8). [accessed 10 July 2023]. 1768 Molly Rose Foundation and The Bright Initiative. 2023. Preventable yet pervasive: The prevalence and characteristics of harmful content, including suicide and self-harm materials, on Instagram, TikTok and Pinterest. [accessed 10 October 2024]; BBC News, 2022. Molly Russell: Social media causes no end of issues, head says. [accessed 10 October 2024]. 1769 O’Dea, B., Achilles, M R., Larsen, M E., Batterham, P J., Calear, A L. and Christensen, H., 2021. The rate of reply and nature of responses to suicide-related posts on Twitter. p.2, Internet Interventions, 13. [accessed 11 July 2023].
1770 Videos analysed took place between 2001 and 2017 and included cases from the USA, India, UK, France, Turkey, Canada, Russia, Sweden, Japan and Thailand. 1771 Phillips, J G. and Mann, L., 2019. Suicide baiting in the internet era p.1, Computers in Human Behaviour, 92. [accessed 11th July 2023]. 1772 Ekō, 2023. Suicide, Incels, and Drugs: How TikTok’s deadly algorithm harms kids p.10. [accessed 11 July 2023]. 1773 Susi, K., Glover-Ford, F., Stewart, A., Knowles Bevis, R. and Hawton, K., 2023. 313
were negative (7.35% abusive or complimentary).1774 Following this study, 59 individuals who had posted self-generated self-harm images to the Instagram accounts identified in the study discussed above were recruited for qualitative interviews.1775 While many participants reported being offered help (N=29, 39.2%), only 15.2% believed that those interactions had actually been helpful. By contrast 40.7% (N=24) of the participants reported receiving negative comments including harassment, abuse, and encouragement of suicide. Furthermore, 39% (N=29) reported feeling angry or sad upon receiving negative comments. Participants also reported receiving problematic positive reinforcement of their self-harming, with 5 reported receiving compliments including comments that their wounds were “cool” or “beautiful”. A recent analysis of communities sharing self-harm content on X found that comments were used to express admiration for severe self-injury and to provide advice on how to injure themselves more severely.1776
Reacting to content and re-posting or forwarding content¶
Direct messaging¶
1774 Brown, R. C., Fisher, T., Goldwich, A. D., Keller, F., Young, R., Plener, P. L. 2018. #cutting: Non-suicidal self-injury (NSSI) on Instagram. Psychological Medicine, 48, 337-346. [accessed 10 October 2024]. 1775 Brown, R. C., Fischer, T., Goldwich, D. A., & Plener, P. L. (2020). “I just finally wanted to belong somewhere”— Qualitative Analysis of Experiences With Posting Pictures of Self-Injury on Instagram. Frontiers in Psychiatry, 11, 274. [accessed 18 November 2024] 1776 [DISTRESSING CONTENT WARNING] Network Contagion Research Institute, 2022. Online Communities of Adolescents and Young Adults Celebrating, Glorifying, and Encouraging Self-Harm and Suicide are Growing Rapidly on Twitter. [accessed 10 October 2024] 1777 Brown, R. C., Fisher, T., Goldwich, A. D., Keller, F., Young, R., Plener, P. L. 2018.
1778 Marchant, A., Hawton, K., Burns, L., Stewart, A., & John, A. (2021). Impact of web-based sharing and viewing of self- harm–related videos and photographs on young people: Systematic review. Journal of medical Internet research, 23(3), e18048. [accessed 10 October 2024]. 1779 Biddle, L., Derges, J., Goldsmith, C., Donovan, J L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, p.12, PLoS ONE, 13 (5). [accessed 10 July 2023]. 314
Group messaging¶
specifically target online communities related to self-harm and eating disorders to find potential victims. They have set up fake support groups where victims are approached on gaming platforms, livestreams, direct messaging apps, and social networks, enabling rapid circulate with limited oversight. Within these group‑based environments, victims have been onboarding into group environments where harmful content and coercive messaging
coerced into self-harm and suicide and, in some cases, encouraged to commit acts against others, including the encouragement of suicide, reflecting the role of group dynamics in escalating harm.1787
1780 The National Crime Agency, 2025 referred to ‘Com networks’ as online forums or communities, where offenders collaborate or compete to cause harm across a broad spectrum of criminality - both on and offline - including cyber, fraud, extremism, serious violence, and child sexual abuse. 1781 National Crime Agency, 2025. Sadistic online harm groups putting people at unprecedented risk, warns the NCA -National Crime Agency 1782 Resolver, Critical Harm Intelligence Briefing: Weaponised Loneliness, 2026. Resolver Reports New Online Threat Targeting Children | Resolver 1783 Please note that research often references ‘chatrooms’ rather than ‘group messaging’. Because chatrooms are centred around enabling users to message one another in groups, we have used research on chatrooms to draw conclusions surrounding group messaging. 1784 Chen R, Wang Y, Liu L, et al. 2021. A qualitative study of how self-harm starts and continues among Chinese adolescents. BJPsych. [accessed 01 October 2024].
1785 Cohen-Almagor, R, and Lehman-Wilzig, S., 2022. Digital Promotion of Suicide: A Platform-Level Ethical Analysis, p.6, Journal of Media Ethics, 32 (2). [accessed 10 July 2023]. 1786 Biddle, L. et al, 2018. 1787 Resolver, Critical Harm Intelligence Briefing: Weaponised Loneliness, 2026. Resolver Reports New Online Threat Targeting Children | Resolver 315
and amplify harmful behaviour.1788 underscoring how group‑based communication channels are used to normalise, reinforce,
Livestreaming¶
AI Chatbots¶
1788 National Crime Agency, 2025. Sadistic online harm groups putting people at unprecedented risk, warns the NCA -National Crime Agency 1789 Cohen-Almagor, R, and Lehman-Wilzig, S., 2022. Digital Promotion of Suicide: A Platform-Level Ethical Analysis, p.2, Journal of Media Ethics, 32 (2). [accessed 10 July 2023].
1790 Shoib, S., Chandradasa, M., Nahidi, M., Amanda, T W., Khan, S., Saeed, F., Swed, S., Mazza, M., Di Nicola, M., Martinotti, G., Di Giannantonio, M., Armiya’u, A Y. and De Berardis, D., 2022. Facebook and Suicidal Behavior: User Experiences of Suicide Notes, Livestreaming, Grieving and Preventive Strategies - A Scoping Review, p.8, International Journal of Environmental Research and Public Health, 19. [accessed 11 July 2023]. 1791 Mental Health UK, 2025. Over one in three using AI Chatbots for mental health support, as charity calls for urgent safeguards. [accessed 12th February 2026]. 316
had no concerns about following advice from them, increasing to 50% for vulnerable children.1792
1792 Internet Matters, 2025. Me, Myself and AI: Understanding and safeguarding children’s use of AI chatbots. [accessed 01 June 2026]. 1793 Center for Countering Digital Hate, 2025. The Illusion of AI Safety. [accessed 12th February 2026] 1794 De Freitas et al, 2023. Chatbots and mental health: Insights into the safety of generative AI. [accessed 01 June 2026]. 1795 Weilnhammer et al, 2026. [2602.01347] Vulnerability-Amplifying Interaction Loops: a systematic failure mode in AI chatbot mental-health interactions. [accessed 01 June 2026].
1796 McBain et al, 2025. Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment | Psychiatric Services. [accessed 01 June 2026]. 1797 Schoene and Canca, 2025. ‘For Argument’s Sake, Show Me How To Harm Myself!’: Jailbreaking LLMS in Suicide and Self-harm Contexts. [accessed 26th January 2026] 317
Content exploring¶
Content tagging¶
User-generated content search filtering¶
Hyperlinking¶
15.100 Hyperlinks may contribute to the risks of harm related to suicide and self-harm content. Some studies have shown that hyperlinks can cause a ‘rabbit-hole’ effect, whereby users engage with links to similar content, leading them to more harmful content which they had not necessarily set out to view.1802 15.101 A study on suicide-related internet use found that many young adults in the sample followed links within and across different online services. The study found that this
1798 Arendt, F., Scherr, S. and Romer, D., 2019. Effects of exposure to self-harm on social media: Evidence from a two-wave panel study among young adults, p.3, New Media & Society, 21 (11-12). [accessed 10 July 2023]. 1799 Among other hashtags identified by the researchers as being commonly associated with suicide and self-harm elated content. 1800 Molly Rose Foundation and The Bright Initiative. 2023. Preventable yet pervasive: The prevalence and characteristics of harmful content, including suicide and self-harm materials, on Instagram, TikTok and Pinterest. [accessed 10 October 2024]. 1801 These participants said that by this point, they had decided that they wanted to end their life and were online to research how to action it, looking only for this type of user-generated content. Source: Biddle, L., Derges, J., Goldsmith, C., Donovan, J L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, p.11, PLoS ONE, 13 (5). [accessed 10 July 2023].
1802 Biddle, L., Derges, J., Goldsmith, C., Donovan, J L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, p.8, PLoS ONE, 13 (5). [accessed 10 July 2023]. 318
behaviour tended to increase as mood lowered, leading to an escalation in browsing and exposure to issues that the participants had not previously considered.1803
Saving Content¶
15.102 The Molly Rose foundation found that, in some cases, potentially harmful suicide and self-harm content has been saved by large numbers of users.1804 Saving suicide and self-harm related content may increase frequency of exposure, facilitate rumination, and increase the risk presented by highly depressive content that becomes harmful when viewed in large quantities over time.
Recommender systems¶
Content recommender systems¶
15.103 Recent research indicates a wide range of features/functionalities are involved when people encounter content promoting suicide and self-harm, with content feeds being the most prominent way. Among UK users aged 16+ who encountered suicide and self-harm content in the previous year, 42% encountered content while scrolling through their feed or ‘for you’ page.1805 15.104 Some evidence suggests that content recommender systems1806 can increase the risk of exposure to suicide and self-harm related content. As recommender systems are understood to maximise user engagement, they can make it more likely that users who engage with harmful content see more of it in the future. In a national survey by Swansea University and Samaritans (where 87% of the sample reported having self-harmed before), more than four in five (83%) respondents reported coming across self-harm and suicide content through feeds of recommended content on social media, despite not having searched for it.1807 Recommender systems can recommend potentially harmful suicide and self-harm-related content very soon after a new user first signs on. Researchers from the Centre for Countering Digital Hate in the USA created four ‘standard’ new accounts with a female username on TikTok for users aged 13 in the USA, the UK, Australia and Canada. Four separate accounts were created with a username that indicated a body image-related concern.1808 The researchers found that the ‘standard’ teen TikTok accounts recommended
1803 Biddle, L. et al, 2018. 1804 Molly Rose Foundation and The Bright Initiative. 2023. Preventable yet pervasive: The prevalence and characteristics of harmful content, including suicide and self-harm materials, on Instagram, TikTok and Pinterest. [accessed 10 October 2024]. The Coroner’s inquest into the death of Molly Russell revealed that large quantities of content that she had saved were depression, self-harm or suicide related. BBC, 2022. Molly Russell: Social media causes no end of issues, head says. [accessed 10 October 2024] 1805 'This was followed by reading articles/reviews (29%), watching content they chose to see (27%), seeing content in comments/replies (27%), watching content selected by autoplay (22%), seeing content in emails (21%), in advertisements (21%) and in user profiles (20%). Other features where fewer users encountered suicide and self-harm content included seeing content using the search function (18%), in a livestream (18%), and in a group chat (18%). Source: MEL Research, 2026. SPARK Report: Suicide Attempts & Suicidal Thoughts Statistics & Tracking Research. [accessed 02 June 2026]. 1806 In the context of online services, a recommender system (or a recommender engine) is a type of information retrieval and ranking system that curates content to a service user. Recommender systems are powered by a set of algorithms which, depending on what they are optimised for, set the decision path for what content is suggested to the user. The goal of a recommender system is to generate recommendations likely to engage the user, although the exact metric/goal will vary by platform.
1807 Samaritans and Swansea University, 2022. How social media users experience self-harm and suicide content. p.4. [accessed 10 July 2023]. 1808 Across all accounts, researchers expressed an interest in body image, mental health and eating disorders by watching and liking relevant videos. 319
self-harm, suicide and eating disorder content within minutes of scrolling the ‘for you’ feed. Suicide content appeared within the first 2.6 minutes.1809 The limited sample of this study means that we cannot be confident that the findings are representative of a realistic user experience on the platform. Nonetheless, this research does demonstrate that it is possible for this content to be served up within a very short period of the first viewing session.
1809 Centre for Countering Digital Hate, 2022. Deadly By Design: TikTok pushes harmful content promoting eating disorders and self-harm into users’ feeds p.19. [accessed 11 July 2023]. 1810 University of Sheffield (Dr Ysabel Gerrard), How we’re helping social media companies remove harmful content and protect their users. [accessed 10 January 2023]. 1811 The Coroner’s Service, 2022. Prevention of Future Deaths. [accessed 28 October 2022]. 1812 Molly Rose Foundation and The Bright Initiative. 2023. Preventable yet pervasive: The prevalence and characteristics of harmful content, including suicide and self-harm materials, on Instagram, TikTok and Pinterest. [accessed 10 October 2024].
1813 Molly Rose Foundation and The Bright Initiative. 2023. 1814 Molly Rose Foundation and The Bright Initiative, 2023. This was a limited study, and as such its findings are not representative of a user experience on the platform. Nonetheless, it does demonstrate that it is possible for a very high volume of potentially harmful content to be served up to a user. 320
Risk factors: Business models and commercial profiles¶
15.109 There is some evidence to suggest that advertising-based revenue models may be a risk factor for suicide and self-harm content. In its 2023 Protection of Children Call for Evidence (CFE) response, the Molly Rose Foundation noted that email and push notifications can direct children to suicide and self-harm content. These are sent to users to encourage continued engagement with a service provider to drive up advertising revenue, increasing the risk by encouraging a user to revisit potentially harmful recommended content that the user may have previously engaged with.1815 Some evidence suggests that there are instances where this revenue model can suggest further suicide and self-harm content to an online user.1816
1815 Molly Rose Foundation response to 2023 Ofcom Call for Evidence. 1816 One example provided was an email sent to Molly Russell before she took her own life. The Call For Evidence response states that this email contained images of self-harm (some of a graphic nature), suicide (including methods) and depression. Source: Molly Rose Foundation response to 2023 Ofcom Call for Evidence. 321
Section 16 Foreign interference offence¶
Summary analysis for the foreign interference offence: how harm manifests online and risk factors The new Foreign Interference Offence (FIO) has been designed to tackle malign activity carried out for, or on behalf of, or intended to benefit, a foreign power. Prohibited conduct captured by this offence will include where there is a misrepresentation of a person’s identity or purpose, or in the presentation of the information, for example, through state-backed disinformation campaigns. In introducing this new offence, the UK Government has explained that: “Foreign interference is intended to sow discord, manipulate public discourse, discredit the political system, bias the development of policy, and undermine the safety or interests of the UK”. Harm that can arise from this offence is wider than the individual and can affect societies as a whole. For example, a foreign state could seek to manipulate whether or how someone participates in an electoral event through state-sponsored disinformation campaigns. This would have implications on the country’s electoral outcomes, undermining the integrity of elections and creating mistrust in online information. The rapid pace of development of generative AI models and technology has been recognised as posing a risk which could be exploited by those engaging in foreign interference. Generative AI models and technology present the opportunity for perpetrators to create an increased volume of foreign interference content, with increased quality and personalisation of content for targeted audiences, as well as reducing costs and barriers to entry for perpetrators of foreign interference campaigns. At present, it appears that generative AI technologies are more likely to significantly exacerbate existing risks of foreign interference and other information-based threats, rather than present wholly new risks. Service type risk factors: There is a particular risk of FIOs happening on social media services, where perpetrators of the offence can create fake profiles which can be manipulated by bots. There is evidence of FIO and influence operations1817 occurring across many different service types, using different tactics. These services include information-
1817 The Carnegie Endowment for International Peace defines influence operations as “organized attempts to achieve a specific effect among a target audience. Such operations encompass a variety of actors—ranging from advertisers to 322
sharing services, discussion forums and chat rooms, and private messaging services. While we know more about how these operations are carried out on certain services, this is not necessarily an accurate reflection of the presence of the harm. More attention and resources have likely been devoted to studying influence operations on some services than on others due to availability of data for study. User base risk factors: Foreign influence operations have previously targeted personal characteristics such as race, religion, sexuality, and gender. Foreign interference operations often exploit and build on narratives that are common in society, including narratives that negatively depict, or target people based on their personal characteristics. For example, previous influence operations have targeted female politicians to spread and amplify gendered narratives and expectations that undermine them, increasing the risks of harm to women from foreign influence operations that amplify these pre-existing narratives. Evidence also suggests that diaspora groups and those who hold intersectional identities may be disproportionately at risk of harm from foreign influence operations. Functionalities and recommender systems risk factors: Some functionalities might increase the likelihood that influence operations will be encountered by users, thereby increasing the risk of harm. The ability to create fake user profiles can be exploited by perpetrators of foreign interference operations – both to disseminate content and to impersonate authoritative and high-profile sources. The use of coordinated networks on social media accounts can also be used to amplify content and spread narratives across services. The functionality of user connections is therefore a risk factor for this offence. Services where users can more easily share this content, both within and across services, are particularly risky. This is because they enable foreign influence operations to spread between services and other online spaces, thereby broadening their effect. These functionalities include re-posting and forwarding content, encrypted messaging, and mechanisms for sharing information across services, such as the use of hyperlinks. Posting from anonymous user profiles can also be used in foreign interference operations and to spread disinformation on services, as well as the ability to post content, especially types of content that combine images or videos and text. Service recommender algorithms can also increase the risks of harm from foreign influence, as they tend to amplify content with high user engagement. Potential perpetrators can therefore manipulate these algorithms to spread harmful content
activists to opportunists—that employ a diverse set of tactics, techniques, and procedures to affect a target’s decision making, beliefs, and opinions”. Source: Carnegie Endowment for International Peace (Thomas, E., Thompson, N., and Wanless, A.), 2020. The Challenges of Countering Influence Operations. [accessed 11 September 2023]. 323
more widely by reposting selected content or coordinating the mass sharing of harmful content. This also allows bad actors to increase user exposure to foreign interference content for the intended purpose of manipulating or misleading users. Business model risk factors: Services which raise income through advertising may be exploited by potential perpetrators who can use advertisements as an opportunity to spread foreign interference content. This will be more effective if it allows them to target specific segments of the population with their adverts, without identifying the funder of the advertising.
Introduction¶
1818 The FIO is a conduct-based offence. The forms of conduct that this offence can fall under are varied and diverse – they count as part of committing the offence if they meet the three conditions required for the offence to be present. This conduct can include diverse tactics, including creating an account on a social media platform impersonating a British politician to post content in support of the interests of a particular nation state, and posting memes with deliberate and strategic intent to sway public opinion in the UK on behalf of another, hostile nation.
1819 Carnegie Endowment for International Peace (Thomas, E., Thompson, N., and Wanless, A.), 2020. The Challenges of Countering Influence Operations. [accessed 11 September 2023]. 324
alleged funding of a Tennessee-based online content creation company by Russian operatives to create and distribute content to American audiences with hidden Russian government messaging as part of an influence operation shows how local influencers and content creators can be used to obfuscate the role of state actors in influence operations.1820 In addition, conflicting motives have made attribution more difficult; for example, when influence campaigns covertly carried out by state-linked operatives generate significant financial gains for the perpetrators, platforms may focus on the commercial aspects and miss the state coordination behind the activity.1821 Recent research has also revealed evidence of inter-state coordination within state-backed information operations, which may further complicate attribution attempts.1822
Relevant offences¶
1820 US Department of Justice, 2024. Two RT Employees Indited for Covertly Funding and Directing U.S. Company that Published Thousands of Videos in Furtherance of Russian Interests. [accessed 18 October 2024]. 1821 Carnegie Endowment for International Peace (Thomas, E., Thompson, N., and Wanless, A.), 2020. The Challenges of Countering Influence Operations. [accessed 11 September 2023]. 1822 Wang, X., Li, J., Srivatsavaya, E. and Rajtmajer, S., 2023. Evidence of inter-state coordination amongst state-backed information operations, Scientific Reports, 13, 7716. [accessed 26 September 2023]. 1823 The Internet Research Agency is a Russian organisation based in St Petersburg that was funded by Yevgeniy Viktorovich Prigozhin and companies he controlled, which conducted social media operations targeted at large US audiences with the goal of sowing discord in the US political system. Source: Mueller, R. S., 2019. Report on the investigation into Russian interference in the 2016 Presidential Election, Volumes I & II. US Department of Justice Publications and Materials. 47. [accessed 11 September 2023]. Following the death of Prigozhin in 2023, the future and ownership of the Internet Research Agency is unclear. In June 2023, before his death but after his attempted mutiny against the Russian state, Russian media reported that the agency had been disbanded. Source: Reuters, 2023. Prigozhin-controlled Russian media group shuts after mutiny, 2 July. [accessed 16 May 2024]. However, in March 2024, Google Cloud’s Mandiant reported that the infrastructure for covert information operation threat activity from Prigozhin-associated entities remained viable for use: Mandiant, 2024. Life After Death? IO Campaigns Linked to Notorious Russian Businessman Prigozhin Persist After His Political Downfall and Death. [accessed 16 May 2024].
1824 Francois, C. and Douek, E., 2021. The Accidental Origins, Underappreciated Limits, and Enduring Promises of Platform Transparency Reporting About Information Operations, Journal of Online Trust and Safety, pp.1-30. [accessed 27 September 2023]. 325
interfere with the exercise of public functions or prejudice the safety or interests of the UK.1825
1825 Sections 13 and 14 of the National Security Act 2023. 1826 Section 15 of the National Security Act 2023. 1827 During his annual threat update in November 2022, MI5 Director General Ken McCallum highlighted that Russia’s covert actions targeting the UK include disinformation and democratic interference, and he highlighted ongoing threats to Chinese diaspora members and Iranian dissidents made by their respective regimes in the UK. Source: McCallum, K., 2022. Annual Threat Update. [accessed 11 September 2023].
1828 An example provided on where this might be needed is when an individual is threatened because of their views on a foreign power’s foreign policy. Source: House of Lords. National Security Bill (parliament.uk) 1829Home Office, 2023. Foreign interference: National Security Act factsheet. [accessed 25 January 2023]. 1830 In the Explanatory Notes for the National Security Act, the UK Government has provided the following example of how the offence may include online conduct. A foreign power runs a covert unit of state actors operating a troll farm, an 326
elections, referendums and health emergencies, among others. For example, the interference operation run by Russia’s Internet Research Agency during the 2016 US Presidential election, is by far the most studied and from which we draw on in this chapter.
How foreign interference manifests online¶
organisation employing people to make deliberately offensive or provocative posts online to manipulate public opinion or cause conflicts via a variety of different tools. The troll farm uses coordinated inauthentic behaviour and online manipulation to create and amplify disinformation on the efficacy and alleged side effects of vaccines for children and uses misrepresentations and false identities to infiltrate legitimate debates on the topic. Through these actions, the foreign power aims to undermine the use of public health services by amplifying an existing ‘wedge’ issue to disrupt social cohesion. Source: Home Office, 2022. National Security Bill: Explanatory Notes. [accessed 27 September 2023]. 1831 Wendling, M., 2019. General election 2019: Reddit says UK-US trade talks document leak 'linked to Russia, BBC News, 7 December. [accessed 27 September 2023]. 1832 Graphika (Nimmo, B.), 2019. ‘UK Trade Leaks‘. [accessed 12 September 2023]. 1833 National Cyber Security Centre, 2023. UK and allies expose Russian intelligence services for cyber campaign of attempted political interference. [accessed 9 July 2024]. 1834 Global Engagement Centre, 2022. GEC Special Report: The Kremlin’s Chemical Weapons Disinformation Campaigns. [accessed 12 September 2023]. 1835 Schliebs, M., Bailey, H., Bright, J. and Howard, P. N., 2021. People's Republic of China’s Inauthentic UK Twitter Diplomacy: A Coordinated Network Amplifying PRC Diplomats, Programme on Democracy & Technology. [accessed 11 September 2023].
1836 Foreign, Development and Commonwealth Office, 2022. UK Exposes Sick Russian Troll Factory plaguing Social Media with Kremlin Propaganda. [accessed 27 September 2023]. 1837 Martin, D. A., Shapiro, J. N. and Ilhardt, J., 2020. Trends in Online Influence Efforts, Empirical Studies of Conflict Project, 2. [accessed 27 September 2023]. 1838 McCallum, K., 2024. Director General Ken MacCallum gives latest threat update. [accessed 17 October 2024]. 327
conduct intended to fall within the scope of the FIO, more details of which are discussed in the Risk Factors: User Base section.
1839 Home Office, 2023. National Security Act: Explanatory Notes. [accessed 13 August 2024] 1840 World Economic Forum, 2024. Global Risks Report 2024. [accessed 16 May 2024]. 1841 Brookings Institute (Brandt, J.), 2023. Propaganda, foreign interference and generative AI. [accessed 16 May 2024] 1842 Simon, F. M., Altay, S., and Mercier, H., 2023. Misinformation reloaded? Fears about the impact of generative AI on misinformation are overblown, Harvard Kennedy School Misinformation Review, 3 (1). [accessed 16 May 2024]. 328
• Lower costs and reduced barriers to entry for perpetrators of foreign interference campaigns
1843 Brookings Institute (Brandt, J.), 2023. Propaganda, foreign interference and generative AI. [accessed 16 May 2024]. 1844 Spitale, G., Biller-Andorno, N. and Germani, F., 2023. AI model GPT-3 (dis)informs us better than humans, Science Advances, 9 (26). [accessed 21 October 2024]. 1845 Martin, D. A., Shapiro, J. N. and Ilhardt, J., 2020. Trends in Online Influence Efforts, Empirical Studies of Conflict Project, 2. [accessed 27 September 2023]. 1846 Carnigie Endowment For International Peace (Thomas, E., Thompson, N. and Wanless, A.), 2020. The Challenges of Countering Influence Operations. [accessed 27 September 2023].
1847 Martin, D. A., Shapiro, J. N. and Ilhardt, J., 2020. Trends in Online Influence Efforts, Empirical Studies of Conflict Project, 2. [accessed 27 September 2023]. 1848 Threat Analysis Group (Butler, Z. and Taege, J.), 2023. Over 50,000 instances of DRAGONBRIDGE activity disrupted in 2022. [accessed 27 September 2023]. 1849 ‘Bots’ is an umbrella term that refers to a software application or automated tool that has been programmed by a person to carry out a specific or predefined task without any human intervention. 329
convincing, and often increasingly personalised, information designed to polarise or mislead.1850 1851
Risks of harm to individuals presented by the foreign interference offence¶
1850 Brookings Institute (Brandt, J.), 2023. Propaganda, foreign interference and generative AI. [accessed 16 May 2024] 1851 Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M., and Sedova, K., 2023. Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations, Georgetown University’s Center for Security and Emerging Technology, OpenAI and Stanford Internet Observatory. [accessed 21 October 2024]. 1852 ‘Like farming’ refers to the use of fake pages on social media services designed to artificially increase the popularity of a page, so it can be sold to buyers seeking accounts with large followings or for scam and fraud activity. 1853 ‘Click farming’ refers to the practice of manually clicking on online adverts to increase the clickthrough rate value, boost engagement metrics, and inflate impressions. This activity can be carried out by bot accounts, as discussed here, or by large groups of workers. 1854 ‘Hashtag hijacking’ refers to the use of a hashtag for a purpose other than it was created – such as tagging a message containing undesirable or harmful content with a popular, but unrelated, hashtag to surface this content to a target audience. 1855 ‘Trend jacking’ refers to when influencers, brands or organisations insert themselves into conversations online that are gaining a lot of attention – for example, by using associated hashtags or trending audios. 1856 US Department of Homeland Security, 2018. Social Media Bots Overview. [accessed 13 September 2023]. 1857 Programme on Democracy & Technology (Bradshaw, S., Bailey, H. and Howard, P. N.), 2021. Industrialized Disinformation: 2020 Global Inventory of Organized Social Media Manipulation. [accessed 13 September 2023].
1858 It is unclear whether some of these bots employ GenAI technologies, but we think that GenAI bots could be used in a similar manner. 1859 Carnegie Endowment For International Peace (Bateman, J., Hickok, E., Courchesne, L.,Thange, I. and Shapiro, J.), 2021. Measuring the Effects of Influence Operations: Key Findings and Gaps From Empirical Research. [accessed 27 September 2023]. 330
Committee’s Russia report notes that the aims of these campaigns can include creating an environment of distrust, casting doubt on the true account of events, fomenting political extremism and ‘wedge issues’, and generally discrediting ‘the West’.1860 In its report on foreign interference in EU democratic processes, the Authority for European Political Parties and European Political Foundations notes that the perception of electoral outcomes being influenced by foreign actors – regardless of effectiveness of any actual interference – erodes public trust in the democratic process of a targeted country, and subsequently undermines the credibility of elected representatives.1861
1860 Intelligence and Security Committee of Parliament, 2020. Russia. [accessed 16 May 2024]. 1861 Authority for European Political Parties and European Political Foundations, 2023. Foreign Electoral Interference Affecting EU Democratic Processes. [accessed 16 May 2024]. 1862 Meta (Gleicher, N.), 2020. Removing Coordinated Inauthentic Behaviour. [accessed 12 July 2024].
1863 Wilson Center (Jankowicz, N., Hunchak, J., Pavliuc, A., Davies, C., Pierson, S., and Kaufmann, Z.), 2021. Malign Creativity: How Gender, Sex and Lies are Weaponised Against Women Online. [accessed 16 May 2024]. 1864 Home Office, 2023. National Security Act: Explanatory Notes. [accessed 13 August 2024]. 1865 Allington, D., McAndrew, S., Moxham-Hall, V., and Duffy, B., 2021. Coronavirus conspiracy suspicions, general vaccine attitudes, trust and coronavirus information source as predictors of vaccine hesitancy among UK residents during the Covid-19 pandemic, Psychological Medicine, 53, p. 236-247. [accessed 21 October 2024]. 331
b) Deepfakes that defraud: by misrepresenting someone else’s identity. a) Deepfakes that disinform: by spreading falsehoods widely across the internet, to influence opinion on key political or societal issues, such as elections, wars, religion, or health.1866
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services¶
Information-sharing services¶
1866 Ofcom, 2024. A deep dive into deepfakes that demean, defraud and disinform. [accessed 06 September 2024]. 1867 British Foreign Policy Group (Gaston, S. and Aspinall, E.), 2021. UK Public Opinion on Foreign Policy and Global Affairs: Annual Survey 2021. [accessed 27 September 2023]. 1868 While our evidence only names discussion forums, we expect a similar risk of harm to arise from chat room services due to similarities in the characteristics typically found on these service types.
1869 See ‘Risk factors: functionalities and recommender systems’ section for more information. Source: Kirchgaessner, S., Ganguly, M., Pegg, D., Cadwalladr, C. and Burke, J., 2023. Revealed: the hacking and disinformation team meddling in elections, The Guardian, 15 February. [accessed 15 February 2023]. 1870 See ‘Risk factors: functionalities and recommender systems’ section for more information. Source: Foreign, Development and Commonwealth Office, 2022. UK Exposes Sick Russian Troll Factory plaguing Social Media with Kremlin Propaganda. [accessed 27 September 2023]. 332
consistent with Russian state-sponsored information warfare.1871 The Wikimedia Foundation has also banned several editors linked to a group from the People's Republic of China.1872
Discussion forums and chat rooms¶
Messaging services¶
Risk factors: User base¶
User base demographics¶
1871 Institute for Strategic Dialogue and CASM Technology (Miller, C., Smith, M., Marsh, O., Balint, K., Inskip, C. and Visser, F.), 2022. Information Warfare and Wikipedia. [accessed 27 September 2023]. 1872 Institute for Strategic Dialogue and CASM Technology (Miller, C., Smith, M., Marsh, O., Balint, K., Inskip, C. and Visser, F.), 2022. Information Warfare and Wikipedia. [accessed 27 September 2023]. 1873 See ‘Risk factors: functionalities and recommender systems’ section for more information. Source: RAND Corporation (Cohen, R. S., Beauchamp-Mustafaga, N., Cheravitch, J., Demus, A., Harold, S. W., Hornung, J.W., Jun, J., Schwille, M., Tryger, E. and Vest, N.), 2021. Combatting Foreign Disinformation on Social Media: Study Overview and Conclusions. [accessed 27 September 2023]. 1874 RAND Corporation (Cohen, R. S., Beauchamp-Mustafaga, N., Cheravitch, J., Demus, A., Harold, S. W., Hornung, J.W., Jun, J., Schwille, M., Tryger, E. and Vest, N.), 2021. Combatting Foreign Disinformation on Social Media: Study Overview and Conclusions. [accessed 27 September 2023]. 1875 Carnegie Endowment for International Peace (Goodwin, C. and Jackson, D.), 2022. Partnership for Countering Influence Operations, Carnegie Endowment for International Peace. Global Perspectives on Influence Operations Investigations: Shared Challenges, Unequal Resources. [accessed 27 September 2023].
1876 Nguyễn, S., Kuo, R., Reddi, M., Li, L. and Moran, R. E., 2022. Studying Mis- and Disinformation in Asian Diasporic Communities: The Need for Critical Transnational Research Beyond Anglocentrism, Harvard Kennedy School Misinformation Review, Volume 3(2). [accessed 27 September 2023]. 1877 Carnegie Endowment for International Peace (Goodwin, C. and Jackson, D.), 2022. 333
1878 Demos (Judson, E., Atay, A., Krasodomski-Jones, A., Lasko-Skinner, R. and Smith, J.), 2020. Engendering Hate: The Contours of State-Aligned Gendered Disinformation Online. [accessed 6 March 2023]. 1879 Di Meco, L. and Wilfore, K., 2021. Gendered disinformation is a national security problem, Brookings Institute, 8 March. [accessed 6 March 2023]. 1880 For example, Ukrainian MP Svitlana Zalishchuk was targeted after her speech to the United Nations on the effects of the war with Russia. The campaign included a screenshot of a falsified tweet claiming she had promised to run through Kyiv naked if the Ukrainian army lost an important battle, accompanied with fake images of her naked. Zalishchuk has suggested that the campaign originated in Russia, as it began during a period of high tension between Russia and Ukraine, and the fake claims and images first appeared on pro-Kremlin platforms. Source: HM Government Stabilisation Unit, 2020. Quick-read guide: gender and countering disinformation. [accessed 6 March 2023]. 1881 HM Government Stabilisation Unit, 2020. Quick-read guide: gender and countering disinformation. [accessed 6 March 2023].
1882 Freedom House (Datt, A. and Dunning, S.), 2022. Beijing’s Global Media Influence 2022. [accessed 17 February 2023]. 1883 Hope, C., 2021. Exclusive: Uighurs harassed and abused by Beijing in UK, minister admits, The Telegraph, 13 March. [accessed 17 February 2023]. 1884 Recorded Future, 2023. Obfuscation and AI Content in the Russian Influence Network “Doppelganger” Signals Evolving Tactics. [accessed 16 May 2024]. 334
jurisdictions, including in the UK, alongside increased online anti-LGBTQ+ conversations1885, suggesting that foreign interference operations advancing anti-LGBTQ+ narratives may have the potential to present a risk of offline harm to LGBTQ+ individuals.
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles¶
Fake user profiles¶
1885 Institute for Strategic Dialogue (Squirrel, T. and Davey, J.), 2023. A Year of Hate: Understanding Threats and Harassment Targeting Drag Shows and the LGBTQ+ Community. [accessed 16 May 2024]. 1886 Network Contagion Research Institute, 2022. Quantitative Methods for Investigating Anti-Hindu Disinformation. [accessed 16 May 2024].
1887 Wilson Center (Jankowicz, N., Hunchak, J., Pavliuc, A., Davies, C., Pierson, S., and Kaufmann, Z.), 2021. Malign Creativity: How Gender, Sex and Lies are Weaponised Against Women Online. [accessed 16 May 2024]. 1888 Graphika, 2020. Step into my Parler. [accessed 22 September 2023]. 1889 Kirchgaessner, S., Ganguly, M., Pegg, D., Cadwalladr, C. and Burke, J., 2023. Revealed: the hacking and disinformation team meddling in elections, The Guardian, 15 February. [accessed 15 February 2023]. 335
According to those reports, the operatives also sold a product which enables the simple creation of fake accounts on several U2U services, which they claim to have sold to unnamed intelligence agencies, political parties and corporate clients.1890 Journalists also found evidence that campaigns using this product had previously targeted the UK, as well as other countries.1891
1890 Kirchgaessner, S., Ganguly, M., Pegg, D., Cadwalladr, C. and Burke, J., 2023. 1891 Ganguly, M., 2023. ’Aims’: the software for hire that can control 30,000 fake online profiles, The Guardian, 15 February. [accessed 15 February 2023]. 1892 TikTok, 2022. Community Guidelines Enforcement Report. accessed 15 February 2023]. 1893 REPORT European Parliament (Kalniete, S.), 2022. REPORT on foreign interference in all democratic processes in the European Union, including disinformation. [accessed 27 September 2023].
1894 Facebook, 2021. January 2021 Coordinated Inauthentic Behaviour Report. [accessed 28 June 2023]. 1895 Meta, 2022. Removing Coordinated Inauthentic Behavior From People's Republic of China and Russia. [accessed 27 September 2023]. 1896 Recorded Future, 2023. Obfuscation and AI Content in the Russian Influence Network “Doppelganger” Signals Evolving Tactics. [accessed 16 May 2024]. 336
Utilising a similar methodology, Recorded Future found another Russian state actor-linked campaign, ‘CopyCop’, targeting audiences in the US, UK and France with content plagiarised from mainstream media outlets via generative AI, and weaponised by introducing partisan bias to coverage of domestic news in targeted countries, alongside pro-Russian coverage of the war in Ukraine and coverage of the Israel-Hamas conflict that is critical of Israeli military operations in Gaza.1897
Anonymous user profiles¶
User networking¶
User connections¶
1897 Recorded Future, 2024. Russia-linked CopyCop Uses LLMs to Weaponise Influence Content at Scale. [accessed 16 May 2024]. 1898 Schliebs, M., Bailey, H., Bright, J. and Howard, P. N., 2021. People's Republic of China’s Inauthentic UK Twitter. Diplomacy: A Coordinated Network Amplifying PRC Diplomats, Programme on Democracy & Technology. [accessed 27 September 2023].
1899 In such cases, the identity of users may also be unknown to services. 1900 Glaser, A., 2017. Macron’s French presidential campaign has been hacked less than 48 hours before the election, Vox, 6 May. [accessed 22 September 2023]. 1901 Schliebs, M., Bailey, H., Bright, J. and Howard, P. N., 2021. People's Republic of China’s Inauthentic UK Twitter Diplomacy: A Coordinated Network Amplifying PRC Diplomats, Programme on Democracy & Technology. [accessed 27 September 2023]. 337
that amplified and engaged with UK-based diplomats from the People's Republic of China. Many of the accounts in this network followed, and focused on, these diplomats, with the sole aim of raising their profile in the UK.1902
User groups¶
advantage. For example, in 2016, Russia’s Internet Research Agency used Facebook groups to organise a protest and counter-protest in Houston, Texas to create division and tension within the community.1904
User tagging¶
User communication¶
Direct messaging, group messaging, encrypted messaging¶
1902 Schliebs, M., Bailey, H., Bright, J. and Howard, P. N., 2021. 1903 Thiel, D. and McCain, M. 2022. Gabufacuturing Dissent: An in-depth analysis of Gab, Stanford Cyber Policy Review. [accessed 27 September 2023]. 1904 Franceschi-Bicchierai, L., 2017. Russian Facebook Trolls Got Two Groups of People to Protest Each Other in Texas, Vice, 1 November. [accessed 17 February 2023].
1905 Meta, 2022. January 2022 Coordinated Inauthentic Behaviour Report. [accessed 17 February 2023]. 1906 Meta (Nimmo, B., Agranovich, D. and Gleicher, N.), 2022. Adversarial Threat Report. [accessed 17 February 2023]. 1907 Graphika (Nimmo, B.), 2019. UK Trade Leaks. [accessed 27 September 2023]. 1908 Wendling, M., 2019. General election 2019: Reddit says UK-US trade talks document leak 'linked to Russia, BBC News, 7 December. [accessed 27 September 2023]. 1909 Graphika (Nimmo, B.), 2019. 338
create rumours and place fabricated content, spreading from these encrypted spaces to closed and semi-closed networks, to conspiracy communities, then mainstream social media, to finally end up being reported on in the mainstream media.1910 Further evidence demonstrated that encrypted applications lack the conventional fact-checking and content moderation that is offered on other services, thereby offering a unique opportunity to those wishing to easily spread disinformation.1911
Reacting to content¶
Posting content¶
1910 Wardle, C., 2018: 5 Lessons for Reporting in an Age of Disinformation, First Draft News, 27 December. [accessed 21 September 2023]. 1911 Gurksy, J., Riedl, M. J. and Woolley, S., 2021. The Disinformation Threat to Diaspora Communities in Encrypted Chat Apps, Brookings Institute, 19 March. [accessed 27 September 2023]. 1912 Koval, I., 2021. “How social media is manipulated — and how Russia is involved”, DW, 14 April, [accessed 22 June 2023].
1913 Hameleers, M., Powell, T. E., Van Der Meer, T. G.L.A. and Bos, L., 2020. “A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated on Social Media”, Political Communication, 37(2), p.281-301. [accessed 27 September 2023]. 1914 DiResta, R., Shaffer, K., Ruppel, B., Sullivan, D., Matney, R., Fox, R., Albright, J. and Johnson, B., 2019. The Tactics and Tropes of the Internet Research Agency, New Knowledge. [accessed 27 September 2023]. 339
leading liberal politician, about vote-rigging and other controversial issues, appeared on social media and video-sharing platforms.1915
1915 Council for Media Services, 2024. Monitoring of platform functionalities in relation to the 2023 Elections to the National Council of the Slovak Republic. [accessed 16 May 2024] 1916 Full Fact, 2023. No evidence that audio clip of Keir Starmer supposedly swearing at staff is genuine. [accessed 16 May 2024]. 1917 Resemble.ai, 2023. Political Deepfake: Keir Starmer. [accessed 16 May 2024] 1918 Spring, A., 2024. Sadiq Khan says fake AI audio of him nearly led to serious disorder, BBC News, 13 February. [accessed 26 September 2024].
1919 Fenimore Harper Communications (Beard, M.), 2024. Over 100 Deep-Faked Rishi Sunak Ads Found on Meta’s Advertising Platform. [accessed 16 May 2024]. 1920 Recorded Future, 2023. Obfuscation and AI Content in the Russian Influence Network “Doppelganger” Signals Evolving Tactics. [accessed 16 May 2024]. 1921 Recorded Future, 2024. Russia-linked CopyCop Uses LLMs to Weaponise Influence Content at Scale. [accessed 16 May 2024]. 340
Re-posting or forwarding content¶
Content exploring¶
Hyperlinking¶
1922 Gurksy, J., Riedl, M. J. and Woolley, S., 2021. The Disinformation Threat to Diaspora Communities in Encrypted Chat Apps, Brookings Institute, 19 March. [accessed 27 September 2023]. 1923 WhatsApp, n.d. About forwarding limits | WhatsApp Help Center. [accessed 27 September 2023]. 1924 Schliebs, M., Bailey, H., Bright, J. and Howard, P. N., 2021. People's Republic of China’s Inauthentic UK Twitter Diplomacy: A Coordinated Network Amplifying PRC Diplomats, Programme on Democracy & Technology. [accessed 27 September 2023].
79 Institute for Strategic Dialogue (Thomas, E.), 2022. Project Nemesis, Doxxing and the New Frontier of Informational Warfare. [accessed 27 September 2023]. 1926 DiResta, R., Shaffer, K., Ruppel, B., Sullivan, D., Matney, R., Fox, R., Albright, J. and Johnson, B., 2019. The Tactics and Tropes of the Internet Research Agency, New Knowledge. [accessed 27 September 2023]. 1927 Recorded Future, 2023. Obfuscation and AI Content in the Russian Influence Network “Doppelganger” Signals Evolving Tactics. [accessed 16 May 2024] 341
content gained illicitly through hack and leak operations, and by amplifying doxxing campaigns1928.
Content editing¶
Editing visual media¶
1928 Institute for Strategic Dialogue (Thomas, E.), 2022. Project Nemesis, Doxxing and the New Frontier of Informational Warfare. [accessed 27 September 2023]. 1929 Thiel, D. and McCain, M., 2022. Gabufacuturing Dissent: An in-depth analysis of Gab, Stanford Cyber Policy Review. [accessed 27 September 2023]. 1930 Wendling, M., 2019. General election 2019: Reddit says UK-US trade talks document leak ’linked to Russia’, BBC, 7 December. [accessed 27 September 2023]. 1931 Institute for Strategic Dialogue, 2022. Tales From the Underside: A Kremlin-Approved Hack, Leak & Doxxing Operation. [accessed 27 September 2023]. 1932 Cheap-fakes are videos that use conventional video editing techniques like speeding, slowing, cutting, restaging or re-contextualising video footage 1933 Deepfakes are a specific type of media that involves the use of AI algorithms, particularly generative AI models, to modify videos, images or audio to create realistic synthetic content. This is often done by superimposing the face of a person onto the body of another person in a video or image as well as voice manipulation with lip syncing. Deepfakes are shared as user generated content on user-to-user services but could also potentially be created using functionalities present on user-to-user services. Deepfake technology is currently used to create content that can be harmful; however, we acknowledge that it may also have positive use cases. 1934 Donovan, J. and Paris, B., 2019. Deepfakes and Cheap Fakes: The Manipulation of Audio and Visual Evidence, Data & Society. [accessed 27 September 2023].
1935 Hameleers, M., Powell, T. E., Van Der Meer, T. G.L.A. and Bos, L., 2020. A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated on Social Media, Political Communication, 37(2), pp.281-301. [accessed 27 September 2023]. 1936 Donovan, J. and Paris, B., 2019. Deepfakes and Cheap Fakes: The Manipulation of Audio and Visual Evidence, Data & Society. [accessed 27 September 2023]. 342
services.1937 Although the video has not been specifically attributed to a state actor, it is widely believed that it was generated by Russia.
Editing posted content¶
1937 Wakefield, J., 2022. Deepfake presidents used in Russia-Ukraine war, BBC News, 18 March. [accessed 6 March 2023]. 1938 Graphika, 2023. Deepfake It Till You Make It. [accessed 17 February 2023]. 1939 Thurston, J, 2023. Russia deepfake video mocks Rishi Sunak and Joe Biden, The Times, 15 June. [accessed 27 September 2023]. 1940 Robson, K., 2022. Will Twitter’s edit button help spread more fake news?, Verdict, 2 September 2022. [accessed 27 September 2023].
1941 Institute for Strategic Dialogue and CASM Technology (Miller, C., Smith, M., Marsh, O., Balint, K., Inskip, C, and Visser, F.), 2022. Information Warfare and Wikipedia. [accessed 27 September 2023]. 1942 Institute for Strategic Dialogue and CASM Technology, 2022. 1943 Institute for Strategic Dialogue and CASM Technology (Miller, C., Smith, M., Marsh, O., Balint, K., Inskip, C, and Visser, F.), 2022. Information Warfare and Wikipedia. [accessed 27 September 2023]. 343
Editing usernames¶
16.103 Evidence suggests that the ability to change a username, handle, or other information presented on a user profile can be exploited by perpetrators of foreign interference operations. Researchers at the City University of London found that 26,538 Twitter accounts suddenly changed their usernames after the EU referendum in 2016, and 5% of all Twitter accounts that had tweeted about the referendum were either deleted or renamed.1944 This changing of account names can be used by perpetrators to quickly repurpose accounts from one influence operation to another. It also enables perpetrators to easily change the focus of an account if it is not performing as well as they would like, or if they want to switch focus to a different topic. There are examples of the Internet Research Agency renaming and rebranding some of its Instagram accounts during its operation targeting the 2016 US Presidential election.1945
Recommender systems¶
Content recommender systems¶
16.104 In addition to personalisation, content recommender systems are commonly designed to suggest content that might be trending or popular (measured by number of likes, shares, or comments). Such systems are understood to learn about popular and trending content through the volume of user feedback; this normally includes explicit feedback (active engagement such as reactions, posts and comments) and implicit feedback (viewing the content many times but not necessarily engaging with it).1946 This fundamental characteristic of recommender system design leaves services vulnerable to manipulation by third parties, particularly if their design is simple (for example, if all content is ranked in the same way and all types of engagement are registered as positive feedback on all types of content). We consider that content recommender systems may be manipulated by perpetrators of foreign influence operations.
Risk factors: Business model and commercial profile¶
Revenue models¶
16.105 Evidence suggests that services which raise income through advertising may be exploited by bad actors who can use advertisements to spread foreign interference content. A report on the political ad policy for an online U2U service recognised this potential risk,1947 saying that “scrutiny of major online advertising platforms intensified due to foreign interference in the 2016 U.S. elections as well as broader concerns on disinformation, voter suppression, and inauthentic behaviour”.1948 The report added that if users are unaware of the political intent
1944 Bastos, M. T. and Mercea, D, 2017. The Brexit Botnet and User-Generated Hyperpartisan News, Social Science Computer Review, 37(1). [accessed 27 September 2023]. 1945 DiResta, R., Shaffer, K., Ruppel, B., Sullivan, D., Matney, R., Fox, R., Albright, J. and Johnson, B., 2019. The Tactics and Tropes of the Internet Research Agency, New Knowledge. [accessed 27 September 2023]. 1946 Perpetrators may take advantage of the design of engagement focused systems by acting in a coordinated fashion to generate high volumes of explicit feedback to artificially inflate the dissemination of specific posts, for example, by using multiple accounts to upload and share the same content many times. By artificially inflating engagement, content recommender systems could then be more likely to promote this content to users.
1947 The report suggested that “online political advertising is a powerful tool for enabling engagement in the political process but that with this power comes the risk of abuse that can harm the integrity of the democratic process.” 1948 Le Pochat, V., Edelson, L., Van Goethem, T., Joosen, W., McCoy, D. and Lauinger, T., 2022. An audit of Facebook's Political Ad Policy Enforcement. [accessed 27 September 2023]. 344
behind the advert, the adverts can be more effective in their malicious intent. Hence, services offering advertising, without effective moderation policies to identify and label adverts that seek to influence public political Fl opinion as ‘political’, are more able to be used by malicious advertisers and thereby weaken the integrity of the online political ad ecosystem.1949
1949 Le Pochat, V., Edelson, L., Van Goethem, T., Joosen, W., McCoy, D. and Lauinger, T., 2022.
1950 DiResta, R., Shaffer, K., Ruppel, B., Sullivan, D., Matney, R., Fox, R., Albright, J. and Johnson, B., 2019. The Tactics and Tropes of the Internet Research Agency, New Knowledge. [accessed 27 September 2023]. 1951 Some services enable advertisers to purchase adverts and target them at specific users based on information that the users have provided to services, and data that the services have gathered on users’ interests and behaviour. 1952 Colliver, C., King, J. and Maharasingam-Shah, E., 2020. Hoodwinked: Coordinated Inauthentic Behaviour on Facebook. [accessed 27 September 2023]. 345
Section 17 Animal cruelty¶
Warning: this chapter contains content that may be upsetting or distressing. Summary analysis for animal cruelty offences: How harms manifest online, and risk factors This section summarises the risks of harm to individuals from the animal cruelty offence. The existence of online activities that encourage, assist or commit acts of animal cruelty may result in content being made available which may distress a user, or cause them to engage in harmful or illegal behaviours and activities themselves. This section covers several factors which could be associated with the offence. Of these, we consider the following to be key and have included them in the Risk Profiles (see Section 7). Service type risk factors: As with almost all kinds of illegal harm, social media services are a prominent risk factor for this offence, since content depicting cruelty to animals (which may in itself encourage, assist or conspire to further animal cruelty) shared on these services can receive wide reach. Our evidence also points to messaging services being a risk factor, in that they allow perpetrators to form a community and to discuss ideas for, acts of cruelty. They may also assist in the production and publication of animal cruelty content or share it via the messaging services. Functionalities risk factors: Services with the ability to post images or videos (which may be social media services) pose a significant risk of this offence, particularly where they can encourage or facilitate further acts of animal cruelty. Commenting on content can also enable people to encourage, assist or conspire to commit further acts of animal cruelty. Our evidence also points to services where users can form user groups or send group messages being a prominent functionality that is a risk factor, for the same reason that we believe messaging services to be a risk factor – it may allow perpetrators to come together to discuss and encourage, facilitate or commit acts of animal cruelty. This may be especially the case for closed and private groups.
Introduction¶
• the use of U2U services for the commission and/or facilitation of these offences (collectively the ‘risks of harm’).
Relevant offences¶
Use of the service for commission or facilitation¶
Other offences¶
How animal cruelty manifests online¶
1953 This subject has been the subject of academic discourse for many years. There are several older studies on the subject including Thompson, K.L. and Gullone, E. 2006. An investigation into the association between the witnessing of animal abuse and adolescents’ behaviour toward animals, Society & Animals 6, 221-243 [accessed 24 June 2024]; McVie, S. 2007. Animal abuse among young people aged 13 to 17. Royal Society for the Prevention of Cruelty to Animals/University of Edinburgh. [accessed 24 June 2024]. More recent research has also drawn links between a child viewing animal abuse or other types of familial violence and going on to abuse humans or animals themselves, such as Jegatheesan, B., Enders-Slegers, M-J., Ormerod, E. and Boyden, P. 2020. Understanding the link between animal cruelty and family violence: the bioecological systems model, International Journal of Environmental Research and Public Health 17. [accessed 24 June 2024].
1954 Ofcom, 2024. Online Experiences Tracker. [accessed 18 November 2024]. Fieldwork was carried out in a four-week period in May and June 2024. Note that in our November 2023 and August 2024 Consultations we referred to previous iterations of this research (Waves 4 and 5). 1955 RSPCA, 2023. Cruelty to cats increased by 25% on last year; RSPCA, 2023. One dog abused every hour: cruelty to dogs on the rise [both accessed 24 June 2024]. 348
believe it is a reasonable assumption that as the prevalence of an offence increases generally, the online manifestation of that offence is likely to rise with it. The National Wildlife Crime Unit in the UK notes that the internet has enabled various forms of wildlife crime, including the promotion or facilitation of illegal activities such as hare coursing competitions.1956
Risks of harm presented by animal cruelty¶
1956 National Wildlife Crime Unit [date unspecified]. Cyber enabled wildlife crime. [accessed 9 May 2024]. 1957 Harrington, L.A., Elwin, A., Paterson, S. and D’Cruze, N. 2023. ‘The viewer doesn’t always seem to care – response to fake animal rescues on YouTube and implications for social media self-policing policies’, People and Nature 5 [accessed 24 June 2024]; World Animal Protection, 2021. Views that abuse: the rise of fake "animal rescue" videos on YouTube; Social Media Animal Cruelty Coalition, 2024. Spot the Scam: Unmasking Fake Animal Rescues. [accessed 28 October 2024]. 349
more of the same, and that its existence can therefore contribute to activities which do encourage, assist or conspire.
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media and video-sharing services¶
1958 Social Media Animal Cruelty Coalition, 2021. Making money from misery, p.26 [accessed 24 June 2024]. Note that SMACC’s publications referenced across this Register of Risks section, and the next section on the obscene torture content offence, are among the very few systematic studies into animal-cruelty related content online, some of which may constitute the animal cruelty or obscene content offence. We refer to these sources as evidence that potentially harmful and illegal content exists online, not as an indicator of the quantity or prevalence of this content. This caveat also applies to the research produced by other charities and non-profit organisations, such as the RSPCA, Born Free USA, Lady Freethinker and the Alliance to Counter Crime Online.
1959 As reported directly to Ofcom by the Social Media Animal Cruelty Coalition based on recent, as-yet-unpublished data from April 2024; also Social Media Animal Cruelty Coalition 2023. The cruelty you don’t see, pp.51-52. [accessed 24 June 2024]. 350
1960 Carvalho, A.F., de Morais, I.O.B., and Souza, T.B. 2023. ‘Profiting from cruelty: digital content creators abuse animals worldwide to incur profit’, Biological Conservation 287; World Animal Protection, 2021. Views that abuse: the rise of fake "animal rescue" videos on YouTube. [accessed 24 June 2024]. 1961 UK Safer Internet Centre, 2024. UK Safer Internet Centre sees concerning rise in animal abuse content. [accessed 24 June 2024]. 1962 RSPCA, 2024. Kindness Index Report 2024. [accessed 20 July 2024]. The data showing that a significant number of young people view animal cruelty content online appears to reflect older research from the RSPCA, which found that 48% of the surveyed 10-18 year-olds had witnessed animal cruelty, of whom almost a quarter (23%) had seen it on social media: RSPCA, 2018. The RSPCA’s Generation Kind, p.2, and footnote 2 [accessed 9 May 2024]. In a related document published at a similar time, they found that one in three 10-15 year-olds reported seeing animal cruelty on social media: RSPCA, 2018. Building a kinder generation, p.7. [accessed 9 May 2024]. 1963 BBC, 2023 Monkey Haters (documentary) [accessed 27 March 2024].
1964 Montrose, Kogan and Oxley. 2021. The role of social media in promoting organised dog fighting, The Veterinary Nurse. [accessed 24 June 2024]. 1965 Mitchell, J, 2023. Gamekeeper who filmed animal fights for TikTok spared jail, Sky News, 12 December. [accessed 28 October 2024]. 1966 Heaney, P, 2021. Dog Mutilation: Breeders Cropping Ears to Follow Social Media Trend, BBC News, 13 December. [accessed 3 October 2024]. 351
Messaging services, discussion forums and chat room services¶
File-storage and file-sharing services¶
1967 BBC, 2023. Monkey Haters (documentary). [accessed 27 March 2024]. 1968 Lady Freethinker, 2021. YouTube 'monkey haters' form private group where members are paying to have baby monkeys tortured and killed on camera [accessed 28 June 2024]. 1969 Scottish SPCA response to August 2024 Illegal Harms Further Consultation, p.2. 1970 RSPCA response to August 2024 Illegal Harms Further Consultation, p.1.
1971 As reported directly to Ofcom by the Social Media Animal Cruelty Coalition based on recent, as-yet-unpublished data from April 2024. 1972 Our analysis suggests that file-storage and file-sharing services post a particularly high risk of disseminating CSAM, terrorist content and non-consensual intimate imagery (as part of intimate image abuse) – this is covered in the relevant chapters in our Register of Risks. 1973 Born Free Foundation response to November 2023 Illegal Harms consultation, p.3. 352
User-to-user pornography services¶
Risk factors: User base¶
User base size¶
1974 McGlynn, C. and Bows., H. 2019. Possessing Extreme Pornography: Policing, Prosecutions and the Need for Reform, The Journal of Criminal Law, 83(6), pp.481-482. [accessed 24 June 2024]. 1975 Born Free USA, 2022. Their lives for your likes: the exploitation of wild animals on social media [accessed 10 May 2024].
1976 BBC, 2023. Monkey Haters (documentary) [accessed 27 March 2024]. 1977 As reported directly to Ofcom by the RSPCA: Ofcom/RSPCA meeting, October 2023, and by the Social Media Animal Cruelty Coalition based on recent, as-yet-unpublished research from April 2024, which found that individuals (including children) can easily join private groups or forum by requesting access. 1978 Alliance to Counter Crime Online, 2020. Two Clicks Away: wildlife sales on Facebook, Appendix B, which documents the Pages and Groups identified, including membership sizes [accessed 8 May 2024]. 353
Risk factors: Functionalities and recommender systems¶
User identification¶
Anonymous user profiles and fake user profiles¶
User networking¶
User connections¶
User groups and group messaging¶
1979 Being able to accrue large followings may build a sense of legitimacy (such as on accounts sharing ‘fake rescue’ content and requesting financial donations), a potential risk factor for fraud. See the Fraud and financial services chapter of the Register of Risk. 1980 Lady Freethinker 2019. The deadly, underground world of dogfighting on Facebook [accessed 8 May 2024]. 1981 BBC, 2023 Monkey Haters (documentary) [accessed 27 March 2024]; also, as reported directly to Ofcom by the RSPCA: Ofcom/RSPCA meeting, October 2023.
1982 In one case prosecuted by the Crown Prosecution Service a user joined a private group on social media, posted videos and wrote comments that showed approval, therefore potentially encouraging engagement with other users. This case was prosecuted under the Obscene Publications Act 1959 – see our Illegal Content Judgements Guidance paragraph xxx on our decision to use s127(1) of the Communications Act 2003 rather than the Obscene Publications Act. Crown Prosecution Service, 2024. Man jailed for posting videos of baby monkeys being tortured. [accessed 25 October 2024]. 354
were then used to encourage their connections around the world to create this type of content.1983
User communication¶
Posting content (images and videos)¶
Re-posting and sharing/forwarding content¶
Livestreaming¶
1983 Lawson, E., Gunter, J. and Henschke, R., 2024, Kidderminster women pleads guilty to role in monkey torture network, BBC News, 7 May. [accessed 8 May 2024]. 1984 Social Media Animal Cruelty Coalition, 2021. Making money from misery [accessed 24 June 2024]; South West Grid for Learning (SWGfL), 2024. ‘Report Harmful Content sees concerning rise in animal abuse content’ [accessed 8 May 2024]; Mitchell, J, 2023. Gamekeeper who filmed animal fights for TikTok spared jail, Sky News, 12 December. [accessed 28 October 2024]; World Animal Protection, 2021. Views that abuse: the rise of fake "animal rescue" videos on YouTube. [accessed 31 October 2024].
1985 On fake rescue videos: Social Media Animal Cruelty Coalition, 2024. Spot the Scam: Unmasking Fake Animal Rescues. [accessed 28 October 2024]; World Animal Protection, 2021. 355
livestreaming to be a risk factor, which could potentially include animal fights and torture.1986 There may be an element of livestreaming of cruelty within animal torture groups, such as the cat torture networks originating in China.1987 Cooking or eating videos are another specific genre of content which may show the unnecessary suffering of animals, albeit with limited evidence for this content being livestreamed.1988
Direct messaging and encrypted messaging¶
Commenting on content¶
1986 For example, the RSPCA response to November 2023 Illegal Harms Consultation, p.2 and the Scottish SPCA response to November 2023 Illegal Harms Consultation, pp.2 and 4. 1987 Cheung, R, 2023. A man said he’d adopt cats and torture them in a livestream. Then vigilantes took action, Vice, 9 March 2023. [accessed 24 October 2024]. 1988 In one case reported in the media (Harris, M., 2020, ‘A YouTuber with over 3 million follower responded to backlash’, Business Insider, 17 April. [accessed 9 May 2024]), a creator of ‘mukbang’ content (a type of cooking and eating video that can aim to share cultural traditions, but also overlaps with “ASMR” videos) was criticised for posting videos of herself eating live animals. In this case, the videos themselves were not livestreamed. However, the genre of ‘mukbang’ is known to be either pre-recorded or livestreamed, indicating that the latter could be risk factor for the animal cruelty offence.
1989 National Wildlife Crime Unit [date unspecified]. Cyber enabled wildlife crime. [accessed 9 May 2024]. 1990 Social Media Animal Cruelty Coalition, 2021. Making money from misery, p.21 [accessed 24 June 2024]. 356
more content of the same type, thereby implicitly validating and enabling the poster to perform more acts of cruelty and/or share more content depicting these acts.
1991 Conversely, the study found some correlation between participants who said they would comment to show their disapproval and higher levels of self-esteem (in that these participants felt they were furthering animal welfare causes). Note this study is in pre-print (not yet fully published) and used a non-representative sample through a survey of self-reported attitudes. We therefore use it as an indicator for the potential for commenting on comment to be a factor in encouraging, assisting or conspiring to commit animal cruelty content, rather than as definite evidence that this occurs. Source: McGuirk, L.Ryan and Alleyne, E. 2024 ‘“Liking,” “Commenting,” and “Reposting”: Psychological factors associated to online animal abuse’, Society & Animals, p.17. [accessed 18 November 2024]. 1992 Social Media Animal Cruelty Coalition 2023. The cruelty you don’t see. [accessed 24 June 2024]. 1993 Social Media Animal Cruelty Coalition, 2022a. Wild animal "pets" on social media. [accessed 24 June 2024]. 1994 Social Media Animal Cruelty Coalition, 2022b. Teasing as Torture. [accessed 24 June 2024]. The positive (if apparently short-lived) impact of the 2015 ‘Tickling as Torture’ campaign which aimed to raise public awareness of an act which is not clearly animal cruelty is referenced in Moloney, G.K., Tuke, J., Dal Grande, E., Nielsen, T. and Chaber, A.-L. 2021. ‘Is YouTube promoting the exotic pet trade? Analysis of the global public perception of popular YouTube videos featuring threatened exotic animals’. [accessed 24 June 2024].
1995 Moloney, G.K. et al. 2021. [accessed 24 June 2024]. 1996 Born Free USA, 2022, p.11. [accessed 24 June 2024]. 357
content may nevertheless motivate them to continue performing these acts. A service with comments functions therefore could facilitate the animal cruelty offence.
Transaction and offers¶
Posting goods and services for sale¶
Recommender systems¶
Content recommender systems¶
1997 Social Media Animal Cruelty Coalition, 2021. Making money from misery, p.21. [accessed 24 June 2024]. 1998 Two UK women charged with encouraging or assisting unnecessary suffering made payments for content via PayPal, as reported in the media: Lawson, E., Gunter, J. and Henschke, R., 2024; and Rack, S. and Cooper, M, Monkey torture video accused woman granted bail, BBC News, 11 June [accessed 8 May 2024]. In the broader international network, individuals charged in the US collected money from group members and sent it to their contact abroad who then created the videos: Office of Public Affairs (US Department of Justice), 2024. ‘Two charged for involvement with online groups dedicated to monkey torture and mutilation’. [accessed 22 October 2024]; US Department of Homeland Security, 2024. ‘Virginia man pleads guilty to producing, distributing sadistic animal torture videos following HSI Norfolk investigation’. [accessed 22 October 2024].
1999 Social Media Animal Cruelty Coalition, 2021. Making money from misery, p.45. [accessed 24 June 2024]. From the context of the wildlife trade, there is some evidence of pages providing options for admins to use shopping feature and allow users to easily send direct messages to sellers (Alliance to Counter Crime Online, 2020. Two Clicks Away: wildlife sales on Facebook. [accessed 8 May 2024]). While this does not necessarily constitute the animal cruelty offence, it indicates it could be a risk factor for this content. 2000 BBC, 2023 Monkey Haters (documentary). [accessed 27 March 2024]. 358
cruelty content – whether or not they realise what it is – they may be shown more similar content.2001
Network recommender systems¶
Content exploring factors¶
User generated content searching¶
Risk factors: Business models and commercial profiles¶
Business model (revenue model and growth strategy)¶
2001 The Social Media Animal Cruelty Coalition has told Ofcom that when conducting a specific piece of research on “fake rescue” content, 20% of the 763 pieces of content that they recorded had been recommended to their volunteer researchers’ personal accounts as opposed to having been searched for; while it is difficult to assess whether this is widespread amongst other users, it suggests that this could be a risk. 2002 The Social Media Animal Cruelty Coalition advises users not to engage with animal cruelty content, noting that views and engagement may increase popularity of the view. For example: Social Media Animal Cruelty Coalition, 2021, p.14. [accessed 24 June 2024].
2003 Alliance to Counter Crime Online, 2020. 2004 Carvalho, A.F., de Morais, I.O.B., and Souza, T.B. 2023; Social Media Animal Cruelty Coalition, 2021. Making money from misery, p.16. [accessed 24 June 2024]; Social Media Animal Cruelty Coalition 2023. The cruelty you don’t see, p.12. [accessed 24 June 2024]. 359
Commercial profile¶
2005 Carvalho, A.F., de Morais, I.O.B., and Souza, T.B. 2023. 2006 Carvalho, A.F., de Morais, I.O.B., and Souza, T.B. 2023, pp.3 and 8; see also Social Media Animal Cruelty Coalition, 2024. Spot the Scam: Unmasking Fake Animal Rescues, pp.17 and 28-29. [accessed 28 October 2024]. 360
Section 18 Cyberflashing¶
Warning: this chapter contains content that may be upsetting or distressing. Summary analysis for cyberflashing offence: how harm manifests online, and risk factors The cyberflashing offence refers to the sending of a photograph or film of genitals, to cause alarm, distress or humiliation, or to obtain sexual gratification. A 2024 study showed that 36% of women in the UK aged between 18 and 39 had received an unsolicited sexual photo at some point in their lives, with rates being higher among young people.2007 Some individuals report very high levels of cyberflashing, receiving several images a day. The risks of harm to individuals from cyberflashing include negative psychological impacts. Victims and survivors describe feeling vulnerable and embarrassed, and view cyberflashing as aggressive and intimidating. In addition, cyberflashing can form part of a pattern of harmful behaviour that includes other harms such as cyberstalking, harassment, and/or controlling or coercive behaviour. Service type risk factors: Cyberflashing offences can occur on any service that enables users to share images. Social media services, messaging services and dating services were found to be particularly risky. User base risk factors: Cyberflashing is a gendered offence; among individuals aged 18-34, women are much more likely than men to have received an unsolicited sexual photo (40% vs 26%).2008 There is also evidence to suggest that women in minority ethnic groups and LGBT+ groups disproportionately experience cyberflashing. Functionalities and recommender system risk factors: User connections allow perpetrators to make contact with victims and survivors. Direct messaging, including ephemeral messaging, can allow perpetrators to cyberflash victims and survivors by sending messages which contain sexual images. Livestreaming can facilitate a form of cyberflashing where a perpetrator exposes themselves live through a video-call.
2007 Source: YouGov (Smith, M.), 2024. More than a third of women under 40 have received unsolicited sexual photos. YouGov. [accessed 11 February 2026]. 2008 Sample consisted of 1629 adults. Source: YouGov (Smith, M), 2018. Four in ten young women have been sent unsolicited sexual images. [accessed 28 July 2023]. 361
Introduction¶
Relevant offences¶
2009 One study found that a majority of girls in a qualitative research project of 144 young people aged 11-18 had received unsolicited naked images of boys or men. Source: Ringrose, J., Regehr, K. and Whitehead, S, 2021. Teen Girl’ Experiences Negotiating the Ubiquitous Dick Pic: Sexual Double Standards and the Normalization of Image Based Sexual Harassment, Sex Roles, 85 (558). [accessed 11 August 2023].
2010 Bluetooth allows for wireless ‘pairing’ between two proximate devices using a peer-to-peer network. Bluetooth ‘pairing’ can be used to share files between devices, and perpetrators can use this to share unsolicited explicit images with nearby devices and cyberflash the device’s user. 2011 New section 66A of the Sexual Offences Act 2003. 2012 Section 187 of the Online Safety Act 2023. 362
How cyberflashing manifests online¶
2013 YouGov (Smith, M.), 2018. Four in ten female millennials have been sent an unsolicited penis photo. [accessed 6 March 2026]. 2014 For example, in a 2019 survey of Canadian men, a notable proportion of those how confirmed they had sent unsolicited sexual images indicated “that they hoped to provoke negative emotions in recipients, with 17% hoping for shock, 15% hoping for fear, and 11% hoping for disgust.” Source: Oswald, F., Lopes, A., Skoda, K., Hesse, C. and Pedersen, C., 2019. I’ll Show You Mine so You’ll Show Me Yours: Motivations and Personality Variables in Photographic Exhibitionism. The Journal of Sex Research, 57 (5), p.1-13. [accessed 6 March 2026]. 2015 For example, gender-based power dynamics can be different in same sex relationships compared to heterosexual ones, resulting in different experience of sharing sexual images. Dietzel, C. 2021. The three dimensions of unsolicited Dick Pics: Men who have sex with men’s experiences of sending and receiving unsolicited Dick Pics on dating apps, Sexuality & Culture, 26(3), pp. 834–852. [accessed 18 November 2024].
2016 Sexual autonomy refers to one’s right to make informed decisions related to their body, sexuality or sexual experiences, free of societal pressure and desires. Willie, T.C.., Callands, T., Alexander, K.A. 2023. Measuring women’s sexual autonomy: Development and preliminary validation of the Women’s Sexual Autonomy Scale, Women’s Health, 19. [accessed 18 November 2024]. 2017 Source: YouGov (Smith, M.), 2024. More than a third of women under 40 have received unsolicited sexual photos. YouGov. [accessed 11 February 2026]. 363
cyberflashing are even higher2018; 47% of women aged 18-24 have been cyberflashed2019 and 19% of girls aged 11-16 have been sent unwanted sexual images.2020
Risks of harm to individuals presented by cyberflashing¶
2018 Sample consisted of 2,353 women aged 18-36 and 1,327 men aged 18-36. 71% of women 18-24 were younger than 18 the first time they were cyberflashed. Source: YouGov (Smith, M.), 2018. Four in ten female millennials have been sent an unsolicited penis photo. [accessed 26 July 2023]. 2019 YouGov (Smith, M.), 2018. 2020 Sample consisted of 2,114 girls and young women aged 7-21. Source: Girlguiding, 2021. Girls’ Attitudes Survey. [accessed 26 July 2023]. 2021 As mentioned above, perpetrators can also use a number of other means out of scope of the Act such as by email. We will not explore the details of these in this chapter. 2022 Law Commission, 2021. Modernising Communications Offences: A final report. [accessed 26 July 2023]. 2023 McGlynn, C., 2021. Written evidence submitted by Professor Clare McGlynn, Durham Law School, Durham University (VAW0007). [accessed 29 July 2023]; McGlynn, C. and Johnson, K., 2022. Cyberflashing: Recognising Harms, Reforming Laws. Bristol: Bristol University Press. [accessed 21 September 2023]. 2024 Glitch, UK, 2023. The Digital Misogynoir Report: Ending the dehumanising of Black women on social media. [accessed 18 November 2024]. 2025 Law Commission, 2021.
2026 Sample consisted of 1,629 adults. Source: YouGov (Smith, M.), 2018. Four in ten young women have been sent unsolicited sexual images. [accessed 26 July 2023]. 2027 Law Commission, 2021. 2028 The National Police Chiefs’ Council response to 2020 Law Commission Consultation. [accessed 2 August 2023]. 2029 McGlynn, C. response to 2020 Law Commission Consultation. [accessed 2 August 2023]; McGlynn, C. and Johnson, K., 2022. Cyberflashing: Recognising Harms, Reforming Laws. Bristol: Bristol University Press. [accessed 21 September 2023]. 364
experienced may vary depending on whether the perpetrator was previously known or not.2030
Evidence of risk factors on user-to-user services¶
Risk factors: Service type¶
Social media services¶
2030 McGlynn, C. and Johnson, K., 2022. Cyberflashing: Recognising Harms, Reforming Laws. Bristol: Bristol University Press. [accessed 21 September 2023]. 2031 McGlynn, C., 2022. Cyberflashing: Consent, Reform and the Criminal Law, The Journal of Criminal Law, 85 (5). [accessed 28 July 2023]. 2032 McGlynn, C., 2021. Written evidence submitted by Professor Clare McGlynn, Durham Law School, Durham University (VAW0007). [accessed 28 July 2023]. 2033 Law Commission, 2021. 2034 Revealing Reality, 2023. Anti-social Media: The violent, sexual and illegal content children are viewing on one of their most popular apps. [accessed 26 July 2023]. 2035 McGlynn, C. and Johnson, K., 2022. Cyberflashing: Recognising Harms, Reforming Laws. Bristol: Bristol University Press. [accessed 21 September 2023]; Brazier, T., 2022. Emily Atack to front new BBC documentary about online sexual harassment after being targeted ‘from a very young age’, Metro, 4 September. [accessed 12 September 2023].
2036 See the Harassment, stalking, threats and abuse and Controlling or coercive behaviour chapters. 2037 See the Intimate image abuse and Extreme pornography chapters. 2038 Law Commission, 2021. Modernising Communications Offences: A final report. [accessed 26 July 2021]. 2039 Q21 (Table 1527). Source: Ofcom, 2023. Experiences of using online services. [. [accessed 1 August 2023]. 365
Messaging services¶
Dating services¶
Risk factors: User base¶
User base size¶
User base demographics¶
2040 The research found that cyberflashing was most prevalent on social media, with 67% of women between 18-39 having encountered it. Source: YouGov (Smith, M.), 2024. More than a third of women under 40 have received unsolicited sexual photos. [accessed 11 February 2026]. 2041 Survey was carried out on OnePoll and the sample consisted of 4,000 single respondents aged 18-65 who were actively using dating sites. Source: Plenty of Fish, 2023. The Desirable Dating Guide. [accessed 28 July 2023]. 2042 McGlynn, C. and Johnson, K., 2022.
2043 Vera-Gray, F., McGlynn, C., Lovett, J. and Butterby, V., 2026. Freedom under threat. [accessed 2nd June 2026]. 2044 Survey commissioned by Bumble and was carried out by Research without Borders, spanning between 15th-18th October 2021 with 1,793 respondents who live in England or Wales: Bumble, n.d. Women’s Safety Experts Join Bumble to Call on UK Prime Ministerial Candidates to Prioritise Anti-Cyberflashing Law. [accessed 10 June 2024] 2045 Sample consisted of 2,353 women aged 18-36 and 1,327 men aged 18-36. Source: YouGov (Smith, M), 2018. Four in ten female millennials have been sent an unsolicited penis photo. [accessed 28 July 2023]. 366
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles
Anonymous user profiles¶
User networking¶
User connections and user searching¶
2046 Ofcom, Online Experiences Tracker 2021-2022. Comprises Wave 1 and 2 combined data set. 2047 Ofcom, Online Experiences Tracker 2021-2022. Comprises Wave 1 and 2 combined data set. 2048 Center for Countering Digital Hate, 2022. Hidden Hate: How Instagram fails to act on 9 in 10 reports of misogyny in DMs. [accessed 28 July 2023]; Bond, K., 2023. ‘It’s in our phone, in our hands, and it’s in our house’: Six women share their experiences of online sexual harassment, Metro, 31 January. [accessed 28 July 2023]. 2049 Suler, J., 2004. The Online Disinhibition Effect, Cyberpsychology & behaviour: the impact of the internet, multimedia and virtual reality on behaviour and society, 7(3). [accessed 21 September 2023]. 2050 McGlynn, C. and Johnson, K., 2022. Cyberflashing: Recognising Harms, Reforming Laws. Bristol: Bristol University Press. [accessed 21 September 2023]. 2051 Gallagher, S., 2018. ‘Violated, Sick, Uncomfortable’: 10 Women On Being Sent Unsolicited Dick Pics, HuffPost, 26 October. [accessed 28 July 2023]; Revealing Reality, 2022. Not Just Flirting: The unequal experiences and consequences of nude image-sharing by young people. [accessed 28 July 2023].
2052 BBC, 2023. Emily Atack: Asking For It? (Documentary), 31 January. [accessed 2 August 2023]; Kelly, K., 2023. Love Island star Amy Hart felt ‘violated’ after being bombared with ‘cyberflashing’ online, LBC, 18 April. [accessed 28 July 2023]; McLoughlin, L., 2023. Vanessa Feltz reveals she ‘regularly’ receives unsolicited sexual images from men, Evening Standard, 19 April. [accessed 28 July 2023]. 367
User communication¶
Direct messaging¶
Ephemeral messaging¶
Posting content (images and videos)¶
Livestreaming¶
Content storage and capture¶
Capturing images¶
Risk factors: Business models and commercial profile¶
2053 McGlynn, C. and Johnson, K., 2022; Revealing Reality, 2022. 2054Sample consisted of 4,040 adults. Source: YouGov (Smith, M.), 2024. More than a third of women under 40 have received unsolicited sexual photos. [accessed 11 February 2026]. 2055 Ringrose, J., Regehr, K. and Whitehead, S, 2021. Teen Girl’ Experiences Negotiating the Ubiquitous Dick Pic: Sexual Double Standards and the Normalization of Image Based Sexual Harassment, Sex Roles, 85 (558). [accessed 11 August 2023]; Revealing Reality, 2022.
2056 Revealing Reality, 2023. Anti-social Media: The violent, sexual and illegal content children are viewing on one of their most popular apps. [accessed 26 July 2023]. 2057 Elmer, G., Neville, S., Burton, A. and Ward-Kimola, S., 2021. Zoombombing During a Global Pandemic, Social Media + Society, 7(3). [accessed 28 July 2023]. 368
Section 19 Epilepsy trolling offence¶
Warning: this chapter contains content that may be upsetting or distressing.
Summary analysis for epilepsy trolling offence: how harm manifests online and risk factors Some individuals with epilepsy may have a physical reaction to online content; they may feel disorientated, uncomfortable or unwell after seeing certain images or patterns. The offence covered in this chapter is sharing an image with the intention to cause harm to an individual with epilepsy. As the epilepsy trolling offence is new, it is difficult to state the prevalence of the harm. However, Epilepsy Action, a British charity, states that around one in 100 people in the UK have epilepsy. Of these individuals, 3% have photosensitive epilepsy.2058 The risks of harm to individuals – both adults and children – from epilepsy trolling can be psychological and physical. Some individuals may lose their life after having a seizure. Targeted attacks can also cause anxiety among epileptic users. Service type risk factors: Social media services have been identified as having higher risks of harm connected to epilepsy trolling, although any service which allows upload of images or videos may be a risk. User base risk factors: Disability and age are risk factors for this offence. Charities supporting people with epilepsy advise that they are being targeted, and that young people may be particularly vulnerable. Functionalities and recommender systems risk factors: Evidence indicated that perpetrators can create multiple user profiles to evade account-blocking efforts, and often carry out epilepsy trolling anonymously, such as by creating anonymous user profiles. The user connections displayed on user profiles can be used to find potential victims. The ability to comment and post content such as videos or images allows perpetrators to share flashing or contrasting visual media, deliberately targeting victims and survivors. Perpetrators can tag users in posted content, which contains flashing or contrasting visual media, to trigger seizures. Tagging content with popular hashtags, can also increase the risk of content with flashing or contrasting visual media (which has the potential to trigger seizures) being disseminated on a service. Recommender systems can increase the risk that this content, if tagged
2058 Epilepsy Action, n.d. Photosensitive epilepsy. [accessed 10 May 2023]. 369
with epilepsy-related hashtags, will be seen by those engaging with epilepsy-related content (perhaps because they have epilepsy themselves).
Introduction¶
Relevant offence¶
How epilepsy trolling manifests online¶
2059 We have considered in other chapters how U2U services can be used to commit or facilitate priority offences, as required by Ofcom’s risk assessment duty. Epilepsy trolling is not a priority offence; however, our analysis has also considered how U2U services might be used to commit or facilitate the offence, and we set out evidence how this may happen in this chapter. 2060 We note that the epilepsy offence itself adopts a different definition of harm (see section 183(13)). 2061 Flashing images include GIFs. 370
Risks of harm to individuals presented by the offence of epilepsy trolling online¶
2062 Epilepsy Action, n.d. Photosensitive epilepsy. [accessed 10 May 2023]. 2063 Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 17 April 2023]. 2064 Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 15 June 2023].
2065 Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 17 April 2023]. 2066 Source: Epilepsy Society, n.d. Epilepsy Society X Page. [accessed 31 July 2023]. 2067 Meeting between Ofcom and Epilepsy Society, April 2023. 2068 Meeting between Ofcom and Epilepsy Society, April 2023. 2069 Epilepsy Society, n.d. Photosensitive epilepsy. [accessed 2 May 2023]. 2070 Meeting between Ofcom and Epilepsy Society, April 2023. 371
Evidence of risk factors on user-to-user services¶
Risk factors: Service types¶
Social media services¶
Risk factors: User base¶
User base demographics¶
2071 Meeting between Ofcom and Epilepsy Society, April 2023. 2072 One example was that of a 25-year-old man who had recently been diagnosed with epilepsy. He visited the Epilepsy Society X page after being recommended by friends for peer support and trustworthy information. Unfortunately, he soon came across a flashing image posted on the page which resulted in him experiencing a serious convulsive seizure which caused him to bite through his tongue. He was psychologically traumatised as a result. Source: Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 17 April 2023].
2073 In May 2020 a young boy aged 8 was harmed when his mum proudly shared a video of him on X of him attempting to raise money for the Epilepsy Society. Zach had epilepsy and cerebral palsy. On posts such as this, perpetrators share multiple flashing images and GIFs to try to trigger seizures in people with photosensitive epilepsy. Zach was a victim of this attack and the campaign against this behaviour was named ‘Zach’s Law’ as a result. Source: Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 17 April 2023]. 2074 Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 17 April 2023]. 372
sometimes individuals with photosensitive epilepsy will experience fewer symptoms once they enter their mid to late twenties.2075
Risk factors: Functionalities and recommender systems¶
User identification¶
User profiles and anonymous user profiles¶
User networking¶
User groups¶
User connections¶
User tagging¶
2075 Meeting between Ofcom and Epilepsy Society, April 2023.
2076 Epilepsy Action, n.d. Photosensitive epilepsy. [accessed 10 May 2023]. 2077 Meeting between Ofcom and Epilepsy Society, April 2023. 2078 Meeting between Ofcom and Epilepsy Society, April 2023. 2079 Suler, J., 2004. The Online Disinhibition Effect, Cyberpsychology & behaviour: the impact of the internet, multimedia and virtual reality on behaviour and society, 7 (3). [accessed 27 April 2023]. 2080 Meeting between Ofcom and Epilepsy Society, April 2023. 373
Epilepsy Society’s X page, as well as tagging users who had recently tweeted about their epilepsy, in the comments of posts with harmful content that could trigger seizures.2081
User communications¶
Direct messaging¶
Posting content and commenting on content (images and videos)¶
Content exploring¶
Content tagging¶
2081 Meeting between Ofcom and Epilepsy Society, April 2023.
2082 Meeting between Ofcom and Epilepsy Society, April 2023. 2083 Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 17 April 2023]. 2084 Meeting between Ofcom and Epilepsy Society, April 2023. 2085 Epilepsy Society, n.d. Call for evidence on the Draft Online Safety Bill. [accessed 17 April 2023]. 2086 Meeting between Ofcom and Epilepsy Society, April 2023. 2087 Meeting between Ofcom and Epilepsy Society, April 2023. 374
Hyperlinking¶
Content editing¶
Editing visual media¶
Recommender systems¶
Content recommender systems¶
2088 Meeting between Ofcom and Epilepsy Society, April 2023. 2089 Meeting between Ofcom and Epilepsy Society, April 2023. 2090 Meeting between Ofcom and Epilepsy Society, April 2023. 2091 Meeting between Ofcom and Epilepsy Society, April 2023. 375
Risk factors: Business models and commercial profiles¶
2092 Meeting between Ofcom and Epilepsy Society, April 2023. 376
Section 20 False communications¶
Summary analysis for false communication offence: how harm manifests online and risk factors A person commits the false communications offence if they send a message, with no reasonable excuse to send it, that they know to be false and intend for that message to cause harm. The risks of harm to individuals are broad. Some individuals might experience distress and anxiety due to false communications shared with the intention to cause harm. Physical harm can also be caused by false communications, for instance from a hoax bomb threat. The rapid pace of development in generative AI technologies and models presents both a risk and an opportunity. At present, it appears that generative AI technologies and models are likely to enhance the risks of false communications, such as by breaking down barriers to distributing content, and reducing the costs of creating persuasive, false content, particularly through the creation and dissemination of deepfakes.2093 Service type risk factors: Social media services were identified as carrying higher risks of harm for false communications. These services can be used to spread disinformation in foreign influence operations, forms of which could be relevant to how the false communications offence can manifest. This is also true of messaging services with encryption, which can be used to spread disinformation due to their closed and encrypted nature. User base risk factors: Research suggests that gender, religious affiliation and ethnicity of users can be risk factors. Disinformation campaigns can often be gendered, with disinformation campaigns targeting women in power more often than men and may also target religious affiliations. Research shows that diaspora communities can also be particularly at risk of being victims of disinformation which could include false communications, falling within the parameters of this offence. Functionalities and recommender systems risk factors:
2093 Deepfakes are a specific type of media that involves the use of AI algorithms, particularly generative AI models, to modify videos, images, or audio to create realistic synthetic content. This is often done by superimposing the face of a person onto the body of another person in a video or image, as well as voice manipulation with lip syncing. Deepfakes are shared as user generated content on user-to-user services but could also potentially be created using functionalities present on user-to-user services. Deepfake technology is currently used to create content that can be harmful; however, we acknowledge that it may also have positive use cases. 377
Fake user profiles can be created by perpetrators of foreign influence operations to hide their identity and impersonate authoritative and high-profile sources, through which they can share false information. The ability to post content anonymously, which can be achieved by creating an anonymous user profile, can also be exploited in foreign influence operations. We believe that this evidence will also apply to the false communications offence. Research shows that the ability to post content is essential to this offence because it enables disinformation, and potentially false communications, to be disseminated. Direct messaging and encrypted messaging are other avenues that perpetrators may use to spread false communications. The ability to edit visual media, such as creating deepfakes, can also be exploited by perpetrators of the false communications offence.
Introduction¶
Relevant offences¶
2094 Section 180 of the Actsets out a number of exemptions from the false communications offence created. For example, a recognised news publisher cannot commit this offence and therefore content will not be illegal where it has been posted by a recognised news publisher. 2095 UK Government (Department for Digital, Culture, Media and Sport), 2022. Online safety law to be strengthened to stamp out illegal content. [accessed 18 November 2024]. 378
How false communication manifests online¶
Risks of harm to individuals presented by the false communication offence¶
2096 The false communications offence will replace the offences in section 127(2)(a) and (b) of the Communications Act 2003 and section 1(a)(iii) of the Malicious Communications Act 1988, which will be repealed by section 189 of the Online Safety Act.
2097 Department for Digital, Culture, Media and Sport, and Home Office, 2022. Online safety law to be strengthened to stamp out illegal content. [accessed 20 September 2023]. 2098 Hameleers, M., Powell, T.E., Van Der Meer, T.G.L.A, and Bos, L., 2020. A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated on Social Media. Political Communication, 37(2), pp.281-301. [accessed 20 September 2023]. (US study) 379
Evidence of risks of harm on user-to-user services¶
Risk factors: Service types¶
Social media services¶
2099 ‘Bots’ is an umbrella term that refers to a software application or automated tool that has been programmed by a person to carry out a specific or predefined task without any human intervention. 2100 ‘Like-farming’ refers to the use of fake pages on social media sites designed to artificially increase the popularity of a page, so it can be sold to buyers seeing accounts with large followings or for scam and fraud activity. 2101 ‘Click-farming’ refers to the practice of manually clicking on online adverts to increase the clickthrough rate value, boost engagement metrics, and inflate impressions. This activity can be carried out by bot accounts, as discussed here, or by large groups of workers. 2102 ‘Hashtag hijacking’ refers to the use of a hashtag for a purpose other than it was created – such as tagging a message containing undesirable or harmful content with a popular, but unrelated, hashtag to surface this content to a target audience.
2103 ‘Trend-jacking’ refers to when influencers, brands or organisations insert themselves into conversations online that are gaining a lot of attention – for example, by using associated hashtags or trending audios. 2104 US Department of Homeland Security, 2018. Social Media Bots Overview. 2105 Facebook, 2021. July 2021 Coordinated Inauthentic Behavior report. [accessed 1 September 2023]. 2106 European Centre For Disease Prevention And Control, 2021. Countering online vaccine misinformation in the EU/EEA. [accessed 20 September 2023]. 380
greater potential reach, thereby increasing the number of users who could encounter it (see Posting content sub-section for more information).
Messaging services¶
Risk factors: user base¶
User base size¶
User base demographics¶
2107 Gorksy, J., Riedl,M.J, and Woolley, S., 2021. The Disinformation Threat to Diaspora Communities in Encrypted Chat Apps, Tech Stream, Brookings Institute. [accessed 18 November 2024]. 2108 Demos (Judson, E., Atay, A., Krasodomski-Jones, A., Lasko-Skinner, R., and Smith, J.) and the US National Democratic Institute, 2020. Engendering Hate: The Contours of State-Aligned Gendered Disinformation Online. [accessed 6 March 2023]. 381
Ukrainian army lost an important battle, and doctored images, purporting to show her doing so.2109
Risk factors: Functionalities and recommender systems¶
User identification¶
Fake user profiles¶
Anonymous user profiles¶
User communication¶
Posting content¶
2109 Jankowicz, N., 2017. How disinformation became a new threat to women. [accessed 6 March 2023]. 2110 Gorksy, J., Riedl, M.J, and Woolley, S., 2021. The Disinformation Threat to Diaspora Communities in Encrypted Chat Apps. Tech Stream, Brookings Institute. [accessed 9 October 2023].
2111 Kotecha, S., 2021. Covid: Fake news 'causing UK South Asians to reject jab, BBC News, 15 January. [accessed 9 June 2023]. 2112 RAND Corporation, (Cohen, R.S., Beauchamp-Mustafaga, N., Cheravitch, J., Demus, A., Harold, S.W., Hornung, J.W., Jun, J., Schwille, M., Treyger, E, and Vest, N.), 2021. Combatting Foreign Disinformation on Social Media. [accessed 20 September 2023]. 382
spaces to create rumours and place fabricated content. This content starts in encrypted spaces before moving to conspiracy communities in closed and semi-closed networks, and then onto mainstream social media services. It is presumably posted on each of these spaces. This content is finally reported on by professional media sources, which might not always have “the training to deep dive into the provenance of the posts, images or videos they find online.”2113
2113 First Draft, (Wardle, C.), 2018. 5 Lessons for Reporting in an Age of Disinformation. [accessed 20 September 2023].
2114 Facebook, 2021. July 2021 Coordinated Inauthentic Behaviour Report. [accessed 12 June 2023]. 2115 Ofcom, 2024. Ofcom’s strategic approach to AI 2024/25, p.3. [accessed 20 November 2024]. 2116 Full Fact, 2023. No evidence that audio clip of Keir Starmer supposedly swearing at staff is genuine. [accessed 16 May 2024]. 2117 Fenimore Harper Communications (Beard, M.), 2024. Over 100 Deep-Faked Rishi Sunak Ads Found on Meta’s Advertising Platform. [accessed 16 May 2024]. 383
also found that the advertisements targeting the UK presented the scam as a government initiative and ran across several social media services and video-sharing platforms.2118
Direct messaging and encrypted messaging¶
Content editing¶
Editing visual media¶
Risk factors: Business models and commercial profiles¶
2118 Bureau of Investigative Journalism (Visser, F. and McIntyre, N.), 2024. Doctored Footage and Hijacked Accounts: Anatomy of a Deepfake Scam Network. [accessed 6 September 2024]
2119 Gorksy, J., Riedl, M.J, and Woolley, S., 2021. The Disinformation Threat to Diaspora Communities in Encrypted Chat Apps. Tech Stream, Brookings Institute. [accessed 18 November 2024]. 2120 Donovan, J, and Paris, B., 2019. Deepfakes and Cheap Fakes: The Manipulation of Audio and Visual Evidence. [accessed 20 September 2023]. 2121 BBC News, (Wakefield, J.), 2022. Deepfake presidents used in Russia-Ukraine war, 18 March. [accessed 6 March 2023]. 2122 Gleicher, N., 2022. Tweet on removal of Zelenskyy deepfake, published 16 March 2022. [accessed 6 March 2023]. 384
Section 21 Obscene content showing torture of humans and animals (the s.127(1) offence)¶
Summary analysis for the offence of obscene content depicting human and animal torture: How harms manifest online, and risk factors This section considers the evidence of risk factors for the obscene content offence (torture of humans and animals). This can manifest as pre-recorded content (images and video) which can cause significant levels of distress. There are several risk factors which may be associated with the offence, such as the following. Service type risk factors: There is some evidence that social media services, and to a lesser extent discussion forums and file-sharing and file-storage services may pose a higher risk of harm connected to this offence, in that they allow users to share content with others, in some cases gaining wide reach, and to engage with the content and other users. Since this content can be publicly available on user-to-user services, it is likely that it can also be discovered through search results on search services. Functionalities and recommender system risk factors: Any service type which allows the posting of images or videos, or the reposting and forwarding of this content could be a risk factor for users of online services coming across this content. Similarly, the ability to share hyperlinks could mean users risk being exposed to content hosted on other services. There is some evidence that content recommendation systems may play a role in surfacing obscene content online, risking users being harmed despite not actively looking for this type of content. Business model and commercial profiles risk factors: It is possible that certain business models could be a risk factor for the offence: content creators may be incentivised to post more extreme content, some of which could depict obscene torture of humans or animals, in order to maximise engagement and associated revenues.
Introduction¶
Relevant offence¶
Torture of humans and animals¶
Content harmful to children¶
How the s.127(1) offence manifests online¶
animals in the preceding Register of Risks section on the animal cruelty offence, because its publication may amount to use of the platform for committing the offence and may facilitate further offences.
Risks of harm to users¶
2123 Section 38 of the Criminal Justice and Licensing (Scotland) Act 2010. 2124 Ofcom, Online Experiences Tracker, 2024. [accessed 18 November 2024]. Fieldwork was carried out in a four-week period in May and June 2024. Note that in our November 2023 and August 2024 Consultations we referred to previous iteration of this research (Waves 4 and 5).
2125 Ofcom, Online Experiences Tracker, 2024. 2126 Ofcom, Online Experiences Tracker, 2024. 2127 Ofcom, Online Experiences Tracker, 2024. 2128 Ofcom, Online Experiences Tracker, 2024. 2129 Ofcom, Online Experiences Tracker, 2024. For this question the number of respondents was much smaller than others (62), which may mean this is not fully representative. 387
Evidence of risk factors on user-to-user services¶
Risk factors: service type¶
Social media services¶
2130 McGuirk, L.Ryan and Alleyne, E. 2024. “Liking,” “Commenting,” and “Reposting”: Psychological factors associated to online animal abuse, Society & Animals, pp.14 and 17. Note this study is in pre-print (not yet fully published), and used a non-representative sample through a survey of self-reported attitudes. It should therefore be interpreted only as a potential indication of the variance in attitudes towards human and animal abuse content.
2131 Stubbs, J. E., Nicklin, L. L., Wilsdon, L., and Lloyd, J. 2024. Investigating the experience of viewing extreme real-world violence online: Naturalistic evidence from an online discussion forum. New Media and Society, 26, 3876-3894. [accessed 19 November 2024]. 388
Messaging services¶
User-to-user pornography services¶
Discussion forums or chat rooms¶
2132 Social Media Animal Cruelty Coalition, 2021. Making money from misery. [accessed 19 November 2024]. As noted in the previous section, SMACC’s publications are referenced here as an indicator that potential harmful content (including that which could constitute the obscene torture content offence) exists online, not as an indicator of the scale of the issue. 2133 Dahl, K, 2018. Exploitation of the internet? The morality of watching death online, The Guardian 12 October. [accessed 21 June 2024]; Stephen, B. 2019. Reddit bans r/watchpeopledie in the wake of the New Zealand mosque massacres, The Verge, 15 March. [accessed 21 June 2024]. 2134 Lorenz, T., 2023. Instagram users are being served gory videos of killing and torture, The Washington Post, 26 February. [accessed 21 June 2024]. 2135 Stubbs, J. E., Nicklin, L. L., Wilsdon, L., and Lloyd, J. 2024. Investigating the experience of viewing extreme real-world violence online: Naturalistic evidence from an online discussion forum. New Media and Society, 26, 3876-3894. [accessed 19 November 2024].
2136 Institute for Strategic Dialogue, 2023. Gore and violent extremism: how extremist groups exploit ‘gore’ sites to view and share terrorist material. [accessed 21 June 2024]. 2137 The Human Digital study noted above analysed 10 of these sites, and include screenshots (images have been removed from the screenshots, but video titles are uncensored and may be distressing). 389
may be shocking or distressing to some individuals without necessarily being s.127(1) content.
File-storage and file-sharing services¶
Risk factors: user base¶
User base size¶
User base demographics¶
Services which allow child users¶
2138 Nicklin, L. L., Swain, E. and Lloyd, J. 2020. Reactions to unsolicited violent, and sexual, explicit media content shared over social media: gender differences and links with prior exposure, International Journal of Environmental Research and Public Health 17. [accessed 19 November 2024]. Note this was a relatively small-scale, non-representative survey (225 survey participants, of whom three-quarters were women), which did not present many options for respondents to choose from when considering their reaction to content. However, it does suggest gender may be a factor. The same study also considered the impact of prior exposure to this type of content.
2139 Stubbs, J. E., Nicklin, L. L., Wilsdon, L., and Lloyd, J. 2024. Investigating the experience of viewing extreme real-world violence online: Naturalistic evidence from an online discussion forum. New Media and Society, 26, 3876-3894. [accessed 19 November 2024]. 2140 McDonald et al 2017. The role of callous/unemotional traits in mediating the association between animal abuse exposure and behaviour problems among children exposed to intimate partner violence, Child Abuse & Neglect 72, 421- 432. [accessed 19 November 2024]. 390
acts of animal cruelty2141 or other crimes.2142 This research does not specifically refer to children witnessing animal cruelty online, and in some cases the conclusions are drawn from a small or non-representative sample. Nevertheless, the evidence does suggest that there is potential for harm. Also, given there is evidence that children are seeing this type of content online,2143 it is a reasonable assumption that this is one mechanism by which they are exposed to it and experience harm. They may also be disproportionately affected by it (particularly psychologically), especially where it is extreme cruelty.
Risk factors: functionalities and recommender systems¶
User networking¶
User connections¶
Group messaging¶
User communication¶
Posting images or videos¶
2141 Thompson, K. L. and Gullone, E. 2006. An investigation into the association between the witnessing of animal abuse and adolescents' behavior toward animals, Society and Animals 14, 221-244. [accessed 19 November 2024]. Note that this
2142 Johnson, S. A. 2018. Animal cruelty, pet abuse & violence: the missed dangerous connection, Forensic Research & Criminology International Journal. [accessed 19 November 2024]. 2143 RSPCA, 2018. The RSPCA’s Generation Kind, p.2, and footnote 2 [accessed 9 May 2024]. 2144 For instance, it was widely reported that one of the killers of Brianna Ghey, prior to her crime, watched ‘real’ murders and torture on the dark web: Gawne, E. and PA Media, 2024. Brianna Ghey inquest to look into killer's school transfer, BBC News, 11 April; Cobham, T. 2024 Kill lists, Sweeney Todd and the dark web: how torture-obsessed teenagers plotted Brianna Ghey's murder, The Independent, 2 February. [both accessed 21 June 2024].
2145 BBC, 2023. Monkey Haters (documentary) [accessed 27 March 2024]. 2146 Social Media Animal Cruelty Coalition, 2021. Making money from misery; Social Media Animal Cruelty Coalition, 2022b. Teasing as Torture, [both accessed 19 November 2024]. 391
Reposting or forwarding of content¶
Commenting on content¶
Content exploring factors¶
Searching for user-generated content¶
Hyperlinking¶
Recommender systems¶
Content recommender systems¶
2147 Institute for Strategic Dialogue, 2023. Gore and violent extremism: how extremist groups exploit ‘gore’ sites to view and share terrorist material [accessed 21 June 2024]. 2148 Alvarez, M. 2017. Online spectatorship of death and dying: pleasure, purpose and community in BestGore.com, Participations: Journal of Audience and Reception Studies. [accessed 19 November 2024]. 392
cruelty offence section. Some of this animal cruelty content may include obscene animal torture content that would constitute the s.127(1) offence.
Network recommender systems¶
Risk factors: Business models and commercial profiles¶
2149 Lorenz, T., 2023. Instagram users are being served gory videos of killing and torture, The Washington Post, 26 February. [accessed 21 June 2024]. 393
Section 22 Threatening communications¶
Warning: this chapter contains content that may be upsetting or distressing.
Introduction and relevant offence¶
How threatening communications manifest online¶
adults having experienced it.2150 2151 Amnesty found that of the one in five women in the UK who had experienced abuse and harassment online, 27% had been threatened with physical or sexual assault.2152 Evidence also suggests that the prevalence of threatening communications is high among certain groups, such as women in public roles.2153
Risks of harm to individuals presented by threatening communications offences online¶
2150 The 2020 figure comes from a panel of 10,093 US adults. Source: Pew Research. 2020. The state of online harassment. [accessed 28 September 2023]. 2151 Where possible, UK data has been used throughout this chapter. However, when this is limited, evidence for comparable cultures has been used, namely the US, Australia and Canada. Where evidence is not UK-based, this will be clearly stated.
2152 From a sample of 500 women aged 18 – 55 years old. Source: Amnesty International UK. Online abuse of women widespread in UK. [accessed 28 September 2023]. 2153 Amnesty International, 2018. Toxic Twitter – Women’s Experiences of violence and abuse on Twitter’, Chapter 3 in Online Violence against Women. [accessed 28 September 2023]; UNESCO (Posetti, J., Aboulez, N., Bontcheva, K., Harrison, J. and Waisbord, S.), 2020. Online violence Against Women Journalists. [accessed 28 September 2023]. 395
Section 23 Search services¶
Warning: this section contains content that may be upsetting or distressing.
Introduction¶
Service types¶
2154 Indexing is the process of collecting, parsing, and storing of data to facilitate fast and accurate information retrieval. 2155 The ‘clear web’ refers to the publicly accessible webpages that can be indexed by search engines. 2156 A search service is defined in section 3 of the Online Safety Act as an “internet service that is, or includes, a search engine”. A search engine “includes a service or functionality which enables a person to search some websites or databases (as well as a service or functionality which enables a person to search (in principle) all websites or databases)” but “does not include a service which enables a person to search just one website or database” (section 229 of the Online Safety Act). 396
Search services¶
2157 Sections 3 and 229 of the Online Safety Act. 2158 Refer to section 4 of the Online Safety Act and Schedule 1 to the Online Safety Act. 2159 Section 226(4), (5) and (13) of the Online Safety Act. Section 226 clarifies that there can only be one entity that is the provider of a search service. Please note, as set out in [Approach to Codes chapter] it is for the entities involved in the provision of a search service to seek their own advice as to whether they are the ‘provider’ of that service. 2160 The search engine index takes the output from the crawler and creates relevant data structures to support later searching within the search engine. The index can comprise document content, images, and metadata. An index will have many repeated refinement algorithms applied to increase its accuracy and relevance.
2161 In its advertising market study, the Competition and Markets Authority (CMA) said none of the contracts it had looked at allowed the downstream general search service to re-rank the search results they received from Google or Bing. Source: Competition and Markets Authority (CMA), 2020. Online platforms and digital advertising: Market study final report. [accessed 22 September 2023]. We discuss our position on who the ‘provider’ of a downstream general search service is in the [Approach to Codes chapter]. 397
• Vertical search services: Also known as ‘speciality search engines’, enable users to search for specific topics, or products or services offered by third party operators with which the provider of the vertical search service has a relevant arrangement. They operate differently from general search services. Rather than crawling the web and indexing webpages, they present users with search results only from selected websites or databases with which they have a contract. An API2162 or equivalent technical means is used to return the relevant content to users. Common vertical search services include price comparison sites.
Scope of Ofcom’s assessment of risk of harm from illegal content¶
23.11 • a user directly encountering illegal content in or via the search results2168 of a search service; or • harm that can occur to third-party individuals who have not directly encountered illegal content via search results but who may be harmed by the words or actions of those who have.
2162 Application Programming Interface (API) is a way for two or more computer programs to communicate with each other. 2163 Please note that this is not an exhaustive list, and service providers should obtain their own legal advice about the ways in which such GenAI content may fall within scope of the OSA’s search duties. 2164 For instance, a GenAI service could draw on more than one website or database by providing real-time information from plug-ins. 2165 For example, a search service could integrate a GenAI that provides a conversational summary of the results produced by the service's existing search engine. 2166 Section 98 of the Online Safety Act. 2167 Section 234 of the Online Safety Act.
2168 Section 57 of the Online Safety Act defines both ‘search content’ and ‘search results’. ‘Search results’ includes content presented to a user by operation of the search engine, and ‘search content’ is content encountered in or via search results. This definition captures content encountered as a result of interacting with search results, for example, by clicking on them and does not include content encountered through subsequent interactions with an internet service other than the search service. Paid-for advertisements, content on the website of a recognised news publisher and other journalistic content is excluded from the definition of ‘search content’. 398
How harm manifests on search services¶
Risk of harm to individuals presented by illegal content on search services¶
Risks of GenAI in Search¶
2169 Offences listed under Schedules 5 (’terrorism offences’), 6 (’child sexual exploitation and abuse offences), and 7 (priority offences) to the Act. 2170 Other offences are defined in section 59(5) of the Online Safety Act and includes all offences under UK law that are not priority offences, where (a) the victim or intended victim of the offence is an individual (or individuals); (b) the offence is created as a result of the Online Safety Act, another Act, an order of Council or other relevant instruments; (c) the offence does not concern the infringement of intellectual property rights, the safety or quality of goods, or the performance of a service by a person not qualified to perform it; and (d) the offence is not an offence under the Consumer Protection from Unfair Trading Regulations 2008.
2171 Communications offences are listed in Part 10 of the Act and include false communications offence, threatening communications offence, offences of sending or showing flashing images electronically, and offence of sending etc photograph or film of genitals. 2172 For example, for a detailed discussion of the harm caused by CSAM, please refer to the CSAM chapter; for a discussion of the harm caused by fraud refer to the Fraud and financial services offences chapter. 399
results page), and GenAI chatbots that ‘search’ the internet to provide responses to prompts. We consider both to have the potential to present illegal content to users.
Evidence of risk factors on search service¶
Risk factors: Service types¶
2173 Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T. and Fritz, M., 2023. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. [accessed 22 September 2023]. 2174 Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T. and Fritz, M., 2023. 2175 For further information on the characteristics, see: Introduction to the Register. 400
services have a far narrower scope, using an API (or equivalent technical means) to present content to users from pre-determined locations on the web.
• Vertical search services are likely to have a materially lower level of risk of harm than
general search services. This is because vertical search services typically only focus on a¶
specific segment of online content (such as particular products or services) and draw results via an API (or equivalent technical means) from pre-determined websites that may contain professional or curated content, rather than indexing sites from across the clear web. For example, a travel search site may be much less likely to present illegal content to a user, as the search feature on the site will be limited to hotels/flights/car rentals on the websites/databases of travel agents. This limited search functionality suggests that the risk of vertical search services providing access to illegal content is lower than with general search services.2176
General search services¶
2176 At the time of writing, we are unaware of any clear web vertical search services that draw their search result content from databases of illegal content. 2177 A 2022 report by Tech Against Terrorism highlights that 198 websites identified as being operated by terrorist actors existed on the surface web and were “often easily discoverable through search engines”. Source: Tech Against Terrorism, 2022. The Threat of Terrorist and Violent Extremist Operated-Websites. [accessed 22 September 2023]. 2178 A study commissioned by the Antisemitism Policy Trust and CST tested the SafeSearch function on Google Images searches for two search terms. Antisemitism Policy Trust, 2021. Unsafe Search: Why Google’s SafeSearch function is not fit for purpose. [accessed 26 September 2023]. 2179 From a dataset of 13,888 search instances from nearly 2,000 users looking for adult content, Ofcom found that 158 of these search instances from 40 users had a search term associated with extreme pornography before visiting an adult content site. This suggests that these search terms lead to content that at the very least contains a relevant matching description, and at worst is illegal extreme pornographic content. Source: Ofcom analysis of Ipsos Iris Clickstream Data, 15th September – 15th October 2021, UK, ages 15+.
2180 Analysing a data set of 9,078 searches that led users to ten of the most popular adult content sites, one in five (1,831) of these searches led directly to a video. 53 (0.6%) instances of these searches included terms associated with extreme pornography. Source: Ofcom analysis of Ipsos Iris Clickstream Data, 15th September – 15th October 2021, UK, ages 15+. 401
2181 Steel, C. M. S., 2015. Web-based child pornography: The global impact of deterrence efforts and its consumption on mobile platforms, Child Abuse & Neglect. [accessed 26 September 2023]. 2182 Westlake, B.G., Bouchard, M. and Girodat, A., 2017. How Obvious Is It? The Content of Child Sexual Exploitation Websites, Deviant Behavior, 38(3), pp. 282-293. [accessed 26 September 2023]. 2183 In a qualitative study on the pathways for accessing CSAM online in which interviews were conducted with 20 people who had viewed CSAM online and had been investigated by law enforcement. 2 respondents reported their initial exposure occurred via intentional searches on search engines. When asked about access methods generally, 13 respondents reported using search engines as a pathway to access CSAM. Note, we understand the data used in this study is from 2015. Source: Bailey, A., Allen, L., Stevens, E., Dervley, R., Findlater, D., and Wefers, S., 2022. Pathways and Prevention for Indecent Images of Children Offending: A Qualitative Study, Sexual Offending: Theory, Research, and Prevention, 17. [accessed 07 August 2024] 2184 Researchers for AntiToxin surfaced CSAM on Bing from a variety of search queries, including overt and less obvious queries. In response to the findings Microsoft made changes to Bing to remove the identified content and block some of the search terms used. Source: TechCrunch (Constine, J.), 2019. ‘Microsoft Bing not only shows child sexual abuse, it suggests it’. [accessed 07 August 2024]. 2185 Commission On Combating Synthetic Opioid Trafficking, 2022. Technical Appendixes. [accessed 17 October 2022]. 2186 Research has found that searches for stun guns, handguns and realistic imitation firearms returned results which appeared to be offers to supply prohibited weapons. Research has also shown that prohibited knives could be found on online marketplaces, which if the pages were indexed – which is highly likely – would also be accessible directly from search engines. Source: Which, 2022. Illegal weapons found for sale on Amazon, eBay, Wish and AliExpress.[accessed 26 September 2023]; Ofcom, 2023. Sale of prohibited items on search services. [accessed 26 September 2023]. 2187 Researchers found numerous search results and webpages offering to supply stolen credit card details to buyers, as well as guidance on how to commit fraud using this kind of information. Source: Ofcom, 2023. Articles and items for use in the commission of fraud – accessibility via search services. [accessed 26 September 2023]. 2188 A 2021 study investigating how search engines handle suicide search queries examined the top 20 search results returned in response to queries related to suicide. The study found that 22% of Microsoft Bing URLs, 19% of DuckDuckGo URLs and 7% of Google Search URLs were “harmful”, that is, assessed by researchers to encourage, promote or facilitate suicide, or contain discussions of suicide methods. The researchers also looked specifically at search results encouraging suicide and found that this was the case for 10% of Microsoft Bing URLs, 8% of DuckDuckGo URLs and 4% of Google Search URLs. Source: Borge, O., Cosgrove, V., Cryst, E., Grossman, S., Perkins, S., and Van Meter, A., 2021. How Search Engines Handle Suicide Queries. Journal of Online Trust and Safety, 1(1). [accessed 07 August 2024]; Research commissioned by Ofcom and conducted by the Network Contagion Research Institute exploring the accessibility of content promoting self-injurious behaviour via search services identified 1% of the search results assessed as ‘extreme’, referring to “posts that encourage others to engage in self-injurious behaviour. This included treating self-injurious behaviour as a game; explicit invitations to join in self-injurious behaviour activities; and apps or other interactive spaces making it easy for people to engage in self-injurious behaviour.” Note that the search queries tested were formulated with the intention that they would return such content if it was accessible via a search engine. Source: Ofcom, 2024. One Click Away: A Study on the Prevalence of Non-Suicidal Self Injury, Suicide, and Eating Disorder Content Accessible by Search Engines, [accessed 07 August 2024]. 402
suicide victims found that both these groups use search engines to access pro-suicide content and to research methods.2189
2189 A study involving self-harm patients admitted to hospital has found that this group would often go online to ‘research’ methods with the intention of planning an effective attempt. The type of content this group would view would include pro-suicide content, including consulting medical, academic and other ‘factual’ resources. Similarly, the most common behaviour among a study of 288 men aged 40-54 who had committed suicide was searching for information on suicide methods (10%). Of this 10%, a third (33%) had died by the method they were known to have had searched about. Source: Biddle, L., Derges, J., Goldsmith, C., Donovan, J.L. and Gunnell, D., 2018. Using the internet for suicide-related purposes: Contrasting findings from young people in the community and self-harm patients admitted to hospital, PLoS One, 13(5); The National Confidential Inquiry into Suicide and Safety in Mental Health (NCISH), 2021. Suicide by middle-aged men. [accessed 26 September 2023]. 2190 Journalists found that pornographic images featuring likenesses of female celebrities were the first images that Google and Bing surfaced in response to search queries including female celebrities names and words such as “deepfake”, “deepfake porn” or “fake nudes”. Source: NBC News (Tenbarge, K.), 2024. Google and Bing put nonconsensual deepfake porn at the top of some search results. [accessed 26 September 2024] 2191 My Image My Choice analysed deepfake abuse content accessible via Google search. On the top 40 websites dedicated to deepfake abuse they analysed, they identified over 270,000 videos which had gained over 4 billion views. Google Search appeared to be was driving 68% of traffic to these sites. They also found deepfake abuse content was returned even when using search queries not explicitly searching for it – with deepfake abuse content often making up the top and majority of the results on the page returned. Source: My Image My Choice, 2024. Deepfake Abuse: Landscape Analysis 2023-24. [accessed 14 August 2024] 2192 Research conducted by the Alliance for Securing Democracy, on behalf of Ofcom, focused on identifying the presence and prevalence of state-backed media outlets in search results pages for a range of general search services. It is important to note that the goal was to assess the potential risk of foreign interference by identifying types of variables – such as the topic searched for and language used – that regularly generated search results from state-linked websites, and not to assess whether the content of those search results would be illegal under the UK’s Foreign Interference Offence (National Security Act 2023). Source: Schafer, B. & Benzoni, P (Alliance for Securing Democracy, on behalf of Ofcom), 2023. Assessing the Risk of Foreign Influence in UK Search Results. [accessed 18 October 2024]. 403
keywords or numbers in an attempt to manipulate rankings in search results)2193 has been identified in research looking at how easily illegal content relating to fraud can be accessed via search services.2194
Risk factors: User base¶
User base size¶
User base demographics¶
2193 Google, n.d. Spam policies for Google web search. [accessed 26 September 2023]. 2194 Ofcom, 2023. Articles and items for use in the commission of fraud – accessibility via search services. [accessed 26 September 2023]. 404
Risk factors: functionalities¶
Search query inputs¶
Image search¶
Search prediction and personalisation¶
2195 For instance, research exploring the impact of exposure to potentially radicalizing information suggests that individuals who actively seek out terrorism content are at a higher risk of radicalisation. Source: Schuman, S., Clemmow, C., Rottweiler, B., Gill, P. 2024. Distinct patterns of incidental exposure to and active selection of radicalizing information indicate varying levels of support for violent extremism. [accessed 30 August 2024]. 2196 RAND, 2022. Commission On Combating Synthetic Opioid Trafficking. [accessed 26 September 2023]. 2197 We recommend readers refer to the related chapters in the U2U section of this volume. 405
of offences via general search services (including downstream search services), unless effective mitigations are in place to prevent this, or indexed content is blocked.
Risk factors: Business models and commercial profiles¶
Revenue models¶
2198 For more detail see the Register of Risks chapters ‘Encouraging or assisting suicide’ and ‘Encouraging or assisting serious self-harm’. Google search results as of 9th March 2022, examples provided to Ofcom by the Samaritans. 2199 A study found that Google recommended “queers should be shot” when the first two words were typed into its search box (Google stopped recommending such phrases a week after these examples were flagged). Loeb, J., 2018. Google is ‘promoting hate speech’, claims internet law expert, E&T, 22 January. [accessed 27 September 2023].; Similarly, the Antisemitism Policy Trust reported that Microsoft Bing directed users to hateful searches with the autocomplete “Jews are bastards”. Antisemitism Policy Trust response to 2022 Ofcom Call for Evidence: First phase of online safety regulation; Wired (Lapowsky, I), 2018. Google Autocomplete Still Makes Vile Suggestions. [accessed 18 November 2024]. Please note some of these sources are from 2018 and search services have made changes to their systems since then, but this highlights the risks present when mitigating measures are not in place or up to date. 2200 Ofcom, 2023. Articles and items for use in the commission of fraud – accessibility via search services. [accessed 26 September 2023]. 2201 In 2019, researchers for AntiToxin who showed that CSAM could be accessed with relative ease on Bing Search, also highlighted that some auto-complete suggestions led to illegal content, including from innocuous search queries not initially constructed to intentionally surface CSAM. In response to the findings Microsoft made changes to Bing to remove the identified content and block some of the search terms used. Source: Constine, J., 2019. ‘Microsoft Bing not only shows child sexual abuse, it suggests it’, TechCrunch, 10 January. [accessed 07 August 2024].
2202 Some search engines use a subscription model in lieu of advertising to generate revenue, although this is rare. 2203 Competition and Markets Authority, 2020. Online platforms and digital advertising. [accessed 26 September 2023]. 2204 Competition and Markets Authority, 2020; Australian Competition and Consumer Commission, 2021. Digital platform services inquiry. [accessed 26 September 2023]. 406
coercive control.2205 Although these devices are illegal, the research suggests that they are sometimes marketed as parental safeguarding tools.2206
Growth strategy¶
Commercial profile¶
2205 Sugiura, L., Blackbourn, D., Button, M., Hawkins, C., Tapley, J., Frederick, B., Nurse, J. R. C. and Belen-Salam, R., 2021. Computer Misuse as a Facilitator of Domestic Abuse. [accessed 26 September 2023]. 2206 Sugiura, L., Blackbourn, D., Button, M., Hawkins, C., Tapley, J., Frederick, B., Nurse, J. R. C. and Belen-Salam, R., 2021.
2207 Bradshaw, S, 2019, Disinformation Optimised: Gaming Search Engine Algorithms to Amplify Junk News, Internet Policy Review, 8(4). [accessed 26 September 2023]. 2208 Google Threat Analysis Group (Huntley, S.), 2022. TAG Bulletin: Q2 2022. [accessed 26 September 2023]. 2209 We have analysed evidence of how services with low capacity, or at early stage, may increase the risk of certain illegal harms on U2U services. We consider that the same reasoning broadly applies for all harms on U2U services (see Part 1: User-to-user services chapters) and also applies to search services. 407
that the risk profile of a low capacity, early-stage service, that is also a downstream search service, may vary from that described at 24.53. It will be the responsibility of the provider of the relevant search service to assess any factors that impact the risk profile of that service.
Section 24 Governance, systems and processes¶
Introduction¶
Definitions¶
2210 For example, content moderation systems can play a crucial role for services to detect and remove illegal content swiftly, such as in cases of terrorist acts being livestreamed on a U2U service. 2211 Milliman, 2021. Report on principles-based best practices for online safety Governance and Risk Management. This report was commissioned by Ofcom. This definition aligns with Milliman’s description of governance as made up of the concepts of individual and overall accountability, non-executive oversight, independent executive oversight, oversight of risk strategy and appetite, monitoring of the effectiveness of risk management, effective communication of risk and setting an appropriate risk culture and aligned incentives. We consider that in the context of online safety, governance relates more broadly to structures which work to ensure that decisions are aligned with user safety at all levels of an organisation.
2212 Section 236 of the Online Safety Act. 409
purposes of this risk assessment, we interpret this to mean any series of actions taken by a service provider, including actions that mitigate the risks of harm arising from illegal content being encountered that may not have been addressed elsewhere in the Register of Risks.
Evidence of risks of harm to individuals arising from governance, systems and processes¶
Governance (U2U and Search)¶
Governance arrangements¶
2213 OECD, 2023. G20/OECD Principles of Corporate Governance. [accessed 11 November 2024]; Milliman, 2021. Report on principles-based best practices for online safety Governance and Risk Management. [accessed 11 November 2024]. 410
with a more risk-averse attitude towards investments, as non-executive directors may be more concerned about their reputations.2214
Senior accountability and responsibility¶
2214Akbar, S., Kharabsheh, B., Poletti Hughes, J. and Shah, SZA., 2017. Board Structure and Corporate Risk Taking in the UK Financial Sector, International Review of Financial Analysis, Volume 50, pp. 101-110. [accessed 11 November 2024]. 2215 The Health and Safety Executive offers several case studies of negative safety consequences when board members do not lead effectively on health and safety management. Source: Health and Safety Executive, n.d. Case studies: When leadership falls short. [accessed 20 November 2024]. 2216 An analysis of past incidents [including major safety incidents in high hazard industries] by the OECD indicates that inadequate leadership have been a recurrent feature, “including the monitoring of safety performance indicators at Board level”. Source: The Organisation for Economic Co-operation and Development, 2012. Corporate Governance for Process Safety: OECD Guidance for Senior Leaders in High Hazard Industries. [accessed 19 September 2023]. 2217 This includes lawsuits filed against Boeing following the crashes of two 737 MAX airplanes in 2018 and 2019, in which shareholders claimed that a failure of the board to account for safety risks contributed to fatality events: “safety was no longer a subject of Board discussion, and there was no mechanism within Boeing by which safety concerns… were elevated to the Board or to any Board committee.” Source: The Washington Post, 2021. Verified Amended Consolidated Complaint. [accessed 5 May 2023].
2218 The importance of individual senior accountability is stressed within the Three Lines Defence model that has been implemented across many sectors and was drawn on Milliman in their analysis. Source: Milliman, 2021. Report on principles-based best practices for online safety Governance and Risk Management. p.18. 2219 Health and Safety Executive, 2013. Leading health and safety at work: Actions for directors, board members, business owners and organisations of all sizes. [accessed 24 August 2023]. 411
evaluate it more carefully.2220 Evaluation of a similar regime in the UK banking sector, introduced to hold senior management to account for failures that occurred on their watch, found that the majority of senior managers and firms which reported these had brought about positive and meaningful changes to behaviour in the industry.2221 Furthermore, senior leadership failures in financial services in relation to the 2008 financial crisis are taken as a case study in in the risks of reduced oversight and subsequent excessive risk-taking.2222
Internal assurance and compliance functions¶
2220 Note that this study has a relatively small sample size of 41 interviews with accountable persons. Source: Elizabeth Sheedy and Dominic Canestrari-Soh, 2023. Does executive accountability enhance risk management and risk culture?, Accounting & Finance, 63(4). [accessed 17 April 2024]. 2221 The Senior Managers and Certification Regime (SM&CR) for banks and insurers, launched in 2016 in the UK, requires the most senior decision-makers in firms to have clearly assigned responsibilities, and to be accountable for actions within their remit. Results from a survey of banks and insurers showed that 94% of senior managers and 96% of firms which responded reported that the SM&CR had brought about positive and meaningful changes to behaviour in industry. Source: Bank of England, 2020. Evaluation of the Senior Managers and Certification Regime. [accessed 17 April 2024]. 2222 Additionally, in the aftermath of the 2008 financial crisis, an inquiry into professional standards and culture of the banking sector by the Parliamentary Commission on Banking Standards concluded that many bankers had been allowed to operate with little accountability, and “claimed ignorance or hid behind collective decision-making”. Financial Conduct Authority, 2013. The FCA’s response to the Parliamentary Commission on Banking Standards. [accessed 3 May 2023]. 2223 ICO, What are the accountability and governance implications of AI?. [accessed 17 April 2024]. 2224 US National Institute of Standards and Technology (NIST), 2023. Artificial Intelligence Risk Management Framework. [accessed 17 April 2024].
2225 European Commission, 2022. Public consultation on the strengthening of the quality of corporate reporting and its enforcement: summary report. [accessed 03 May 2023]. 2226 A report by Ofcom on the Buffalo attack concluded that services should make efforts in product and engineering design processes to prevent the upload of terrorist content in an effort to prevent similar incidents in the future. Ofcom, 2022. The Buffalo Attack: Implications for Online Safety. [accessed 4 October 2023]. 412
Staff incentives, policies and processes¶
2227 Di Miceli Da Silveira, A., 2011. ‘Corporate Scandals of the Earlier 21st Century: Have We Learned the Lessons?’ [accessed 03 May 2023]. 2228 This includes the case study of petrochemical operators Petrobras and PdVSA, where systematic violation of internal controls and the absence of controls in key areas led to a failure to prevent or mitigate fraudulent activity. Source: Hamilton, S. and Micklethwait, A., 2006. Greed and corporate failure: The lessons from recent disasters. Springer; Omoteso, K., Obalola, M., 2014 ‘The Role of Auditing in the Management of Corporate Fraud’ in Said, R., Crowther, D., Amran, A. (eds.) Ethics, Governance and Corporate Crime: Challenges and Consequences, Emerald Group Publishing Limited, pp.129-151.; Burger, M., Taken Smith, K., Murphy Smith, L. and Wood, J., 2022. An examination of fraud risk at oil and gas companies, Journal of Forensic and Investigative Accounting, pp.74 – 85. 2229 2017, the FDA sent a warning letter to Indian pharmaceutical company Wockhardt, warning of repeated failures in oversight and controls that had contributed to the deletion of data related to failed tests. US Food & Drug Administration, 2017. Warning Letter, Morton Grove Pharmaceuticals, Inc. [accessed 4 October 2023]. 2230 “In the absence of any focus or controls on airplane safety, the Boeing Board pushed for achievement of production deadlines and competition with its chief rival, Airbus. In reviewing and approving the 737 MAX project, the Board never examined, considered, or questioned potential safety issues resulting from the re-design of the earlier generation 737 NG.” Source: Volkov Law Group, 2021. Boeing’s Board Governance Failures and the 737 MAX Safety Scandal (Part III of IV). [accessed 4 October 2023].
2231 In the case of Siemens, which in 2008 was subject to regulatory investigations for bribery, the failure to embed a programme of compliance and Code of conduct for staff has been cited as playing a “decisive role” in the scandal. Source: Primbs, M. and Wang, C., 2016. Notable Governance Failures: Enron, Siemens and Beyond. Comparative Corporate Governance and Financial Regulation. 3. [accessed 21 September 2023]. 2232Primbs, M. and Wang, C., 2016. 413
Moderation (U2U and Search)¶
Ineffective moderation¶
2233 Automated moderation technology can support the identification and removal of priority illegal content, either when it is uploaded or once it is on a service. This includes tools that can compare each piece of content against a database or list of known illegal content using methods such as hash-matching, URL detection, or text detection. Any content that matches existing content in such a list or database can then be flagged for further review or automatically removed. This type of moderation can be highly effective to identify and remove specific types of illegal material, particularly due to its advantages at scale. Due to the sheer volume of content that may be available on a service (particularly on larger services) human moderation often benefits from assistance from automated processes. 2234 A note on our automated moderation measures in our Codes of Practice (Volume 2) at the time of publication: We recognise that automated moderation systems and processes may be used to address several types of illegal harms, including child sexual abuse material, fraud, and terrorism. However, at the time of this publication, our approach focuses on the use of automated moderation systems, for both user-to-user and search services, that operate by detecting matches for known child sexual abuse material (CSAM). In our measures we did consider proposing the use of automated moderation, specifically keyword detection, to address content containing articles of fraud. However, we are still considering the most appropriate type of automated moderation technology to use to address this type of illegal behaviour. 2235 For example, research has identified that users sharing content potentially amounting to hate and terror offences – in this case, those on the ‘extreme right’ – have shown a preference for utilising online services perceived as having more limited content moderation as well as utilising multiple services in an attempt to evade content moderation efforts of any one particular service provider. Source: The Institute for Strategic Dialogue (O’Connor, C.), 2021. Gaming and Extremism: The Extreme Right on Twitch. [accessed 11 November 2024].
2236 Tech Against Terrorism, 2023. TCAP Insights. Patterns of online terrorist exploitation. [accessed 20 November 2024]. 2237 Molly Rose Foundation, 2023. Preventable yet pervasive: The prevalence and characteristics of harmful content, including suicide and self-harm material, on Instagram, TikTok and Pinterest. [accessed 17 April 2024]. 414
in the politics of hate and has enabled extremist networks to propagandise, network and organise”, which could eventually result in individuals being exposed to radicalisation.2238
Resourcing and time constraints¶
2238 HOPE not hate and the Antisemitism Policy Trust, 2021. Antisemitism and Misogyny: Overlap and Interplay. [accessed 4 October 2023]. 2239 For example, users have been found to send certain emojis to indicate that they sell drugs in place of text on a social media service. Source: Moyle, L., Childs, A., Coomer, R. and Barrat, M.J., 2019. #Drugsforsale: An exploration of the use of social media and encrypted messaging apps to supply and access drugs, International Journal of Drug Policy, 63, 101-110. [accessed 3 June 2023]. 2240 Ofcom, 2023. Content Moderation in user-to-user online services. 2241 A report by CASM Technology and ISD found a major increase in the number of antisemitic posts, coinciding with a reduction in content moderation staff at one social media service, saying the analysis demonstrates “the broader and longer-term impact that platforms de-prioritising content moderation can have on the spread of online hate.” Note: On its methodology, the report comments there are ‘inherent challenges in training language models on as nuanced a topic as antisemitism, but this architecture is evaluated to operate with an accuracy of 76%. Source: CASM Technology and the Institute for Strategic Dialogue (ISD), 2023. Antisemitism on Twitter Before and After Elon Musk’s Acquisition. [accessed 17 April 2024]. 2242 In late 2022, ADL noted an increase in antisemitic content on the same service and a decrease in the moderation of antisemitic posts. Source: The Anti-Defamation League, 2022. Extremists, Far Right Figures Exploit Recent Changes to Twitter. [accessed 4 October 2023].
2243 Demos (Krasodomski-Jones, A.), 2020. Everything in Moderation: Platforms, communities and users in a healthy online environment. [accessed 17 April 2024]. 2244 Google, 2020. Information quality & content moderation. [accessed 20 November 2024]. 2245 The Glitch response to Ofcom’s 2022 Call for Evidence states that without “comprehensive training for moderators about online gender-based violence and different tactics of online abuse, and how abuse specifically targets women, Black 415
tech abuse can often be hard to recognise without an understanding of the broader context of domestic abuse and coercive control.”2246 As part of our video-sharing platform (VSP) regulation, we said that although providers for most of the regulated services we examined did have training materials for content moderators in place, including definitions of prohibited content, it is also important to build moderators’ awareness of the cultural, linguistic, historical, and political context in the UK, to help them protect UK users.2247
Search design (Search)¶
Predictive search functionalities¶
and minoritised communities and users with intersecting identities is paramount – without this moderation risks being ineffective, inequitable and/or discriminatory”. Glitch is a UK charity which exists to end online abuse and to increase digital citizenship across all online users. The Antisemitism Policy Trust response to Ofcom’s 2022 Call for Evidence says that without quality assurance and independent scrutiny of moderator training, it risks not being effective at responding to harm. The Antisemitism Policy Trust is a charity that works to educate and empower parliamentarians and policy makers to address antisemitism. In the NSPCC response to Ofcom’s 2022 Call for Evidence it says that “Some online service providers rely primarily on volunteers in their own communities to self-police, with administrators doing occasional checks. There is a real concern here that the subjective nature of this moderation process creates inconsistency and potential for gaps in protection of users”. The NSPCC is a UK charity with over 130 years in experience safeguarding children from harms. 2246 Refuge response to Ofcom’s 2022 Call for Evidence. 2247 Ofcom, 2023. Regulating Video-Sharing Platforms (VSPs). Our first 2023 report: What we’ve learnt about VSPs’ user policies. 2248 Ofcom, 2022. Ofcom’s first year of video-sharing platform regulation.
2249 Predictive search functionalities are algorithmic features embedded in the search bar of a search service. 2250 Samaritans’ response to Ofcom’s 2022 Call for Evidence. Samaritans is the UK and Ireland’s largest suicide prevention charity. 2251 The Antisemitism Policy Trust response to Ofcom’s 2022 Call for Evidence noted that “Google’s Search autocomplete algorithm has been found to suggest antisemitic, racist and sexist content to users and that Microsoft Bing has been found to direct users to hateful searches via autocomplete”. 416
found, through their systems, to direct people to hate material and racist content that is legal but can easily direct users to more extreme and illegal content when they follow search prompts.”2252 The latter is also substantiated by an investigation by The Observer in 2016.2253 There is also research that points to similar risks regarding child sexual abuse material (CSAM).2254 2255 A 2019 report by the Antisemitism Policy Trust and Community Security Trust found that Google’s removal of a specific antisemitic predictive search suggestion resulted in 10% (one in ten) fewer search requests related to that suggestion in the 12 months following its removal compared to the 12 months prior.2256 This indicates that the removal of suggestions deemed to present an illegal content risk could materially reduce the likelihood of users encountering illegal content in search results.
Web indexing¶
2252Antisemitism Policy Trust response to Ofcom’s 2022 Call for Evidence. 2253 Note: The Guardian later reported that Google had altered autocomplete in response and removed some suggestions, while others remained. Source: Cadwalladr, C., 2016. Google, democracy and the truth about internet search, The Observer, 4 December. [accessed 4 October 2023]; Gibbs, S., 2016, Google alters search autocomplete to remove ‘are Jews evil’ suggestion, The Guardian, 5 December. [accessed 4 October 2023]. 2254 Note: Microsoft removed the offending suggestions in response. Source: TechCrunch (Constine, J.), 2019. Microsoft Bing not only shows child sexual abuse, it suggests it. [accessed 4 October 2023]. 2255 The WeProtect Global Alliance notes that algorithms that suggest CSAM can have the effect of “encouraging or inspiring new offending, as well as increasing re-victimisation of those victims of abuse”. WeProtect, 2020. Voluntary Principles to Counter Online Child sexual Exploitation and abuse. [accessed 4 October 2023]. 2256 Antisemitism Policy Trust, Community Security Trust (Stephens-Davidowitz, S.). 2019. Hidden Hate: What Google searches tell us about antisemitism today. [accessed 11 October 2024]. 2257 Indexing is the process of collecting, parsing, and storing data to facilitate fast and accurate information retrieval. 2258 Steel, C.M.S, 2015. Web-based child pornography: The global impact of deterrence efforts and its consumption on mobile platforms. Child Abuse & Neglect, 44, pp.150-158. [accessed 4 October 2023]. 2259 A qualitative study on the pathways for accessing CSAM online conducted interviews with 20 people who had viewed CSAM online and had been investigated by law enforcement. When asked about their initial exposure, two of the respondents reported that initial exposure occurred through intentional searches on search engines, and when asked about access methods, 13 responded reported using search engines as a pathway to access CSAM. Source: Bailey, A., Allen, L., Stevens, E., Dervley, R., Findlater, D. and Wefers, S., 2022. Pathways and Prevention for Indecent Images of Children Offending: A Qualitative Study. Sexual Offending: Theory, Research, and Prevention, 17, pp.1-24. [accessed 4 October 2023]. 2260 The National Crime Agency carried out research on the availability of CSAM on mainstream search engines and found that access is discoverable within three clicks. UK Government, 2020. Interim code of practice on online child sexual exploitation and abuse. [accessed 4 October 2023]. 2261 The research by the Network Contagion Research Institute (NCRI) found that 22% of the 37,647 individual search results links they assessed across five search engine services contained content that celebrates, glorifies, or instructs self-injurious behaviour within a single click from the main search results page. The report defined self-injurious behaviour as non-suicidal self-injury, suicide, and eating disorders. The research found that 1,580 links were likely to be in scope (promoting self-injury) of extreme (encouraging others to engage in self-injurious behaviour). Source: Network Contagion Research Institute, 2024. One Click Away: A Study on the Prevalence of Non-Suicidal Self Injury, Suicide, and Eating Disorder Content Accessible by Search Engines. [accessed October 2024]. 417
consider it evidence of a clear risk. Further evidence on the risks of encountering illegal content via search services can be found in the Register of Risks chapter ‘Search’.
Recommender systems (U2U)¶
2262 A content recommender system is an algorithmic system which determines the relative ranking of an identified pool of content that includes regulated user-generated content from multiple users on content feeds. 2263 Ofcom, 2023. Evaluating recommender systems in relation to illegal and harmful content. [accessed 11 November 2024]. 2264 Global Internet Forum to Counter Terrorism, 2021. Content-Sharing Algorithms, Processes, and Positive Interventions Working Group: Part 1. [accessed 27 September 2023]. 2265 Ofcom, 2023. Evaluating recommender systems in relation to illegal and harmful content. [accessed 4 October 2023]. 2266 Cook, J. and Murdock, S., 2020. YouTube Is A Pedophile’s Paradise, HuffPost, 20 March. [accessed 4 October 2023]. 2267 Waters, G. and Postings, R., 2018. Spiders of the Caliphate: Mapping the Islamic State’s Global support network on Facebook. Counter Extremism Project; Ofcom, 2023. Evaluating recommender systems in relation to illegal and harmful content. [accessed 4 October 2023].
2268 Digital avatar accounts were used to examine how recommender systems affected user exposure to extremist content on different platforms. It found that while some platform recommender systems did disseminate extremist and so-called ‘fringe’ content, others did not, indicating the importance of different design choices on the risk of encountering harmful. 418
commented on the speed at which a service’s recommender system identified the account’s preferences, noting that a large video-sharing platform’s For You Page “rapidly identified our interest in suicide and self-harm related material, and we were quickly presented with a range of disturbing and potentially harmful videos.” 2269
Reporting and complaints (U2U and Search)¶
Accessibility of reporting/complaints functionalities¶
Source: Whittaker, J., Looney, S., Reed, A. and Votta, F., 2021. Recommender systems and the amplification of extremist content, Internet Policy Review, 10(2). [accessed 4 October 2023].
Preventable yet pervasive: The prevalence and characteristics of harmful content, including suicide and self-harm material, on Instagram, TikTok and Pinterest. [accessed 17 April 2024].
2270 Tech UK response to 2020 Video-Sharing Platform Regulation Call For Evidence , p.3. [accessed 31 August 2023]. 419
complain, or believe it will be difficult.2271 2272 This can hinder content takedown, for instance, in the case of CSEA.2273
Speed of action¶
2271 Many children do not know how to use reporting systems or find them difficult to use. Ofcom, 2023, Children’s Media Use and Attitudes study: Children’s Online Knowledge and Understanding Survey, QC57 2272 “Of the 91% of 8-17s who would tell someone if they saw something worrying or nasty online, only 6% would tell the website/app where they encountered the harmful content/behaviour about what they had seen. 35% of 12-17s said they knew how to use a reporting or flagging function, and of those only 14% said they had used it before” Source: Children and parents: Media Use and Attitudes Report 2022; Of the 88 responses to Ofcom’s Call for Evidence: First phase of online safety regulation referring to the need for user reporting to be easy to use for those with vulnerabilities, 14 responses related specifically to children. Some responses highlighted the lack of trust that children have due to problems being difficult to report, or their belief that nothing would be done. 2273 A report by the Canadian Centre for Child Protection analysed reporting functions across major social media sites and found that CSEA reporting is inaccessible on most, with a lack of specific reporting functions for CSEA, specifically: when reporting nudity; a lack of user (not just content) reporting features; less comprehensive reporting features on mobile compared to desktop apps; and an inability to add contextual information to reports, all deemed to contribute to a greater risk of harm. Source: The Canadian Centre for Child Protection, 2020. Reviewing child sexual abuse material reporting functions on popular platforms: executive summary. [accessed 4 October 2023]. 2274 Online services sometimes report that they struggle to identify activity as foreign interference because of a lack of information to determine whether it is state-linked. The adversarial nature of foreign interference makes attribution a constant challenge. For example, a recent tactic deployed by those engaging in foreign interference is to employ journalists, influencers or ‘dark’ PR firms to carry out their desired influence activities for them. Without third-party information (e.g. from journalists or information from governments) attribution is difficult. Source: Empirical Studies of Conflict Project (Martin, D. A., Shapiro, J. N., Ilhardt, J.), 2020. Empirical Studies of Conflict Project, Princeton University; Thomas, E., Thompson, N. and Wanless, A. 2020. The Challenges of Countering Influence Operations, Partnership for Countering Influence Operations, Policy Perspectives #2, Carnegie Endowment for International Peace. Trends in Online Influence Efforts. [accessed 4 October 2023]; Carnegie Endowment for International Peace (Thomas, E., Thompson, N. and Wanless, A.), 2020. The Challenges of Countering Influence Operations. [accessed 4 October 2023]. 2275 For example, the Integrity Institute highlighted that complaints may be ‘denied’ without context. Source: Integrity Institute response to May 2024 Consultation on Protecting Children from Harms Online, pp.9-10.
2276 Refuge response to Ofcom 2022 Call for Evidence: First phase of online safety regulation, pp.7-8. 2277 Ofcom, 2022. Just one in six young people flag harmful content online. [accessed 25 September 2023]. 2278 Users can often wait a long time to receive any information about their report; e.g., children and women who have suffered online abuse can wait months or years for any action to be taken, if it is taken at all. Children are dissuaded from reporting as the process ‘comes to nothing’ and they have to chase up reports. Source: Refuge, 2021. Unsocial Spaces. [accessed 4 October 2023]. 420
trauma they may feel.2279 A study with LGBT+ users found that they fear they will not be taken seriously when reporting content.2280 Children in particular are often dissuaded from submitting complaints about illegal content as they do not think anything will come of their complaint.2281
Service design and user support (U2U)¶
2279 Refuge notes that concerns about long waiting periods for content to be removed, once reported, can compound stress or trauma experienced in some instances, including online abuse. Source: Refuge, 2021. Unsocial Spaces. [accessed 4 October 2023]. 2280 Galop (Hubbard, L.), 2020. Online Hate Crime Report 2020: Challenging online homophobia, biphobia and transphobia. [accessed 4 October 2023]. 2281 Ofcom, 2021. Online Experiences Tracker. 2282 European Parliament, 2021. New rules adopted for quick and smooth removal of terrorist content online. [accessed 4 October 2023]; Tech Against Terrorism, 2021. The Online Regulation series: European Union (update) [accessed 4 October 2023]; European Commission, 2022. Terrorist content online. [accessed 4 October 2023].
2283 Service design, in its broadest sense, includes the design of all components that shape a user’s end-to-end experience with a service. These components can include the business model/decision-making structures, back-end systems and processes, the user interface, and off-platform interventions. In line with the illegal content safety duties, our recommendations in this area will focus on the following categories of measure identified in section 10(4) of the Act: “design of functionalities, algorithms and other features”, “functionalities allowing users to control the content they encounter” and “user support measures” in the context of preventing users from encountering illegal content online. 421
Blocking, muting and account strikes procedures¶
2284 66% of respondents to a 2021 study from Thorn reacted to a harmful online experience by blocking the user and 27% muted the user. Source: Thorn, 2021. Responding to Online Threats: Perspectives on Disclosing, Reporting, and Blocking. [accessed 22 October 2024]. 2285 Tokunaga, R. S. and Aune, K. S., 2017. Cyber-Defense: A Taxonomy of Tactics for Managing Cyberstalking. Journal of Interpersonal Violence, 32 (10). [accessed 24 October 2024]. 2286 Pen America, Online harassment Field Manual; Blocking, Muting and Restricting. [accessed 24 October 2024]. 2287 Thorn, 2021. Responding to online threats: minors' perspectives on disclosing, reporting and blocking. [accessed 21 September 2023]. 2288 A survey carried out by Refuge found that 15% of female survivors who had experienced harassment and abuse online said: ‘the abuse worsened when they reported the perpetrator or took an action to mitigate the abuse, such as blocking the perpetrator online’. The survey was carried out with 2,264 UK adults, including 1,158 females. 36% of females reported experiencing at least one behaviour suggestive of online abuse or harassment. Source: Unsocial Spaces. [accessed 4 October 2023]. 2289 Volteface found that profiles suspected of supplying drugs on social media sites had multiple back-up accounts in case their current active account was closed. Source: Volteface, 2019. DM for details: Selling drugs in the age of social media. [accessed 17 October 2023]. 2290 Business Insider, 2015. Transgender Tinder Users Reported and Banned. [accessed 4 October 2023]. 2291 A Meta report into intent of CSAM sharers showed “patterns of persistent, conscious engagement with CSAM and other minor-sexualising content if it existed” when 200 accounts that were reported to NCMEC were analysed. Source: Meta, 2021. Understanding the intentions of Child Sexual Abuse Material (CSAM) sharers. [accessed 27 March 2023]. 2292 One research study found that, of a group of 78 perpetrators of child sexual abuse, 42% had attempted to collect all images in an abuse series, or of an individual, indicating a likelihood of persistent offending. Source: Steel, M.S., Newman, E., O'Rourke, S. and Quayle, E., 2021. Collecting and viewing behaviors of child sexual exploitation material offenders — University of Edinburgh Research Explorer. [accessed 4 October 2023].
2293 This report refers to internal documents from the terrorist group that highlighted the importance of its remaining an influential presence on social media to continue spreading the ISIS message. Source: Berger, J. M. and Morgan, J., 2015. The ISIS Twitter Census: Defining and describing the population of ISIS supporters on Twitter. [accessed 4 October 2023]. 422
behaviour in general.2294 Similarly, proscribed terrorist organisations with access to services can lead to the spreading of terrorism content. This in turn can increase the risks of harm to individuals encountering illegal content, as pointed out in the Terrorism chapter.
Safety default settings¶
2294 In an enforcement update TikTok shared that 90% of repeat violators violate use of the same feature consistently, and over 75% violate the same policy category repeatedly. This demonstrates that continued access for users who commit some kinds of illegal harms poses a high risk, for services of those kinds, of illegal harms being repeated. Source: TikTok (de Bailliencourt, J.), 2023. Supporting creators with an updated account enforcement system. [accessed 27 March 2023]. 2295 See the Grooming chapter of the Register of Risks. 2296Office for National Statistics, 2021. Children’s online behaviour in England and Wales: year ending 2020. [accessed 15 October 2024]. 2297 This could include, for example, illegal harm related to threats, harassment, stalking, abuse (including hate), coercive and controlling behaviour (CCB), and even terrorism.
2299 Thaler, R. H., Sunstein, C. R., and Balz, J. P., 2013, Choice architecture. In E. Shafir (Ed.), The behavioral foundations of public policy (pp. 428-439). Princeton, NJ: Princeton University Press. 2300 Ofcom, 2024. Behavioural insights to empower social media users. Testing tools to help users control what they see, Behavioural Insights Discussion Paper. 423
Labelling of user profiles¶
Terms of service and publicly available statements (U2U and Search)¶
2301 Verification and labelling schemes refers to schemes operated by services to verify the accounts of certain users, such as notable users or those who subscribe to a paid-for scheme. These schemes may involve labelling a user’s profile to indicate that it is verified. Verification in this context may take different forms but usually involves the service carrying out a process before a user profile is labelled as being part of a particular scheme. 2302 Farrell, L, 2023. Martin Lewis issues warning over fake Twitter account after major change to app. Daily Record, 4 April. [accessed 4 October 2023]. 2303 Sardarizadeh, S, 2022. Twitter chaos after wave of blue tick impersonations. BBC News, 12 November. [accessed 4 October 2023]. 2304 Our Media Use and Attitudes trackers look at the experience of UK users online and their attitudes towards this. Participants were asked to judge whether a social media post appeared to be genuine and why they came to their conclusion. This research involved showing social media users a real social media post and asking them if they thought the post was genuine or not, and to give their reasons for doing so. Of the 44% of adult social media users who correctly identified a Money Saving Expert Facebook post as genuine, 51% identified the verification tick as among their reasons for making this judgement. Source: Ofcom, 2023. Adults’ Media Use and Attitudes report 2023. [accessed 4 October 2023]. 2305 Respondents aged 12-17 who go online were shown a real NHS Instagram post and asked whether they thought it was genuine or not, and to give their reasons for their opinion. Of the 80% of who correctly recognised that it was a genuine post, nearly three in ten identified the inclusion of a verification tick as one of the factors behind this judgment. Ofcom, 2023. Children and Parents: Media Use and Attitudes. [accessed 4 October 2023]. 2306 Respondents were asked “when using social media platforms, how often, if at all, do you look out for these kinds of labels (e.g. a tick on a profile) when deciding to follow or interact with an account?”. Nearly three in ten respondents (28%) claimed they ‘always’ (2%), ‘often’ (7%) or ‘sometimes’ (19%) used verification labels when deciding to follow or interact with an account on social media. A further fifth (22%) said they used these labels ‘rarely’, suggesting that these respondents may find verification labels helpful in certain contexts or situations. Ofcom, 2023. Verification schemes to label accounts poll via YouGov panel. [accessed 4 October 2023]. 2307 On defining terms of service, the Act includes duties that apply in relation to: a) U2U services’ terms of service (‘terms’), meaning “all documents (whatever they are called) comprising the contract for use of the service (or of part of it) by United Kingdom users” (source: Section 236 of the Online Safety Act 2023); b) search services’ publicly available statements (‘statements’): search services are required to produce and make available to members of the public in the United Kingdom, a statement setting out certain information about how they operate (source: section 236 of the Online Safety Act 2023); and c) combined services, which have both functionalities, are permitted to set out what would be required in a publicly available statement in terms of service instead (Source: Section 25(2)(a) of the Online Safety Act 2023). 424
accessible to all users, including children. We consider that a service’s terms of service and publicly available statements should be able to be understood by all, so that they can make better-informed choices about what services to use, and how to stay safe online.2308 It is reasonable to infer that this should reduce users’ risk of being exposed to illegal content on a service. Further information as to how services can present their terms of service and publicly available statements effectively can be found in our Codes of Practice (Volume 4).
2308 Several stakeholders mentioned in response to our November 2023 consultation that consideration should be given to ensuring that terms and statements are clear and accessible for children, and users who have disabilities or learning difficulties. Source: Scottish Government response to November 2023 Consultation, p.9. Scottish Government response to May 2024 Consultation, p.17; Parenting Focus response to May 2024 Consultation, p.31. Children’s Commissioner for England response to November 2023 Consultation, p.22. Ofcom Advisory Council for Northern Ireland response to November 2023 Consultation, p.9; Mencap response to November 2023 Consultation, p.12; The Cyber Helpline response to November 2023 Consultation, p.16; Glitch response to November 2023 Consultation, p.10. Children’s Commissioner for England response to November 2023 Consultation, p.22. 2309 The ICO’s Age Appropriate Design Code states that for terms of service to be accessible to children, they must be prominent, visible and easy to find. Source: ICO, 2020. Age appropriate design: a code of practice for online services. [accessed 17 April 2024]. 2310 Ofcom calculated a 'reading ease’ score for the terms of service of the providers in scope of our video sharing platform regulation. All but one was assessed as being “difficult to read and best understood by high-school graduates.” Source: Ofcom, 2023. Regulating video sharing platforms (VSPs) - Our first 2023 report: What we’ve learnt about VSPs’ user policies. 2311 5Rights for example reported that when they looked at 123 privacy policies for websites likely to be accessed by children, only 9 (7%) had a specific policy targeted at children. Source: 5Rights, 2021. Tick to Agree - Age appropriate presentation of published terms. [accessed 17 April 2024]. 2312 Ofcom, 2023. Regulating video sharing platforms (VSPs) - Our first 2023 report: What we’ve learnt about VSPs’ user policies 2313 Only 6% of UK internet users aged 16+ said they always read terms and conditions. Source: Ofcom, 2022. Adults' Media Literacy Tracker (table 66). And 33% of UK internet users aged 16-24 reported having ever needed to access social media terms and conditions. Source: Ofcom, 2023. Platform Terms and Accessibility (Q1). 2314 The Behavioural Insights Team, 2019. Best practice guide: Improving consumer understanding of contractual terms and privacy policies: evidence-based actions for businesses. p.12 [accessed 7 October 2024]. We note that BIT found that using icons with long blocks of text did not work very well. They compared a long privacy policy with no icons to an identical policy that was illustrated with over 20 icons but found that icons did not help customers understand the policy better in that case. This points to the importance of combining icons with short, easy to understand information.
2315 Danish Competition and Consumer Authority, 2018. Improving the effectiveness of terms and conditions in online trade. Competitive Markets and Consumer Welfare, 15, p.5 [accessed 7 October 2024]. 425
User access (U2U)¶
2316 Ofcom research found that 18% of internet users aged 16-24 reported having had difficulty reading information online because the content was not keyboard navigable, or was difficult to navigate using a keyboard. The same proportion reported the same difficulty because the content was not compatible, or was difficult to use, with a screen reader or screen-reading technology. Source: Ofcom, 2023. Platform Terms and Accessibility (Q6).
2317 Web Aim, 2022. Keyboard accessibility. [accessed 7 October 2024]. 2318 Royal National Institute of Blind people, 2023. Screen reading software. [accessed 7 October 2024]. 2319 ‘User access’ refers to a user’s entry into a service and ability to use the functionalities present on that service. 426
A1 Glossary of terms¶
This glossary of terms contains definitions for terms used throughout the Register of Risks. These terms may also be referenced in other documents set out for consultation, such as Risk Profiles.
This glossary of terms explains how we have used some key words and phrases in the Register of Risks. It is intended to assist the reader, but to the extent that it simplifies, or is otherwise inconsistent with, any of the legal definitions set out in the Online Safety Act (the “Act”), the definitions in the Act prevail. In case of any conflict between terms used in this glossary and in any Code of Practice, the definition in the Code of Practice takes precedence.
General¶
Term Definition¶
Characteristic In respect of a regulated service, includes references to its functionalities, user base, business models, governance and other systems and processes.2320
Content Anything communicated by means of an internet service, whether publicly or privately, including written material or messages, oral communications, photographs, videos, visual images, music and data of any description.2321
Harm Means physical or psychological harm. References to harm presented by content, and any other reference to harm in relation to content, have the same meaning given to it by section 235 of the Act.2322
Illegal content Content that amounts to a relevant offence.
Kinds of illegal harms Refers to harm caused by different categories of relevant offences.
Act Means the Online Safety Act 2023.
Part 3 or regulated Refers to a search service that falls within the definition of section 4 of search service the Act.
Priority illegal content Content that amounts to a priority offence.
2320 Section 98(11) of the OS Act. 2321 Section 207(1) of the OS Act. 2322 Section 201 of the OS Act. 427
Priority offences Offences set out in Schedules 5 (Terrorism offences), 6 (CSEA offences) and 7 (Priority offences) to the OS Act.
Relevant offence Means a priority offence or an offence within the meaning of section 59(4) of the OS Act. This includes both priority offences and non-priority offences.
Risk factor A characteristic associated with the risk of one or more kinds of harm.
Risk of harm Means the possibility of individuals encountering harm on a Part 3 service. With reference to a Part 3 U2U service, it means the risk of harm to individuals presented by (a) content on that U2U service that may amount to illegal content; and (b) the use of that U2U service for the commission and/or facilitation of a priority offence. With reference to a Part 3 search service, it refers to the risk of harm to individuals presented by search content on that service that amounts to illegal content.
Search result In relation to a search service, it means content presented to a user of the service by operation of the search engine in response to a search request made by the user.2323
Search services An internet service that is, or includes, a search engine.
User-to-user services An internet service by means of which content that is generated directly on the service by a user of the service, or uploaded to or shared on the service by a user of the service, may be encountered by another user, or other users, of the service.
Functionalities and recommender systems¶
Term Definition¶
Accepting User-to-user service functionality allowing to make and/or receive cryptocurrency cryptocurrency payments by means of the service payments
Accepting online User-to-user service functionality allowing users to make and/or payments receive financial payments by means of the service
Anonymous user User-to-user service functionality allowing users to create a user profile profiles where their identity is unknown to an extent. This includes instances
2323 Section 57(3) of the OS Act. 428
where a user's identity2324 is unknown to other users, for examplebake off through the use of aliases ('pseudonymity'). It also includes where a user's identity may be unknown to a service, for example services that do not require users to register by creating an account.2325
Building lists or User-to-user service functionality allowing users to create lists, directories collections, archives or directories of content or users of the service.
Commenting on User-to-user service functionality that allows users to reply to content, content or post content in response to another piece of content, visually accessible directly from the original content without navigating away from that content.
Content editing Functionality type that comprises user-to-user functionalities that allow users to alter user-generated content before or after it is shared.
Content exploring Functionality type that comprises user-to-user functionalities that allow users to explore and search for user-generated content.
Content recommender Type of recommender system that is used to suggest and curate systems content that users are likely to find engaging, based on, for example, user preferences and/or history, but also content that is popular and trending on the service at a given moment. Recommender systems exclusively used to suggest goods and services for hire or for sale are a subtype of content recommender system, and we have defined these independently as ‘product recommender systems’.
Content storage and Functionality type that comprises user-to-user functionalities that capture allow users to record and store user-generated content.
Content tagging User-to-user service functionality allowing users to assign a keyword or term to content that is shared.
Crowdfunding User-to-user service functionality allowing users to raise money from a large number of users who each contribute a relatively small sum.
Direct messaging User-to-user service functionality allowing a user to send and receive a message to one recipient at a time and which can only be immediately viewed by that specific recipient.
Downloading content User-to-user service functionality allowing users to copy content from an online service to their device for local storage.
Editing or deleting User-to-user service functionality allowing users to modify content that posted content has already been posted the same users or remove it altogether.
2324 Identity refers to an individual’s formal or officially recognised identity. 2325 The majority of our evidence base speaks of the risks posed by user-to-user anonymity. However, we have indicated where research indicates specifically service-to-user anonymity presents a risk. 429
Editing usernames User-to-user service functionality allowing users to alter the name displayed on their user profile.
Editing visual media User-to-user service functionality that allows users to alter or manipulate images and videos by means of the service.
Encrypted messaging User-to-user service functionality that allows users to send and receive messages that are end-to-end encrypted.
Ephemeral messaging User-to-user service functionality that that allows users to send messages that are automatically deleted after they are viewed by the recipient, or after a prescribed period of time has elapsed.
Functionalities In relation to a user-to-user service, includes any feature that enables interactions of any description between users of the service by means of the service.2326 In relation to a search service, includes (in particular): (a) a feature that enables users to search websites or databases; (b) a feature that makes suggestions relating to users’ search requests (predictive search functionality).2327 In practice, when referring to functionalities in the Register of Risks, functionalities refer to front-end features of a service. For user-to-user services, functionalities refer to features that enable interaction between users. Functionalities for search services refer to features that enable users to search websites or databases, as well as features that make suggestions relating to users’ search requests. Functionality type Grouping of functionalities allowing users to engage in a similar online activity.
Group messaging User-to-user service functionality allowing users to send and receive messages through a closed channel of communication to more than one recipient at a time.
Hyperlinking User-to-user service functionality enabling users to access other internet services by clicking or tapping on content present on the service.
Live audio User-to-user service functionality that allows users to communicate with one another in real-time through speech or other sounds.2328
2326 Section 233(1) of the OS Act. Please refer to section 233(2) of the OS Act for a non-comprehensive list of user-to-user functionalities.
2327 Section 233(3) of the OS Act. 2328 While one-to-one live aural communications are not regulated-user generated content, they may be in scope of the OS Act when ‘accompanied by user generated content of any other kind, except identifying content’ or when they are recorded. Aural communications can be regulated user-generated content if they allow more than two users to communicate by means of the service. Section (55)5 of the OS Act. 430
Livestreaming User-to-user service functionality that allows users to simultaneously create and broadcast online streaming media in, or very close to, real time.
Network recommender Type of recommender system that suggests users and/or groups of systems other users to connect with. Network recommenders may consider a variety of user interactions, mutual connections, and group memberships to determine which network recommendations might be relevant and useful.
Posting content User-to-user service functionality allowing users to upload and share content on open channels of communication.
Posting goods or User-to-user service functionality allowing users to post content services for sale dedicated to offering goods and services for sale. This does not include paid-for advertisements,2329 but may serve the function of allowing users to promote goods or services
Posting or sending User-to-user service functionality allowing users to share their current location information or historic location, record a user’s movement, or identify which other users of the service are nearby.
Product recommender Under the Act’s definition of user generated content, product systems recommender systems are considered a subtype of content recommender systems. This is because product listings on U2U online marketplace and listing services are considered user-generated content. Product recommender systems are exclusively used to suggest good and services for sale or for hire. Product recommender systems suggest goods and services that a user might want to purchase. These recommendations are typically based on search and purchasing history.
Reacting to content User-to-user service functionality allowing users to express a reaction, such as approval or disapproval, of content that is shared by other users through dedicated features that can be clicked or tapped by users.2330
Recommender systems An algorithmic system which, by means of a machine learning model, determine the relative ranking of suggestions made to users on a U2U service. The overarching objective of recommender systems is to ensure users receive suggestions they are likely to find relevant and engaging, thereby improving allocative efficiency in the digital marketplace. This can include suggesting connections, groups, events, and content.
Re-posting or User-to-user service functionality that allows users to re-share content forwarding content that has already been shared by a user.
2329 See ‘advertising-based revenue model’ in business models for more information. 2330This for instance includes ‘liking’ or ‘disliking’ a post. 431
Reverse image Search service functionality enabling users to find similar images based searching on sample images used as a search query.
Screen capturing or User-to-user service functionality that allows users to capture an image recording or record a video showing the contents of their display.2331
Search prediction and Functionality type that comprises search service functionalities personalisation allowing suggestions to be made relating to users’ search requests.
Search query inputs Search service functionality type by means of which users input search queries.
Transactions and offers Functionality type that comprises user-to-user service functionalities that allow users to buy, sell, and exchange goods and services with each other. Includes non-profit transactions and offers.
User communication Functionality type that comprises user-to-user service functionalities that allow users to communicate with one another either synchronously or asynchronously. Includes communication across open and closed2332 channels.
User connections User-to-user service functionality that allows users to follow or subscribe to other users. Users must sometimes be connected to view all or some of the content that each user shares.
User events User-to-user service functionality that enables users to create an online space to share content that is dedicated to a particular event. This can include a date, description, and attendance of users.
User generated content User-to-user service functionality allowing users to narrow the search filtering parameters of the returned search results, which display user-generated content.
User generated content User-to-user service functionality allowing users to search for user searching generated content by means of a user-to-user service
User groups User-to-user service functionality allowing users to create online spaces that are often devoted to sharing content surrounding a particular topic. User groups are generally closed to the public and require an invitation or approval from existing members to gain access. However, in some cases they may be open to the public.
User identification Functionality type that comprises user-to-user service functionalities that allow users can identify themselves to other users.
2331 While users can often record or capture content using third-party services, screen recordings and captures are often shared on user-to-user services as user-generated content and some user-to-user services have dedicated screen recording and screen capturing functionalities. 2332 See content audiences for definition of open and closed channels of communication. 432
User networking Functionality type that comprises user-to-user service functionalities that allow users to find or encounter each other and establish contact.
User profiles User-to-user service functionality that is associated with a user account, that represents a collection of information shared by a user which may be viewed by other users of the service. This can include information such as username, biography, profile picture, etc., as well as user-generated content generated, shared or uploaded by the user using the relevant account. 23332334
User searching User-to-user service functionality that enables users to search for other users of a service.
User tagging User-to-user service functionality allowing users to assign other users, typically by their username, to content that is shared.
Video calling User-to-user service functionality allowing users to communicate with one another in real-time through video communications.
Business models and commercial profile¶
Term Definition¶
Advertising-based Revenue models that generate income through payments for the revenue models display of advertisements promoting a product or service. Business models Way in which a business operates to achieve its goals. For the purposes of this risk assessment, this includes a service’s revenue model and growth strategy. 2335 Commercial profile Size of the service in terms of capacity,2336 the stage of service maturity and rate of growth in relation to users or revenue Early-stage services Services in the initial phases of their lifecycle, typically encompassing the startup and early growth stages. This is characterized by its early establishment, limited operational history, and ongoing efforts to establish itself in the market Growth strategy How the service plans to expand its business. For example, through growing revenue and number of users.
2334 Users can sometimes create fake user profiles, which are not a functionality in themselves, but are user profiles that impersonates another entity or are intentionally misleading. 2334 Users can sometimes create fake user profiles, which are not a functionality in themselves, but are user profiles that impersonates another entity or are intentionally misleading.
2335 ‘Business model’ can be defined more widely to describe the way in which a service creates value to its users (value proposition), how it delivers this value to users, and how it captures value for itself. However, we adopt a narrow definition in the risk assessment to avoid overlap with the other risk characteristics. This does not affect the overall risk assessment as risk factors that would have been identified under the broader definition are captured elsewhere. 2336 In terms of number of employees and/or revenue. 433
High-capacity services Services with a large number of employees and/or revenue2337 Low-capacity services Services with a small number of employees and/or revenue2338 Revenue model How a service generates income or revenue Subscription-based Revenue models that generate income by selling access (or premium revenue models access) to a service for a period of time in return for a fee
User base¶
Term Definition¶
Child user A user under the age of 18 Protected (user) Means age; disability; gender reassignment; marriage and civil characteristics partnership; pregnancy and maternity; race; religion or belief; sex; and sexual orientation.2339 User base Users of a service. A user does not need to be registered with a service to be considered a user of that service.2340 User base Demographic make-up of the user base, including selected demographics characteristics, intersectional dynamics and other relevant demographic factors.
Governance, systems and processes¶
Term Definition¶
Account blocking Process of removing users and often also preventing them from using a service. It is usually deployed for serious or multiple infringements of service policies as it has a high level of impact on the user. A user can be temporarily blocked or have their account and access permanently suspended (often referred to as a ‘ban’). Account strikes Process of adding a mark on a user or their account to note that they have contravened the service’s policies. Content moderation When a service reviews content to decide whether it is permitted on its platform. Governance Structures that ensure the adequate oversight, accountability, and transparency of decisions within a service which impact user safety.
2337 Our evidence does not currently allow for quantitative thresholds to be drawn for service capacity. Services should nevertheless consider the number of employees and revenue as a risk factor.
2338 Our evidence does not currently allow for quantitative thresholds to be drawn for service capacity. Services should nevertheless consider the number of employees and revenue as a risk factor. 2339 Section 4 of the Equality Act 2010. 2340 Section 195 of the OS Act makes clear that ‘it does not matter whether a person is registered to use a service’ for them to be considered a ‘user.’ 434
This is in relation to organisational structure as well as product and content governance.
Service design Design of all components that shape a user’s end to end experience with a service. These components can include the business model or decision-making structures, back-end systems and processes, the user interface, and off-platform interventions. Systems and processes Characteristic concerning the actions taken by a service, including procedures to mitigate the risk of harm arising from illegal content being encountered. This can be either human or automated, or a combination of the two, and include technology. User access A user’s entry into a service and ability to use the functionalities present on that service. Verification and Schemes operated by services to grant verified status to the profiles of labelling schemes certain users, such as notable users or those who subscribe to a paid-for scheme. These schemes may involve labelling a user’s profile to indicate that it is verified. Verification in this context may take different forms but usually involves the service carrying out a process before a user profile is labelled as being part of a particular scheme.
Service type¶
Term Definition¶
Discussion forums and A user-to-user service type describing general services that generally chat room services allow users to send or post messages that can be read by the public or an open group of people. Downstream general Search service type describing a subsection of general search services. search service Downstream general search services provide access to content from across the web, but they are distinct in that they obtain or supplement their search index from other general search services. File-storage and file-User-to-user service type describing services whose primary sharing services functionalities involve enabling users to store digital content and share access to that content through links. Fundraising services User-to-user service type describing services that typically enable users to create fundraising campaigns and collect donations from users. General search services Search service type describing services that enables users to search the internet and which derives search results from an underlying search index (developed by either the service or a third party). Information-sharing User-to-user service type describing services that are primarily focused services on providing user-generated informational resources to other users. Messaging services A user-to-user service type describing services that are typically centred around the sending and receiving of messages that can only be viewed or read by a specific recipient or group of people. 435
User-to-user User-to-user service type whose principal purpose is to disseminate pornography services user-generated pornography. Dating services User-to-user service type describing services that enable users to find and communicate with romantic or sexual partners. Gaming services User-to-user service type describing services that allow users to interact within partially or fully simulated virtual environments. Marketplaces and User-to-user service type describing services that allow users to buy listings services and sell their goods or services. Payment services User-to-user service type describing websites or applications that financial payment providers often have that enable users to send and receive money.
Service type Service type is a characteristic that in general refers to the nature of the service.2341 This, for instance, includes social media services and private messaging services
Social media services User-to-user service type describing services that connect users and enable them to build communities around common interests or connections. Vertical search services Search service type describing services that enable users to search for specific topics, or products or services offered by third party providers. Unlike general search services, they do not return search results based on an underlying search index. Rather, they use an API or equivalent technical means to directly query selected websites or databases with which they have a contract, and to return search results to users Video-sharing services User-to-user service type describing services that allow users to upload and share videos with the public.
Other terms¶
Term Definition¶
Adult Services Website Marketplace type services which allow for the listing of a sexual service.
Augmented reality Involves overlaying digital content, which could include a combination of sound, video, text, and graphics, onto a real-world environment using a headset or a device with a camera, such as a mobile phone.
2341 Certain service types have been selected because our evidence suggests that they can be used to facilitate or commit relevant offences. 436
Blockchain A decentralised, distributed ledger that stores the record of ownership of digital assets.2342
Bot An umbrella term that refers to a software application or automated tool that has been programmed by a person to carry out a specific or predefined task without any human intervention. Clear web Publicly accessible websites that are indexed by search engines.
Click farming Practice of manually clicking on online adverts to increase the clickthrough rate value, boost engagement metrics, and inflate impressions. This activity can be carried out by bot accounts, as discussed here, or by large groups of workers.
Content audience Refers to whether content is shared on open or closed channels of communication. Open channels are areas of services where content is visible to the general public or any user. Closed channels are areas of a service where content is limited to a smaller audience, and where users can expect more privacy, such as direct messaging or user groups that have controls or restrictions on who can join.
Content format Refers to the format in which content is made available. This, for instance, includes content in the form of images, video, audio, text and emojis.
CSAM URL A URL at which CSAM is present, or which includes a domain which is entirely or predominantly dedicated to CSAM.
Deepfake Specific type of media that involves the use of AI algorithms, particularly generative AI models, to modify videos, images or audio to create realistic synthetic content. This is often done by superimposing the face of a person onto the body of another person in a video or image as well as voice manipulation with lip syncing. Deepfakes are commonly shared as user generated content on user-to-user services but could also potentially be created using functionalities present on user-to-user services. Deepfake technology is currently used to create content that can be harmful; however, we acknowledge that it may also have positive use cases.
Down-blousing Refers to someone taking a photo down a woman’s top without consent.2343
Generative artificial Also known as ‘GenAI,’ generative artificial intelligence is an emerging intelligence form of AI that refers to machine learning models which can create new content in response to a user prompt. These tools can be used to produce text, images, audio, video and code, which closely resemble the broad datasets on which the models are trained.
2342 Builtin (Daley, S.), 2022. What Is Blockchain? [accessed 18 June 2023]. 2343 Ministry of Justice, 2022. New laws to better protect victims from abuse of intimate images. [accessed 3 August 2023]. 437
Hashtag hijacking Use of a hashtag for a purpose other than it was created – such as tagging a message containing undesirable or harmful content with a popular, but unrelated, hashtag to surface this content to a target audience.
Indexing Process of collecting, parsing, and storing of data by a search engine to facilitate fast and accurate information retrieval.
Like farming Use of fake pages on social media services designed to artificially increase the popularity of a page, so it can be sold to buyers seeing accounts with large followings or for scam and fraud activity.
Mixed reality Refers to the blending of physical and virtual worlds to produce new environments where physical and digital objects co-exist and interact in real time.
Money mule Someone who receives money from a third party in their bank account and transfers it somewhere else, or who withdraws it as cash and gives it to someone else, obtaining a commission for it or payments in kind. These individuals are targeted by ‘money mule recruiters’, sometimes referred to as ‘mule herders’, who recruit money mules.
Financially motivated Form of blackmail that involves threatening to publish sexual sexual extortion information, photos or videos about someone. This may be to extort (‘sextortion’) money or to force the victim to do something against their will. Photos or recordings are often made without the victim realising or consenting.2344
Trend jacking Refers to when influencers, brands or organisations insert themselves into conversations online that are gaining a lot of attention – for example, by using associated hashtags or trending audios.
Up-skirting Refers to someone taking a photograph that appears to have been taken up a person’s clothing (such as a skirt) without consent.
Virtual reality Involves the use of a head mounted display to access a virtual experience, which could be digitally created or a captured 360° photo or video.
2344 Metropolitan Police, n.d. Sextortion. [accessed 4 August 2023]. 438
A2 Updating the Register of Risks¶
This Annex was originally published with the Register or Risks as part of the Illegal Harms Statement of [December 2024]
Introduction¶
2345 Section 98(5) of the Act. 439
characteristics.2346 The Risk Profiles did not set out all the risk factors from the Register of Risks, but only those we considered to be particularly important for service providers to consider.
Methodology¶
Evidence base¶
2346 Characteristics include a service’s user base, business model, functionalities and any other matters we deem relevant to risk. Risk Profiles focus predominately on user base demographics, functionalities and business models. Step 2 of the risk assessment guidance provides information for services on user base size, governance, and systems and processes. 2347 ‘Method’ examined the strengths and weaknesses of the methodology for that particular topic, such as whether appropriate data collection methods were used. ‘Robustness’ considered both the size and coverage of the sample, and quality of analysis – for example, how missing data values were accounted for. ‘Ethics’ refers to how well ethical considerations were addressed in the study, such as how personal data was handled. ‘Independence’ examined the origins of the research and whether any stakeholder interests might have influenced findings. ‘Narrative’ refers to the commentary within the report and whether conclusions are sufficiently backed by the research, and whether there is a clear distinction between the findings and the interpretation. 440
judgement and expertise about specific harms to draw conclusions where we think this can help service providers to identify potential risks. We have also referred to evidence related to content or activity that is broader than the offences discussed, where we consider that this is still likely to be relevant to those offences, including the risk of a service being used to commit or facilitate a priority offence.
Characteristics and risk¶
• There is some evidence to suggest that certain service types with common features and functionalities, are more likely to be used to commit and facilitate some offences.2348 We have therefore identified some service types as a driver of risk. • We have also identified recommender systems as a relevant characteristic because of the key role they play in determining what content users see and engage with, therefore contributing significantly to a user’s experience of services that use them. Recommender systems can be used in many ways which can influence how a user might experience risk of harm on a service. Most commonly this includes content recommender systems designed for the curation of content feeds, and network recommender systems that are used to recommend other users to follow/befriend. • We have also included commercial profiles as our evidence showed that services with certain commercial profiles are likely to have weaker risk management, which can make them targets for perpetrators.
Table 1. Definitions for each characteristic¶
2348 For example, social media services are relevant to most types of illegal harm due to the wide range of functionalities they provide for sharing content and connecting users. In contrast, marketplaces and listing services are associated with fewer kinds of illegal harm, primarily those where the supply or sale of illegal goods or services is particularly important. 2349 The Act makes clear that ‘it does not matter whether a person is registered to use a service’ for them to be considered a ‘user’ (section 227 of the Online Safety Act). The Act is only concerned with the number of ‘United Kingdom users’ of the 442
service, so where the user is an individual, they count as a user only where they are in the United Kingdom; similarly, where the user is an entity, they count only where they have been formed or incorporated in the United Kingdom (section 227(1) of the Online Safety Act 2023). 2350 Section 233: For U2U: (a) creating a user profile, including an anonymous or pseudonymous profile; (b) searching within the service for user-generated content or other users of the service; (c) forwarding content to, or sharing content with, other users of the service; (d) sharing content on other internet services; (e) sending direct messages to or speaking to other users of the service, or interacting with them in another way (for example by playing a game); (f) expressing a view on content, including, for example, by— (i) applying a ‘like’ or ‘dislike’ button or other button of that nature, (ii) applying an emoji or symbol of any kind, (iii) engaging in yes/no voting, or (iv) rating or scoring content in any way (including giving star or numerical ratings); (g) sharing current or historic location information with other users of the service, recording a user’s movements, or identifying which other users of the service are nearby; (h) following or subscribing to particular kinds of content or particular users of the service; (i) creating lists, collections, archives or directories of content or users of the service; (j) tagging or labelling content present on the service; (k) uploading content relating to goods or services; (l) applying or changing settings on the service which affect the presentation of user-generated content on the service; (m) accessing other internet services through content present on the service (for example through hyperlinks). For Search: (a) a feature that enables users to search websites or databases; (b) a feature that makes suggestions relating to users’ search requests (predictive search functionality).
2351 ‘Business model’ can be defined more widely to describe the way in which a service creates value to its users (value proposition), how it delivers this value to users, and how it captures value for itself. However, we adopt a narrow definition in the risk assessment to avoid overlap with the other risk characteristics. This does not affect the overall risk assessment as risk factors that would have been identified under the broader definition are captured elsewhere. 443
Kinds of illegal harm¶
2352 ‘Maturity’ refers to the stage the service or company is at in the typical business lifecycle. The stages can be split into four: i) introductory or start-up stage, ii) growth stage, iii) maturity stage, and iv) decline. The maturity stage is characterised by high revenues, cashflow and profitability. 2353 As explained in Overview of Illegal Harms, ‘inchoate offences’ include assisting someone else to commit a priority offence, encouraging someone else to commit a priority offence, attempting to commit a priority offence or conspiring to commit a priority offence.
2354 Referred to in the Act as ‘other offences’, they are all offences under UK law that are not priority offences, where (a) the victim or intended victim of the offence is an individual (or individuals); (b) the offence is created as a result of the Act, another Act, an order of Council or other relevant instruments; (c) the offence does not concern the infringement of intellectual property rights, the safety or quality of goods, or the performance of a service by a person not qualified to perform it; and (d) the offence is not an offence under the Consumer Protection from Unfair Trading Regulations 2008. 444
Risks and Risk Profiles on this basis, and thus expect service providers to take account of this when conducting their risk assessments.2355 The scope of certain recommended meausures in our Codes of Practice will depend on the level of risk of particular kinds of illegal harms on a service. For further details on the relationship between the kinds of illegal harms in the Register of Risks and the Codes of Practice please see Volume 2 – Service design and user choice.
Stakeholder responses and decisions by theme¶
Support for our methodology and overall approach¶
2355 However, within each grouping we sometimes refer to individual offences where appropriate, for example where the particular observation or evidence is relevant only to specific offences. 445
Although, as noted, stakeholders provided extensive feedback about specific aspects of the Register of Risks and the conclusions drawn from our analysis (discussed below).
Our decision¶
Grouping of illegal harms¶
Our decisions¶
2356 GAATW response to November 2023 Illegal Harms Consultation, p. 1 2357 [] 2358 Scottish Government response to November 2023 Illegal Harms Consultation, p. 2 2359 Meta and WhatsApp response to November 2023 Illegal Harms Consultation, p. 14 2360 Assisting serious self-harm was expected to be a priority harm until late on in the drafting of our November 2023 Consultation, so the two were originally addressed together 447
c) Hate d) Harassment, stalking, threats and abuse e) Controlling or coercive behaviour f) Intimate image abuse g) Extreme pornography h) Sexual exploitation of adults i) Human trafficking j) Unlawful immigration k) Fraud and financial offences l) Proceeds of crime m) Drugs and psychoactive substances n) Firearms, knives and other weapons o) Encouraging or assisting suicide p) Foreign interference
2361 In our November 2023 Consultation we proposed to define a service as multi-risk where it is high or medium risk for at least two kinds of illegal harms. This is important because some of the measures we are proposing target a wide range of online harms and we propose to apply the most onerous of these measures in our Codes only to services which are large and/or multi-risk. 2362 For example, advertising a fake job could likely amount to both fraud and human trafficking offences, but we would still expect service providers to assess the risk of both kinds of illegal harm separately. 448
q) Animal cruelty r) Non-priority offence - Epilepsy trolling s) Non-priority offence - Cyberflashing t) Non-priority offence - Encouraging or assisting self-harm u) Non-priority offence - False communications v) Non-priority offence - Obscene content showing torture of humans and animals (the s.127(1) offence) w) Non-priority offence - Threatening communications x) Search services y) Governance, systems, and processes
New risk factors for certain illegal harms¶
Our decision¶
2363 Jonathan Hall, Independent Reviewer of Terrorism Legislation response to November 2023 Illegal Harms Consultation, pp. 1-6 2364 Children’s Commissioner response to November 2023 Illegal Harms Consultation, p. 20 2365 Victims’ Commissioner for England and Wales response to November 2023 Illegal Harms Consultation, pp.1, 4 2366 Institute for Strategic Dialogue response to November 2023 Illegal Harms Consultation, pp. 3-5. 449
• We have added risk factors to most chapters, drawing in particular on new research and evidence that links particular functionalities and service types with kinds of illegal harm, ultimately providing a much more comprehensive evidence base than we had when publishing the draft Register of Risks in November 2023. For example, this includes adding network recommender systems as a risk factor in relation to harassment, stalking, threats and abuse; social media and messaging services as important service types in the commission of human trafficking offences; and gaming services in relation to proceeds of crime offences. • We have added significantly to our commentary surrounding user base charactersitics across the Register of Risks, providing more information about who may experience a greater risk of harm. For example, including additional evidence in relation to a service’s users who may be victims of intimate image abuse, foreign interference, or terrorism offences – where the age of a service’s users has been added to ensure this is something that can be accounted for when assessing risks – among others. On the same theme, where appropriate, we have provided further information regarding user base characteristics that relate to perpetrators of offences online.
A3 Updating the Risk Profiles¶
This Annex was originally published with the Register or Risks as part of the Illegal Harms Statement of [December 2024]
Our proposals¶
November 2023 Illegal Harms Consultation¶
2367 Section 98(5) of the Act. The Register of Risks is Ofcom’s own risk assessment of the impact of characteristics of services on the risks of harm to individuals from illegal content. For U2U services, this includes the risk of harm from the facilitation and commission of illegal harms, as well as users encountering illegal content. For Search, this includes only the risk of harm from users encountering illegal content. Details on our approach and our full findings from our risk assessment are available in the Register of Risks document.
2368 Characteristics include a service’s user base, business model, functionalities and any other matters we deem relevant to risk. Risk Profiles focus predominately on user base demographics, functionalities and business models. Step 2 of the risk assessment guidance provides information for services on user base size, governance, and systems and processes. 2369 We recognised that Risk Profiles cannot fully capture the complexity and context of risk factors across all the harms considered. 451
opted to produce two Risk Profiles presented as two tables of risk factors; one for U2U services to consult (‘U2U Risk Profile’) and one for Search services to consult (‘Search Risk Profile’). We proposed that services should consult the relevant table and decide which risk factors are relevant to them when doing their risk assessment. • Some risk factors in the tables were those that only relevant service providers had to take account of in their risk assessment because they represent characteristics that only certain services have (for example, being able to ‘comment’). We referred to these as specific risk factors, and service providers were expected to identify which of these apply to them. To help services do this accurately, we provided a list of Yes (Y) or No (N) questions, where each ‘Y’ answer corresponds to an additional risk factor in the tables. • Some of the risk factors in the tables were things that all services must take account of such as user base demographics, business model and commercial profile. We referred to these as general risk factors. Given that there were only three general risk factors, we included high level information about all three in both the U2U and Search tables. We also provided information about different kinds of illegal harms where possible.
2370 We determined that a qualitative methodology was better able to provide an accurate assessment of the evidence available given the complexity of the evidence and the lack of consistent or comparable numerical data across illegal harms. The methodology considered the strength of the evidence for different risk factors, common trends across illegal harms, and alignment with other aspects of our regulatory approach. For example, when considering “hyperlinks” as a risk factor, we considered how the evidence in the Register explained the relationship between hyperlinks and each kind of illegal harm individually, as well as considering the relationship between hyperlinks and illegal content more broadly. We also considered the relationship between hyperlinks and our wider regulatory approach, for example the Codes of Practice. 452
broadly, we explained service providers should use this information to help them assess their risk level for each kind of priority illegal offence in Step 2 of the risk assessment process.
Alternative approaches to the Risk Profiles¶
August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty¶
2371 See Table 9.3 of the November 2023 Illegal Harms Consultation for further details on alternative options that were considered. 2372 We considered each option against two main objectives: a) our approach should effectively present our evidence on what makes services risky; and b) our approach should be easy for all services to use. 453
facilitate the illegal harm in manifesting. We did not propose any changes to the Search Risk Profile with regards to animal cruelty.
Changes to U2U Risk Profile based on new evidence¶
Table 2. Changes to key kinds of illegal harms associated with specific risk factors in the U2U Risk Profiles
454
2373 Please note, this is due to encouraging or assisting serious self-harm being a non-priority illegal harm. See paragraph A2.32 to A2.37 for further details. 2374 This is due to human trafficking and unlawful immigration now being treated as separate kinds of illegal harm. See A2.32 to A2.37 for further details. 2375 The ‘firearms and other weapons’ illegal harm has been changed to ‘firearms, knives and other weapons’ in the Register of Risks so this has been reflected in the Risk Profiles. 455
456
Stakeholder responses and decisions by theme¶
• Whether the Risk Profiles should have more context specific information • Suggested changes to the risk factors and key kinds of illegal harm in the Risk Profiles • Responses regarding the Search Risk Profile • Review of and updates to the Risk Profiles • Other amendments to the Risk Profiles
Views on the proposed approach and format of the Risk Profiles¶
2376 Microsoft response to November 2023 Illegal Harms Consultation, p.6. 2377 Meta response to November 2023 Illegal Harms Consultation, p.12. 2378 LinkedIn response to November 2023 Illegal Harms Consultation, p.6. 2379 Match Group response to November 2023 Illegal Harms Consultation, pp.5-6. 2380 Betting and Gaming Council response to November 2023 Illegal Harms Consultation, p.4; Evri response to November 2023 Illegal Harms Consultation, p.3; Mencap response to November 2023 Illegal Harms Consultation, p.5; Stop Scams UK response November 2023 Illegal Harms Consultation, p.6. 2381 Blue Cross response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.5; Born Free Foundation response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.2; Dogs Trust response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.2; International Cat Care response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.2; RSPCA response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.5; Scottish SPCA response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.5; The Links Group response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.4; Wildlife and Countryside Link response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.1. 458
Profiles clearer for service providers and allow nuance between risk factors associated with different service types by, for example, identifying how a particular risk factor can cause heightened risk of illegal harms occurring on one service type but not another.2382
2382 Airbnb response to November 2023 Illegal Harms Consultation, p.6; Booking.com response to November 2023 Illegal Harms Consultation, pp.3, 6. 2383 Centre for Competition Policy response to November 2023 Illegal Harms Consultation, pp.11-13. 2384 Scottish Government response to November 2023 Illegal Harms Consultation, p.4. 2385 Christian Action, Research and Education (CARE) response to November 2023 Illegal Harms Consultation, pp.7-8.
2386 The British and Irish Law, Education and Technology Association response to November 2023 Illegal Harms Consultation, pp.3-4. 2387 WeProtect Global Alliance response to November 2023 Illegal Harms Consultation, p.7. 2388 []; Booking.com response to November 2023 Consultation, p.3; Reddit response to November 2023 Illegal Harms Consultation, p.20; Roblox response to November 2023 Illegal Harms Consultation, p.9; Safe Space One response to November 2023 Illegal Harms Consultation, p.6; techUK response to November 2023 Illegal Harms Consultation, p.14. 459
Our decision¶
2389 See Table 9.3 of the November 2023 Illegal Harms Consultation for further details on alternative options that were considered. 2390 We considered each option against two main objectives: a) our approach should effectively present our evidence on what makes services risky; and b) our approach should be easy for all services to use. 460
2391 We retained aspects of this structure in our proposed approach. When presenting functionality-based risk factors, we organised them based on the groupings described in this option, which match those used in the Register (for example, grouping user identification factors together within the U2U Risk Profile). 461
Whether the Risk Profiles should have more context specific information¶
2392 Airbnb response to November 2023 Consultation, p.5. 2393 Reddit response to November 2023 Consultation, pp.24-25. 2394 Roblox response to November 2023 Consultation, pp.9-11; []. 2395 Roblox response to November 2023 Consultation, p.10; []. 2396 Mobile Games Intelligence Forum response to November 2023 Illegal Harms Consultation, pp.1-3. 2397 techUK response to November 2023 Consultation, p.14. 462
storage services pose an elevated risk of the illegal harms associated with each risk factor.2398
2398 []. 2399 Trust Alliance Group response to November 2023 Illegal Harms Consultation, p.5. 2400 UK Interactive Entertainment (Ukie) response to November 2023 Illegal Harms Consultation, p.3; UKie response to May 2024 Consultation on Protecting Children from Harms Online, p.12.
2401 Google response to November 2023 Illegal Harms Consultation, pp.24-25. 2402 Google response to November 2023 Consultation, pp.23-25. 2403 Which? response to November 2023 Illegal Harms Consultation, p.3. 2404 UK Safer Internet Centre response to November 2023 Illegal Harms Consultation, pp.4, 23. 2405 Cybersafe Scotland response to November 2023 Illegal Harms Consultation, p.1. 463
Our decision¶
do not explicitly refer to minorities in the Risk Profiles, we do call out several protected characteristics such as race (including ethnicity) and religion under the general risk factors of user base demographics in the U2U Risk Profile, which would likely also cover ethnic or religious minorities. However, for the avoidance of doubt and to add further clarity to our Risk Profiles, we have decided to include direct mention of minorities.
Suggestions for new risk factors for some illegal harm¶
2406 FCA response to November 2023 Illegal Harms Consultation, p.4. These risk factors included: 1e. Services with discussion forums and chat rooms, 3b. Services where users can post or send content anonymously, including without an account, 4b. Services where users can form user groups or send group messages, 5a. Services with livestreaming, 5g. Services with re-posting or forwarding of content, 7b. Services with hyperlinks, and 8. Services with recommender systems. 2407 UK Finance response to November 2023 Illegal Harms Consultation, p.4. These risk factors included: 1e. Services with discussion forums and chat rooms, 3a. Services with user profiles, 3b. Services where users can post or send content anonymously, including without an account, 5a. Services with livestreaming, 5d. Services with commenting on content, and 7b. Services with hyperlinks.
2408 Samaritans response to November 2023 Illegal Harms Consultation, p.4. 2409 Trust Alliance Group response to November 2023 Consultation, p.2. 2410 Trust Alliance Group response to November 2023 Consultation, p.7. 465
2411 Battersea Dogs & Cats Home response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, pp.5-8; Born Free Foundation response to August 2024 Further Consultation, p.2; Cats Protection response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, pp.6-8; Social Media Animal Cruelty Coalition (SMACC) response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, pp.2-3. These risk factors included varying combinations of: 1d. Adult services, 1e. Discussion forums and chat rooms, 1f. Marketplace and listing services, 1g. File-storage and file-sharing services, 3b. Services where users can post or send content anonymously, including without and account, 4a. Services with user connections, 5a. Services with livestreaming, 5b. Services with direct messaging, 5c. Services with encrypted messaging, 5g. Services with re-posting or forwarding content, 6. Services where users can post goods or services for sale, 7a. Services where users can search for user-generated content, 7b. Services with hyperlinks, and 8. Services with recommender systems. 2412 International Cat Care response to August 2024 Further Consultation, p.4; RSPCA response to August 2024 Further Consultation, p.7; Scottish SPCA response to August 2024 Further Consultation, p.6. 2413 International Cat Care response to August 2024 Further Consultation, p.4; Scottish SPCA response to August 2024 Further Consultation, p.6.
2414 Battersea Dogs & Cats Home response to August 2024 Further Consultation, p.5. 2415 Cats Protection response to August 2024 Further Consultation, p.5. 2416 Born Free Foundation response to August 2024 Further Consultation, p.2. 2417 South West Grid for Learning (SWGfL) response to August 2024 Illegal Harms Further Consultation on Torture and Animal Cruelty, p.3. 466
Our decision¶
• 5a. Livestreaming • 5d. Commenting on content • 5e. Posting images or videos
illegal harms. We have not referenced animal cruelty specifically as it has not been identified as a key kind of illegal harm for either of them.
Search Service Risk Profile¶
Our decision¶
2418 DuckDuckGo response to November 2023 Illegal Harms Consultation, pp.2-3; []. 2419 Mid Size Platform Group response to November 2023 Illegal Harms Consultation, p.4; Skyscanner response to November 2023 Illegal Harms Consultation, p.9; Skyscanner response to May 2024 Consultation on Protecting Children from Harms Online, pp.9-10.
2420 Skyscanner response to November 2023 Consultation, p.10; Skyscanner response to May 2024 Consultation, p.6. 2421 See the Search Risk Profile in the Risk Assessment Guidance and Risk Profiles regulatory document. For example, we have added a clarificatory footnote at Figure 2 Question 1 to explain how downstream general search services should use the Search Risk Profile. 469
Review of and updates to the Risk Profiles¶
2422 Trustpilot response to November 2023 Illegal Harms Consultation, pp.4-5. 2423 techUK response to May 2024 Consultation on Protecting Children from Harms Online, p.9. We consider this response is relevant to our approach to reviewing and updating the Illegal Harms Risk Profiles.
2424 Snap response to November 2023 Illegal Harms Consultation, p.7. 2425 Lloyds Banking Group response to November 2023 Illegal Harms Consultation, pp.4-7. 2426 UK Finance response to November 2023 Consultation, p.6. 2427 INVIVIA response to November 2023 Illegal Harms Consultation, p.7. 2428 5Rights Foundation response to November 2023 Illegal Harms Consultation, p.14. 2429 Yoti response to November 2023 Illegal Harms Consultation, p.7. 470
Our decision¶
Other amendments to the Risk Profiles¶
2430 Google response to November 2023 Consultation, pp.23. Figure 1, Question 2: Does your service allow child users to access some or all of your service? 2431 Trust Alliance Group response to November 2023 Consultation, pp.6-7. 2432 Canadian Centre for Child Protection response to November 2023 Illegal Harms Consultation, p.10. 471
many U2U services, for terrorist content enhances the risk of harm and the severity of potential impacts associated with terrorism content online.2433
Our decision¶
A3.102 Regarding Google’s point on seeking clarification about whether question 2 of the U2U Risk Profile question list should align with the Children’s Access Assessment, we acknowledge that additional clarity would help here as the two concepts are distinct. Question 2 is to help services include the risk of illegal harm to a child as part of their illegal content risk assessment. Children’s Access Assessments are a new assessment that all regulated U2U and Search services must carry out to establish whether their service – or a part of it – is likely to be accessed by children. Services likely to be accessed by children will have additional duties to protect children online and they will also need to undertake a Children’s Risk Assessment and implement safety measures to protect children online. As a result of this feedback, we have added a footnote to this question to clarify this point. A3.103 In relation to this same question, we acknowledge Trust Alliance Groups comments and note that our own research produced insights showing that children under 13 years old readily used services, even where the terms of service did not allow them to do so.2435 This shows that we would need to clearly set out that service providers must look at the reality of whether child users are accessing their service and not whether their terms of service allow child users. We have therefore amended the wording of question 2 in both Figure 1 and 2 of the Risk Assessment Guidance to reflect that we are referring to whether children are actually using the service – which is the determinant of risk.2436 We have also amended the section 2 title of both the U2U and Search Risk Profile for clarity.2437 A3.104 We acknowledge Canadian Centre for Child Protection’s point in relation to the risk to children from adult services. We have therefore made an addition the risk description box of 1d. Adult services in the U2U Risk Profile to highlight this point.2438 A3.105 In relation to Dr Sandy Schumann’s point, we note that we already have a similar risk factor in the U2U Risk Profile called ‘user-generated content searching’ that we believe covers the ‘search’ functionality that they have suggested. We have added their evidence to the Register of Risks and now include terrorism as a key kind of illegal harm associated with 7a. User-generated content searching.
2433 Dr Sandy Schumann response to November 2023 Illegal Harms Consultation, p.3. 2434 Trustpilot response to November 2023 Consultation, p.8. Section 8 of the U2U Risk Profile table was titled ‘Services with recommender systems’. 2435 ‘A window into young children’s worlds’, Ofcom, 2024. https://www.ofcom.org.uk/media-use-and-attitudes/media-habits-children/a-window-into-young-childrens-online-worlds. [accessed 1 September 2024].
2436 Figure 1, Question 2: Do child users access some or all of my service? Figure 2, Question 2: Do child users access of my service? 2437 U2U and Search Risk Profiles, section 2: Services which are accessed by child users. 2438 “Furthermore, children could be harmed from exposure to this material if they are not effectively restricted from accessing these services.” 472
A3.106 We have amended the title of section 8 of the U2U Risk Profile table to ‘Services with content and/or network recommender systems’ to align with the wording in our respective question and address Trustpilot’s concerns.2439
Conclusion¶
A3.107 Having reviewed all consultation responses relating to our Register of Risks and Risk Profiles, we have decided to broadly proceed with our proposed approach for both the Register of Risks and Risk Profiles. A3.108 Our Register of Risks has now been expanded and bolstered by hundreds of new pieces of evidence submitted to us in response to our consultations. We are grateful for the depth and breadth of relevant and high-quality research that has been brought to our attention. We have reflected the vast majority of these suggestions in our final Register of Risks. A3.109 This has meant that we now have new evidence linking kinds of illegal harm to some risk factors that we did not have before. This has in turn led to the most notable change to the Risk Profiles where new kinds of illegal harm have been added to most of the U2U specific risk factors. A3.110 As set out in the Risk Profiles section above, there were some legitimate concerns and requests regarding various elements of the Risk Profiles. Where appropriate, we clarified our approach or addressed these concerns by making changes to the Risk Profiles. For example, we clarified that the Risk Profiles are just one of several inputs that service providers need to consider when assessing risk. They provide a guide as to what characteristics can be risky based on the evidence gathered in the Register of Risks to help service providers conduct their risk assessments. However, as the Risk Assessment Guidance makes clear, services can and should consider a range of other factors including wider contextual factors alongside the Risk Profiles when doing their risk assessments. A3.111 We have also added further information regarding our approach to downstream general search services and how they should undergo the risk assessment process in the Search Risk Profile. This includes additional commentary to clarify how the Search Risk Profile applies to downstream general search services and signposting to appropriate sections of this Statement, such as our chapter on ‘Our approach to developing Codes measures’, to ensure the necessary information can be located. A3.112 Overall, we consider that our approach to the Risk Profiles works well to highlight key relevant findings from the Register of Risks. It is a crucial starting point for service providers to conduct their four-step risk assessments and will continue to be a valuable resource for service providers.
2439 Question 8: Does my service use content or network recommender systems? 473