New findings from Qoria reveal 50% of UK schools surveyed report difficulty detecting harmful behaviour as the biggest barrier in protecting children from AI-generated Child Sexual Abuse Material
Key findings from UK schools surveyed*:
- 21% of schools reported incidents of students as young as 8 years old sharing with or requesting nude content from their peers, with the highest prevalence in 11-13-year-olds (58%).
- 37% of schools experience an incident of students sharing or requesting nude content at least every four months. 11% reported experiencing weekly incidents.
- 50% of schools said detecting harmful behaviour is the most significant challenge when addressing the issues of AI Child Sexual Abuse Material (CSAM) and the sharing of explicit content among students.
- 21% of schools said they weren’t aware of the most common ways AI can be exploited to target children online.
- 91% of schools were concerned about the potential for adult perpetrators to use AI to groom their students.
- 10% of schools had seen at least one incident of technology used to generate sexually graphic images by students in the last 12 months. However, 26% weren’t aware if students were using technology to create a fake sexually graphic image of a classmate.
- 78% of schools said a lack of awareness among parents was affecting their ability to address issues around CSAM and explicit content
London, UK – 25.11.2024: A new report by Qoria, a technology organisation which helps protect the digital safety and wellbeing of 24 million children globally with its digital safeguarding software, reveals new challenges faced by UK schools in protecting children against AI-generated Child Sexual Abuse Material (CSAM). With the findings backed by the Internet Watch Foundation (IWF), the report highlights the increasing difficulty experienced by schools in detecting harmful behaviours involving AI and explicit content and raises concerns about AI being used by perpetrators to groom young people online. It also uncovers the troubling prevalence of explicit content shared among students on platforms like Snapchat and through text messages.
Commenting on the report Tim Levy, Managing Director of Qoria, said “The findings of our report are significant. AI has very swiftly entered our children’s lives and well before guardrails could be erected and education provided. In educational settings, AI in all forms requires caution and diligence. Of particular importance for schools should be a renewed focus on filtering and device monitoring to tightly restrict and scan for irresponsible platforms and use.
“Our report clearly shows there are increasing risks around CSAM and schools are being impacted. As always, our remarkable educators are stepping up and doing what they can with an emergent challenge. We also acknowledge that governments and regulators are becoming increasingly aware of the risks of AI. But their work and investments must accelerate to keep up with the pace of change.
“We offer our report to the community to stimulate thought, discussion and necessary action.”
Catherine Brown, Chair of the Internet Watch Foundation (IWF) added, “This report puts a spotlight on the impact the abuse of AI technology can have. It is vital reading for anyone who works with children and young people, and it is an important asset in helping understand the evolving threats they face online.”
AI-generated CSAM is an emerging concern for schools
Artificial intelligence is amplifying risks associated with Child Sexual Abuse Material (CSAM) in UK schools with 91% of schools surveyed concerned about the potential for adult perpetrators to leverage AI for grooming children.For example, by using fake profile generators to simulate being a child or using nudification tools, AI-powered apps that can digitally remove clothing from images.
The urgency of the problem is echoed by recent data from the IWF, which identified 2023 as a record-breaking year for AI-generated child abuse material. According to the IWF, 20,254 AI-generated images were discovered within a single month on a dark web forum dedicated to CSAM in 2024.
Reflecting on the impact of AI-generated CSAM, Brown, said: “In the last few months, we have really seen AI image generators become much more sophisticated, to the point where almost anyone can make or manipulate life-like sexual or nude imagery of children at the click of a button.
“So-called ‘nudifying’ apps have increased the ease with which this imagery can be made, and we have seen what happens when this technology is abused. The resultant imagery can be used to bully and distress children and young people. It can be used by predators to groom, entrap, and ensnare them into cycles of online sexual abuse.
AI facilitating the rise of peer-to-peer CSAM
37% of schools surveyed experience incidents of students sharing or requesting nude content at least once every four months, and 11% reported an incident every week. AI is expected to worsen this already prevalent issue.
The highest rates of peer-to-peer CSAM occurred in 11–13-year-olds (58%), however, 21% report children as young as 8 years old being involved in these incidents. Notably, Snapchat is the primary platform used for sharing this content (49%), followed by Text/SMS (16%).
One UK school commented, “I have seen first hand the impact on our female students of being sent deepfake nudes derived from their social media profiles, by male students they counted as friends (hiding behind the anonymity of AI tools).”
While a relatively small proportion of schools surveyed (10%) reported cases where students used AI technology to create explicit images, 26% shared they weren’t aware if students were using this technology to create a fake sexually graphic image of a classmate.
Commenting on the findings, Yasmin London, Global Online Safety Expert and Brand Director of Qoria, said, “We are witnessing a worrying trend where very young children are engaging in the sharing of explicit content. This is not just a challenge for the school community but for society as a whole. AI is providing new tools that make creating and sharing explicit content easier, so it’s essential that our approach to digital education and student safety evolves to meet these emerging risks.”
She continued, “We must equip schools with the tools and resources they need to support students to navigate the digital world safely and empower parents to have open and honest discussions with their children about these dangers.”
Lack of resources is hindering schools’ ability to address CSAM
The report indicates that limited staff knowledge, training and time are major barriers to schools addressing CSAM.
50% of schools surveyed also identified difficulty in detecting harmful behaviour as their primary challenge in addressing issues of AI-enabled CSAM and explicit content shared among students. Adding to the issue is the prevalence of significant knowledge gaps about how AI technology can be misused, with 21% of schools surveyed unfamiliar with common methods AI can be used for to target children.
Sue Bailey, Safeguarding lead at The Arthur Terry Learning Partnership (ATLP), explains more about schools’ experience with AI and the challenges they face, “CSAM is a rapidly growing issue for schools, most notably because of the sophistication of the technology used and how difficult it is to detect harmful behaviour using AI.
“Our schools are on the AI front line and at this current time are lacking the resources and training to tackle this challenge on top of their very heavy workload.”
‘Not My Child Syndrome’ is a key barrier to addressing CSAM
A lack of awareness among parents is also affecting the ability of schools to address issues around CSAM and explicit content according to 78% of UK schools surveyed.
One reason parents may hesitate to engage with this issue is a common assumption known as ‘not my child’ syndrome, a mindset where parents assume their child would never engage in concerning online behaviour and, therefore, see no need for education or intervention on the topic. However, this assumption can create blind spots, leaving parents unprepared to recognise risks. The report emphasises that educating parents on online dangers is essential to help them notice warning signs early.
39% of the schools surveyed say they engage with and educate parents on CSAM and explicit content at least once a year, however, Bailey notes, “many of the schools I work with are trying to engage parents in online safety discussions despite limited staff training and knowledge on the issues of AI, CSAM and the sharing of explicit content.”
Community Solution Roadmap:
In addition to government action, the report highlights the need for a community-focused approach to address the risks of AI-enabled CSAM in schools.
Strengthening staff knowledge through resources and training, and equipping educators with the tools to detect and manage digital risks is essential. Schools should also actively engage parents, providing them with resources and clear guidance on digital dangers to help create a shared responsibility for online safety. Educating students about AI-specific risks is equally vital in equipping children with the digital literacy skills to recognise inappropriate content, technology or interactions, enhancing their online protection.
London concludes, “AI poses significant technical challenges for schools, but understanding the behaviours that increase children’s vulnerability is also key.”
“With a well-rounded strategy that addresses both the root behaviours and the technological risk associated with CSAM, schools will build a safer environment for their students.”
Qoria recommends schools establish dedicated AI safety working groups to lead efforts in updating policies and implementing robust safeguards that reflect the evolving risk of AI. Additionally, enhanced visibility through online monitoring technology and parental controls can help schools respond to harmful behaviours in real-time.
Recognising that student digital safety requires collective vigilance, Qoria has made its report freely accessible and will be running an open-access webinar for schools on the 27th of November, offering practical guidance on addressing digital safeguarding challenges relating to AI. By openly sharing these findings, Qoria aims to share the collective wisdom of UK schools and others around the world, uniting them in creating a safe space for children online.
Qoria’s report is available via the website here and includes practical guidance to support schools in addressing digital safeguarding issues.
