UN urges countries to protect children from escalating AI dangers
The UN is pressing governments to move faster on online safety as children face new AI-driven risks. Experts warn that "predators can use AI to analyze a child's online behavior, emotional state, and interests to tailor their grooming strategy," and that offenders are using synthetic media to generate explicit fake images of real children.
The scale is spiking. A 2025 study from Childlight Global Child Safety Institute reported a 1,325% rise (2023-2024) in harmful AI-generated abuse material. Some countries are reacting-Australia banned social media accounts for under-16s in December 2025, while the UK and EU are weighing similar thresholds-yet critics call age bans an "ineffective quick fix" without deeper, enforceable safeguards.
What the UN is calling for
In a November 2025 joint statement on Artificial Intelligence and the Rights of the Child, UN bodies flagged a "collective inability" to keep pace. Gaps include limited AI know-how among children, teachers, parents, and caregivers, and thin technical training on "AI frameworks, data protection methods and child rights impact assessments." Too many tools still ship without children's well-being in mind.
- Enforcement: "Explicitly criminalize, investigate, appropriately sanction and bring to justice" perpetrators of online child sexual abuse or exploitation conducted with AI systems.
- Safety by design: Bake child-safety into the provision, regulation, design, management, and use of platforms-by default, not as an afterthought.
- Education and capacity: Equip schools, caregivers, and frontline services with AI literacy and practical protocols.
- Governance: Require child-rights impact assessments, strong data protection, and clear accountability for platforms and developers.
These align with the UN Committee on the Rights of the Child's 2021 guidance, which says children's right to life, survival, and development must be protected from violent and sexual content, cyberaggression, harassment, gambling, exploitation and abuse, and content that promotes suicide or life-threatening activities. See the official guidance from OHCHR for detail: General comment No. 25 (2021).
Why this matters across government, IT, and development
AI lowers the time, cost, and skill needed to generate convincing grooming scripts, fake personas, and hyper-realistic images. Offenders can automate outreach at scale, test which messages work, and personalize lures.
Detection is harder because harmful content can be generated on demand, modified to evade filters, or laundered across platforms. Without clear rules and shared signals, abuse moves faster than response.
What governments can act on now
- Criminal law updates: Explicitly cover AI-assisted grooming, sextortion with synthetic media, and distribution of AI-generated child sexual abuse material-plus extraterritorial reach where feasible.
- Age assurance with guardrails: If setting 16+ thresholds for social media, pair with privacy-preserving age assurance, appeal processes, and independent audits. Avoid over-collection of minors' data.
- Mandatory risk and impact assessments: Require child-rights impact assessments for high-risk AI features (image/video generation, chat, recommendation systems), and publish summaries.
- Transparency and data access: Annual child-safety reports, incident disclosures, and secure researcher access to enable oversight.
- Content provenance and detection: Incentivize adoption of verifiable content credentials and cross-platform signals for grooming and sextortion indicators, with privacy safeguards.
- Procurement levers: For public-sector tools, mandate safety-by-design, data minimization, and crisis escalation playbooks.
- Specialized units and hotlines: Fund digital forensics, survivor support, and 24/7 reporting channels. Enable fast cross-border preservation and takedown orders.
- Training at scale: Build capacity for educators, social workers, and law enforcement on AI risks, evidence handling, and trauma-informed response.
What platforms and developers should ship next
- Tiered safety by default: Age-appropriate modes, private-by-default settings, limited contact/finderability, and strict DM controls for minors.
- Sextortion safeguards: Automated detection for grooming patterns and coercion scripts; friction and additional checks for high-risk features (image generation, unknown DMs, file sharing).
- Provenance and traceability: Embed content credentials and maintain audit trails for synthetic media. Offer trusted signals to downstream platforms for faster moderation.
- Red-teaming on child safety: Test against grooming, deepfake extortion, and doxxing scenarios. Fix failure modes before release.
- Guardrails in genAI: Block prompts targeting minors, sexualization of minors, and corrective fine-tuning to close jailbreaks. Monitor embeddings and outputs for misuse patterns.
- Crisis response: One-tap reporting, rapid image hashing and takedown, survivor support workflows, and law-enforcement escalation with due process.
- Data minimization: Short retention, strict access controls, and strong abuse logging to reduce insider risk.
Schools and parents: practical moves
- AI literacy: Teach children how grooming works, what sextortion looks like, and how synthetic media can fake faces, voices, and messages.
- Boundaries and reporting: Lock down contact settings, disable geotags, and practice reporting steps. Encourage "pause, don't reply, save evidence, tell an adult."
- Device and app hygiene: Updates, strong passwords, passkeys, and MFA on key accounts. Review default privacy settings after every major app update.
- Support first: If abuse occurs, preserve evidence, avoid shame, involve school safeguarding leads, and contact specialist hotlines.
The social media ban debate
Age-based bans can reduce exposure but often push activity to unsupervised spaces, create workarounds, and miss the root issue: high-risk design and weak enforcement. The UN emphasis points elsewhere-clear criminalization, safety-by-design standards, real transparency, and trained responders.
Set the age floor if you choose, but measure what actually reduces harm: fewer grooming attempts reaching minors, faster takedowns, better survivor outcomes, and lower prevalence of AI-generated abuse material.
Quick checklist by audience
- Government: Update laws, require child-rights impact assessments, fund capacity, and enforce transparency.
- IT and Security: Deploy content provenance, risk scoring for high-risk interactions, and incident playbooks.
- Product and Engineering: Default protections for minors, strict genAI guardrails, and dedicated red-teaming for child safety.
- Education and Care: Teach AI safety basics, set clear device rules, and make reporting pathways obvious and judgment-free.
If you're building internal training for teams on safe and responsible AI use, see curated options by role here: Complete AI Training - Courses by Job.
Your membership also unlocks:
