Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

AI agent for community moderators

Serious Harm Escalation Agent

Serious cases escalated fast, completely and safely

Risk managementComplianceCustomer support
See how it worksGet access
Serious Harm Escalation Agent: what goes in, what the agent does and what you get

What it does

Content about violence, self-harm or child safety cannot wait in the normal queue, and a sloppy handoff can lose key evidence. When this agent detects such content, it moves it out of the normal queue and alerts a senior moderator. It preserves the evidence as the escalation policy requires, collects the user's account details for the trust and safety team, and drafts the escalation report. It checks the report is complete: content ID, time stamps, account info and severity. If anything is missing, it collects it before going on. For self-harm cases it drafts the approved support resources message for the moderator. A senior moderator approves every escalation and any report to outside authorities. Edge case: child safety content is never copied around, only referenced by its ID.

How it works

Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.

Start and resultWhat it doesA check on its own workWaits for your OKGoes back and retries
Yes, continueApprovedNo 1 STARTS WHEN Serious harm content detected 2 DOES Move it out of the normal queue and alert a seniormoderator 3 USES A TOOL Preserve evidence as policy requires 4 USES A TOOL Collect account details from the account record 5 USES A TOOL Draft the escalation report 6 CHECKS THE RESULT Does the report have content ID, times, account infoand severity? If not: collect the missing items from records beforegoing on. Back to step 4. 7 DOES Draft the support resources message for self-harmcases 8 YOU APPROVE Senior moderator approves escalation and any outsidereport 9 RESULT Case handed to trust and safety and logged
Read the steps as a list
  1. Serious harm content detected
  2. Move it out of the normal queue and alert a senior moderator
  3. Preserve evidence as policy requires
  4. Collect account details from the account record
  5. Draft the escalation report
  6. Does the report have content ID, times, account info and severity?If not: collect the missing items from records before going on. Back to step 4.
  7. Draft the support resources message for self-harm cases
  8. Senior moderator approves escalation and any outside reportThe agent waits here for your OK.
  9. Case handed to trust and safety and logged

How it decides

It follows the escalation policy for each harm type and treats unclear cases as serious until a person decides.

  • Unclear cases are treated as serious
  • Child safety content is referenced by ID only
  • Self-harm cases get the support resources message

Make it yours

Every agent is a starting point. You choose these settings for your own situation.

  • Harm types and their routes
  • Who gets alerts out of hours
  • Support resources by country
  • Evidence retention rules

What keeps you in control

It always asks you first

  • Escalation to trust and safety
  • Reports to outside authorities

Hard limits

  • Never contacts authorities itself
  • Never stores or forwards illegal content

It stops when

  • Done: case handed over
  • Stop: never stops itself; only a senior moderator closes a case

Set it up

We guide you through the set-up, step by step

Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.

10 minto set it up in your AI
5 AIsChatGPT, Claude, Copilot, Gemini, Grok
  • One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
  • The agent then walks you through connecting your own data, one source at a time
  • A downloadable copy with the flow chart, the rules and the full guide
Get access to this agent

An example run

What happensOn Thursday at 6 p.m., a post said someone would bring something to a meetup on Saturday. The agent flagged it as a possible threat, preserved it and drafted the report. The completeness check failed because the account email was missing; the user had deleted their profile. It pulled the email from the account record, and the senior moderator approved the escalation within 20 minutes.

More agents for community moderators

Risk management · Customer support

Spam and Scam Report Triage Agent

Clear scam and spam reports quickly while keeping false removals low.

11 steps2 checks2 jobs
Risk management · Customer support

New Member Welcome and Rules Check Agent

Reduce rule breaks from new members through timely, friendly guidance.

11 steps2 checks3 jobs
Risk management · Monitoring

Raid and Coordinated Abuse Response Agent

Contain a raid within minutes and prepare an accurate ban list.

11 steps2 checks1 job
Planning · Scheduling

Moderator Wellbeing Rotation Agent

Spread harmful content exposure fairly while keeping every queue covered.

11 steps2 checks1 job