Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

AI agent for data scientists

Model Fairness Audit Agent

A documented review of how a model performs across groups, with tested mitigations and tradeoffs

AnalysisIT and development
See how it worksGet access
Model Fairness Audit Agent: what goes in, what the agent does and what you get

What it does

Models are checked for accuracy but not for uneven results across groups. This agent computes model performance by group and tests the gaps against your threshold. If a gap is too large, it tries mitigations such as changing decision thresholds for each group or reweighting data, and it measures the cost to overall accuracy. It then rechecks the gap after each mitigation. It writes a draft fairness review with the numbers and tradeoffs. The owner approves the model for release. Edge case: one group has very few records, so the agent states the numbers are too small to judge and suggests collecting more.

How it works

Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.

Start and resultWhat it doesA check on its own workWaits for your OKGoes back and retries
Yes, continueYes, continueApprovedNoNo 1 STARTS WHEN Model is ready for review 2 USES A TOOL Compute performance by group 3 DOES Compare the gaps with the thresholds 4 CHECKS THE RESULT Does each group have enough records to judge? If not: mark it too small to judge and request moredata. Back to step 2. 5 DOES Try mitigations such as threshold changes orreweighting 6 USES A TOOL Measure the gap and accuracy after each mitigation 7 CHECKS THE RESULT Is the gap within the threshold at acceptableaccuracy cost? If not: try the next mitigation or report that the gapremains. Back to step 5. 8 DOES Draft the fairness review with the tradeoffs 9 YOU APPROVE Owner approves the model for release 10 RESULT Fairness review on file
Read the steps as a list
  1. Model is ready for review
  2. Compute performance by group
  3. Compare the gaps with the thresholds
  4. Does each group have enough records to judge?If not: mark it too small to judge and request more data. Back to step 2.
  5. Try mitigations such as threshold changes or reweighting
  6. Measure the gap and accuracy after each mitigation
  7. Is the gap within the threshold at acceptable accuracy cost?If not: try the next mitigation or report that the gap remains. Back to step 5.
  8. Draft the fairness review with the tradeoffs
  9. Owner approves the model for releaseThe agent waits here for your OK.
  10. Fairness review on file

How it decides

A gap is a problem when it exceeds the threshold and the group has enough records. A mitigation is kept when it narrows the gap at an acceptable accuracy cost.

  • Use the metrics set by the policy
  • Require a minimum group size
  • Accept a mitigation only within the accuracy cost limit
  • Report any gap that remains

Make it yours

Every agent is a starting point. You choose these settings for your own situation.

  • Fairness metrics
  • Gap threshold (default: 5 points)
  • Accuracy cost limit (default: 2 points)
  • Minimum group size

What keeps you in control

It always asks you first

  • Owner approves the model for release

Hard limits

  • Never releases a model
  • Never hides a gap that remains

It stops when

  • Done: the review is complete and the owner decides
  • Stop: no group labels are available

Set it up

We guide you through the set-up, step by step

Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.

10 minto set it up in your AI
5 AIsChatGPT, Claude, Copilot, Gemini, Grok
  • One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
  • The agent then walks you through connecting your own data, one source at a time
  • A downloadable copy with the flow chart, the rules and the full guide
Get access to this agent

An example run

What happensA loan model had an approval rate gap of 9 points between two groups against a 5 point limit. A threshold change cut it to 4 points but lowered accuracy by 1.2 points, within the 2 point limit. One small group had 60 records, so it was marked too small to judge. The owner approved with a note to collect more data.

More agents for data scientists

Quality control · Science and research

Research Data Intake Validation Agent

A validated, reproducible working dataset with raw data untouched.

7 steps2 checks14 jobs
Risk management · Compliance

AI Model Risk Register Agent

A current register of live models with owners, reviews and risk status

10 steps2 checks6 jobs
Quality control · IT and development

A/B Test Validity Check Agent

A test result read only when the data is valid, with a clear recommendation

10 steps2 checks7 jobs
Forecasting · IT and development

Forecast Backtest and Refresh Agent

Forecasts that improve over time because every method is tested on the same history

9 steps2 checks8 jobs