Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

AI agent for prompt engineers

Retrieval Grounding Check Agent

A pipeline setting where nearly every claim is supported by a cited passage

Quality controlIT and development
See how it worksGet access
Retrieval Grounding Check Agent: what goes in, what the agent does and what you get

What it does

A search-based assistant may quote a source that does not say what the answer claims. This agent takes a set of test questions and runs them through the full pipeline. For each answer it splits the text into separate claims and checks each claim against the passages that were retrieved. Claims with no support are flagged, and so are citations that point to the wrong passage. It then looks for the cause. If the right passage was never retrieved, it adjusts retrieval settings such as passage count or chunk size. If the passage was there but ignored, it changes the prompt wording about sticking to sources. After each change it reruns the questions and compares the unsupported claim rate. It loops until the rate stops falling. The engineer approves the final settings. Edge case: a claim is true but only in a passage the pipeline never fetched.

How it works

Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.

Start and resultWhat it doesA check on its own workWaits for your OKGoes back and retries
Yes, continueYes, continueApprovedNoNo 1 STARTS WHEN Engineer supplies test questions and access to thepipeline 2 USES A TOOL Run each question and capture the answer andretrieved passages 3 DOES Split each answer into claims and match each claimto a passage 4 CHECKS THE RESULT Is the unsupported claim rate below the target? If not: find the cause for each miss, retrieval orprompt, and pick one change. Back to step 3. 5 USES A TOOL Adjust passage count, chunk size or the sourcinginstruction 6 USES A TOOL Rerun the questions with the new settings 7 CHECKS THE RESULT Did the unsupported rate fall without hurting answercoverage? If not: undo the change and try a different one. Back tostep 3. 8 DOES Record the final rate and the remaining unsupportedexamples 9 YOU APPROVE Engineer approves the new settings 10 RESULT Settings change log with before and after rates
Read the steps as a list
  1. Engineer supplies test questions and access to the pipeline
  2. Run each question and capture the answer and retrieved passages
  3. Split each answer into claims and match each claim to a passage
  4. Is the unsupported claim rate below the target?If not: find the cause for each miss, retrieval or prompt, and pick one change. Back to step 3.
  5. Adjust passage count, chunk size or the sourcing instruction
  6. Rerun the questions with the new settings
  7. Did the unsupported rate fall without hurting answer coverage?If not: undo the change and try a different one. Back to step 3.
  8. Record the final rate and the remaining unsupported examples
  9. Engineer approves the new settingsThe agent waits here for your OK.
  10. Settings change log with before and after rates

How it decides

A claim counts as supported only if a retrieved passage states it. The agent changes retrieval when the passage is missing and the prompt when the passage was ignored.

  • Flag any claim that no retrieved passage states
  • Change retrieval first if the right passage was never fetched
  • Reject a change that drops answered questions by more than 5 percent
  • Stop after the rate stops falling for two rounds

Make it yours

Every agent is a starting point. You choose these settings for your own situation.

  • Target unsupported rate (default 5 percent)
  • Settings it may change
  • Number of test questions
  • Citation format to check

What keeps you in control

It always asks you first

  • Settings or prompt changes before they go live

Hard limits

  • Never edits the document collection
  • Never changes live settings without approval

It stops when

  • Done: unsupported rate meets the target
  • Stop: test questions have no known answer source

Set it up

We guide you through the set-up, step by step

Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.

10 minto set it up in your AI
5 AIsChatGPT, Claude, Copilot, Gemini, Grok
  • One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
  • The agent then walks you through connecting your own data, one source at a time
  • A downloadable copy with the flow chart, the rules and the full guide
Get access to this agent

An example run

What happensOn 60 questions, 14 percent of claims had no supporting passage. Eight misses came from missing passages, so the agent raised passage count from 4 to 6. The rate dropped to 7 percent, but two answers became vague. It then tightened the sourcing instruction, the rate reached 3 percent, and coverage held. The engineer approved both changes.

More agents for prompt engineers

Risk management · IT and development

Prompt Injection Red-Team Agent

Find the ways your prompt can be attacked and fix them without hurting normal use

11 steps1 check3 jobs
Quality control · IT and development

Golden Test Set Maintenance Agent

Keep the test set current so real failures become permanent tests

11 steps2 checks5 jobs
Decisions · IT and development

Model Selection Bake-Off Agent

Choose a model with evidence on quality, cost and speed for the real task

11 steps2 checks7 jobs
Data cleanup · IT and development

Prompt Library Cleanup Agent

Leave a smaller library where each purpose has one tested, owned prompt

11 steps2 checks1 job