Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

AI agent for prompt engineers

User Feedback to Prompt Fix Agent

A tested prompt change for the biggest feedback cause with no regressions

Quality controlAutomationIT and development
See how it worksGet access
User Feedback to Prompt Fix Agent: what goes in, what the agent does and what you get

What it does

Thumbs-down signals arrive daily and rarely lead to a change. This agent reads the negative feedback and its comments, groups it by likely cause such as wrong tone, missing facts, format errors or refusals, and ranks the groups by size. For the top group it finds real conversations and reproduces the failure by running the same input through the current prompt. If it cannot reproduce, it marks the case as noise. For reproducible cases it drafts a prompt change, tests it on those cases and on the standard regression set, and compares scores. If regressions appear, it revises the change and tests again. The engineer approves the final change and its release. Edge case: two groups need opposite fixes, so the agent writes both and asks the engineer to choose.

How it works

Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.

Start and resultWhat it doesA check on its own workWaits for your OKGoes back and retries
Yes, continueYes, continueApprovedNoNo 1 STARTS WHEN Weekly feedback export arrives 2 USES A TOOL Read negative feedback and comments 3 DOES Group items by likely cause and rank groups by size 4 USES A TOOL Rerun sample inputs from the top group on thecurrent prompt 5 CHECKS THE RESULT Can the failure be reproduced? If not: mark non-reproducible items as noise and move tothe next group. Back to step 4. 6 DOES Draft a prompt change for the reproduced cases 7 USES A TOOL Test the change on the cases and on the regressionset 8 CHECKS THE RESULT Do targeted cases improve with no regression? If not: revise the change to fix the regression andretest. Back to step 6. 9 YOU APPROVE Engineer approves the change and its release 10 RESULT Change proposal with before and after scores
Read the steps as a list
  1. Weekly feedback export arrives
  2. Read negative feedback and comments
  3. Group items by likely cause and rank groups by size
  4. Rerun sample inputs from the top group on the current prompt
  5. Can the failure be reproduced?If not: mark non-reproducible items as noise and move to the next group. Back to step 4.
  6. Draft a prompt change for the reproduced cases
  7. Test the change on the cases and on the regression set
  8. Do targeted cases improve with no regression?If not: revise the change to fix the regression and retest. Back to step 6.
  9. Engineer approves the change and its releaseThe agent waits here for your OK.
  10. Change proposal with before and after scores

How it decides

It fixes the largest reproducible cause first, and keeps a change only if the targeted cases improve and the regression set does not drop.

  • Fix the largest reproducible group first
  • Treat non-reproducible items as noise unless they repeat next week
  • Reject a change that drops the regression set by more than 1 point
  • Present both options when two groups conflict

Make it yours

Every agent is a starting point. You choose these settings for your own situation.

  • Minimum group size to act on (default 20)
  • Regression drop allowed (default 1 point)
  • Feedback sources
  • Day of the week it runs

What keeps you in control

It always asks you first

  • Prompt change
  • Release to users

Hard limits

  • Never changes the live prompt
  • Never stores user text outside the test run

It stops when

  • Done: change approved and regression set holds
  • Stop: feedback has no logs to reproduce from

Set it up

We guide you through the set-up, step by step

Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.

10 minto set it up in your AI
5 AIsChatGPT, Claude, Copilot, Gemini, Grok
  • One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
  • The agent then walks you through connecting your own data, one source at a time
  • A downloadable copy with the flow chart, the rules and the full guide
Get access to this agent

An example run

What happensThis week 212 thumbs-downs came in. The largest group, 61 items, said answers were too long. The agent reproduced 48, drafted a length rule and tested it, with targeted cases improving from 52 to 81. But the regression set dropped 4 points on detailed requests. It added an exception for users who ask for detail, and the drop closed to 0. The engineer approved.

More agents for prompt engineers

Risk management · IT and development

Prompt Injection Red-Team Agent

Find the ways your prompt can be attacked and fix them without hurting normal use

11 steps1 check3 jobs
Quality control · IT and development

Golden Test Set Maintenance Agent

Keep the test set current so real failures become permanent tests

11 steps2 checks5 jobs
Decisions · IT and development

Model Selection Bake-Off Agent

Choose a model with evidence on quality, cost and speed for the real task

11 steps2 checks7 jobs
Data cleanup · IT and development

Prompt Library Cleanup Agent

Leave a smaller library where each purpose has one tested, owned prompt

11 steps2 checks1 job