AI agent for product analysts
Product Experiment Sample Integrity Agent
Experiment samples that are valid before anyone draws conclusions.
What it does
Experiment results are only as good as the sample behind them. Duplicate users, reused anonymous IDs and outcomes recorded before exposure can quietly bias a test. This agent runs before anyone analyzes an experiment. It reads the experiment rules and event exports, checks how the sample was built, and runs deterministic checks for duplicates, missing exposure records and timing errors. When records fail, it investigates them, decides whether they should be excluded under the written rules, and reruns the checks. It then recomputes the comparison inputs and writes a sample integrity review. It never draws conclusions about which variant won, and changes to user targeting stay with the experiment owner. Edge case: one anonymous ID shared by two signed-in users is excluded, not merged.
How it works
Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.
Read the steps as a list
- Analysis requested
- Load experiment rules and event exports
- Check sample construction
- Are all records valid?If not: investigate anomalous records and apply the exclusion rules. Back to step 3.
- Is the excluded share under the threshold?If not: flag the sample as unreliable and ask the owner whether to re-export. Back to step 2.
- Recompute comparison inputs
- Analyst approves the cleaned sampleThe agent waits here for your OK.
- Sample-integrity review
How it decides
It checks duplicates, missing exposure and timing.
- Exclude outcomes that have no matching exposure record.
- Reused anonymous IDs across accounts are excluded, never merged.
- If exclusions exceed the threshold, stop and ask instead of continuing.
- The agent reports sample quality only and never states which variant won.
Make it yours
Every agent is a starting point. You choose these settings for your own situation.
- Exclusion rules for duplicates and reused IDs (default: exclude, never merge)
- Maximum share of excluded records before the sample is flagged as unreliable (default 2%)
- Data sources and event tables to read
- Who signs off the cleaned sample (default: experiment analyst)
- Report format (default: one-page integrity review plus record list)
What keeps you in control
It always asks you first
- Experiment conclusions
- User targeting
Hard limits
- No conclusions.
It stops when
- Done: valid inputs.
Set it up
We guide you through the set-up, step by step
Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.
- One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
- The agent then walks you through connecting your own data, one source at a time
- A downloadable copy with the flow chart, the rules and the full guide
