Complete AI Training

AI agent for data engineers

Research Data Intake Validation Agent

A validated, reproducible working dataset with raw data untouched.

Research Data Intake Validation Agent: what goes in, what the agent does and what you get

What it does

Research datasets often arrive with schema, unit or provenance problems that are only discovered months later, after analysis has started. This agent runs when a dataset is received. It validates the file structure and metadata against the data dictionary, then runs unit and range checks on every column without touching the raw files. Missing units, unknown sources or impossible values are logged as exceptions for the researcher, who decides what they mean and whether anything should be excluded. Only transformations the researcher has approved are applied, and only to a cleaned working copy, which is checked again after each transformation. Edge case: a temperature column with no unit is logged, never assumed to be Celsius because the numbers look plausible.

How it works

Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.

Start and resultWhat it doesA check on its own workWaits for your OKGoes back and retries
Yes, continueYes, continueNoNo 1 STARTS WHEN Dataset received 2 USES A TOOL Validate structure and metadata 3 USES A TOOL Run unit and range checks 4 CHECKS THE RESULT Are units and provenance complete? If not: log an exception for the researcher. Back tostep 3. 5 USES A TOOL Apply approved transformations to a working copy 6 CHECKS THE RESULT Did the transformation pass its checks? If not: correct and rerun. Back to step 5. 7 RESULT Intake report and working dataset
Read the steps as a list
  1. Dataset received
  2. Validate structure and metadata
  3. Run unit and range checks
  4. Are units and provenance complete?If not: log an exception for the researcher. Back to step 3.
  5. Apply approved transformations to a working copy
  6. Did the transformation pass its checks?If not: correct and rerun. Back to step 5.
  7. Intake report and working dataset

How it decides

It applies only approved transformations and stops on unknown units.

  • Raw data is never altered.
  • Unknown units are logged, never guessed.
  • Only approved transformations are applied, to a working copy.
  • Exclusion decisions belong to the researcher.

Make it yours

Every agent is a starting point. You choose these settings for your own situation.

  • Data dictionary and required metadata fields
  • Range limits per variable
  • Which transformations are pre-approved (default: none)
  • Who decides on exceptions (default: lead researcher)
  • Intake report format

What keeps you in control

It always asks you first

  • Changing raw data
  • Excluding observations
  • Scientific interpretation

Hard limits

  • Raw data read-only.

It stops when

  • Done: working copy validated.

Set it up

We guide you through the set-up, step by step

Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.

10 minto set it up in your AI
5 AIsChatGPT, Claude, Copilot, Gemini, Grok
  • One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
  • The agent then walks you through connecting your own data, one source at a time
  • A downloadable copy with the flow chart, the rules and the full guide
Get access to this agent

An example run

What happensA 14-site field dataset arrives on March 4 with 62 columns. The agent finds column temp_c3 has no unit and 18 soil pH values above 14. The check fails, so it logs both for the researcher. She confirms Celsius and approves excluding the 18 values. The agent applies this to the working copy, reruns the range checks, which pass, and issues the intake report.

More agents for data engineers