AI agent for data engineers
Research Data Intake Validation Agent
A validated, reproducible working dataset with raw data untouched.
What it does
Research datasets often arrive with schema, unit or provenance problems that are only discovered months later, after analysis has started. This agent runs when a dataset is received. It validates the file structure and metadata against the data dictionary, then runs unit and range checks on every column without touching the raw files. Missing units, unknown sources or impossible values are logged as exceptions for the researcher, who decides what they mean and whether anything should be excluded. Only transformations the researcher has approved are applied, and only to a cleaned working copy, which is checked again after each transformation. Edge case: a temperature column with no unit is logged, never assumed to be Celsius because the numbers look plausible.
How it works
Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.
Read the steps as a list
- Dataset received
- Validate structure and metadata
- Run unit and range checks
- Are units and provenance complete?If not: log an exception for the researcher. Back to step 3.
- Apply approved transformations to a working copy
- Did the transformation pass its checks?If not: correct and rerun. Back to step 5.
- Intake report and working dataset
How it decides
It applies only approved transformations and stops on unknown units.
- Raw data is never altered.
- Unknown units are logged, never guessed.
- Only approved transformations are applied, to a working copy.
- Exclusion decisions belong to the researcher.
Make it yours
Every agent is a starting point. You choose these settings for your own situation.
- Data dictionary and required metadata fields
- Range limits per variable
- Which transformations are pre-approved (default: none)
- Who decides on exceptions (default: lead researcher)
- Intake report format
What keeps you in control
It always asks you first
- Changing raw data
- Excluding observations
- Scientific interpretation
Hard limits
- Raw data read-only.
It stops when
- Done: working copy validated.
Set it up
We guide you through the set-up, step by step
Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.
- One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
- The agent then walks you through connecting your own data, one source at a time
- A downloadable copy with the flow chart, the rules and the full guide
An example run
More agents for data engineers
Government Data Publication Preflight Agent
Datasets that are valid and safe to publish.
Insurance Bordereaux Validation Agent
Bordereaux that pass schema and reconcile to control totals.
Database Migration Rehearsal Agent
Migrations rehearsed safely with schema and data intact.
Data Pipeline Schema Adaptation Agent
Pipelines adapt to upstream changes without corrupting data.