Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

AI agent for data engineers

Historical Backfill Planning Agent

A backfill that finishes with matching counts and checksums, without harming the source or downstream tables

PlanningRisk managementIT and development
See how it worksGet access
Historical Backfill Planning Agent: what goes in, what the agent does and what you get

What it does

A backfill of eighteen months looks simple until it slows the source database and doubles rows in a report table. This agent sizes the job first: row counts per day, the load each day puts on the source and the downstream tables it touches. It splits the work into safe date batches and runs a small test batch into a copy of the target. It then compares row counts and checksums with the source and checks whether the downstream tables stay correct. While the run goes on it watches source load and error rates and shrinks or pauses batches when they rise. The engineer approves the production run and any change in batch size beyond the agreed range. Edge case: a day with an unusual spike in rows is split into smaller pieces automatically.

How it works

Follow the arrows from top to bottom. The orange dashed arrow is the loop: when a check fails, the agent goes back and tries again.

Start and resultWhat it doesA check on its own workWaits for your OKGoes back and retries
Yes, continueApprovedYes, continueNoNo 1 STARTS WHEN Engineer requests a backfill 2 USES A TOOL Count rows per day and read downstream dependencies 3 DOES Size the job and split it into date batches 4 USES A TOOL Run a small test batch into a copy of the target 5 CHECKS THE RESULT Do row counts and checksums match the source? If not: find the mismatch, change the batch rule andrerun the test batch. Back to step 2. 6 YOU APPROVE Engineer approves the production run 7 USES A TOOL Run batches in production in order 8 CHECKS THE RESULT Is source load and error rate below the limit? If not: halve the batch size, pause for the cool-downand continue. Back to step 7. 9 USES A TOOL Compare final counts and checksums for every batch 10 RESULT Backfill report with batch results
Read the steps as a list
  1. Engineer requests a backfill
  2. Count rows per day and read downstream dependencies
  3. Size the job and split it into date batches
  4. Run a small test batch into a copy of the target
  5. Do row counts and checksums match the source?If not: find the mismatch, change the batch rule and rerun the test batch. Back to step 2.
  6. Engineer approves the production runThe agent waits here for your OK.
  7. Run batches in production in order
  8. Is source load and error rate below the limit?If not: halve the batch size, pause for the cool-down and continue. Back to step 7.
  9. Compare final counts and checksums for every batch
  10. Backfill report with batch results

How it decides

Batch size is set so the source stays under its load limit. A batch is accepted only when counts and checksums match the source.

  • Start with a test batch of one day or 1 percent of rows, whichever is smaller
  • Halve batch size when source load passes 70 percent of the limit
  • Split any day with more than 3 times the average row count
  • Stop and alert after 3 failed batches in a row

Make it yours

Every agent is a starting point. You choose these settings for your own situation.

  • Source load limit (default 70 percent)
  • Test batch size (default 1 day)
  • Cool-down pause between batches
  • Run window (for example nights only)

What keeps you in control

It always asks you first

  • Starting the production run
  • Changing batch size beyond the agreed range

Hard limits

  • Never writes to production tables without approval
  • Never deletes target data to retry

It stops when

  • Done: all batches match the source
  • Stop: source load stays above the limit after reducing batch size

Set it up

We guide you through the set-up, step by step

Members get the full set-up guide for this agent. No technical skills needed: you copy, paste and upload.

10 minto set it up in your AI
5 AIsChatGPT, Claude, Copilot, Gemini, Grok
  • One set of instructions to paste into your AI, with the clicks for ChatGPT, Claude, Microsoft 365 Copilot, Gemini and Grok
  • The agent then walks you through connecting your own data, one source at a time
  • A downloadable copy with the flow chart, the rules and the full guide
Get access to this agent

An example run

What happensA request covered 18 months of orders, about 62 million rows. The agent planned 540 daily batches and tested one on 3 March. The checksums failed because of a timezone cut-off, so it shifted the day boundary and the retest matched. After approval the production run slowed the source at batch 120, so the agent halved batch size and added a pause. All 540 batches matched.

More agents for data engineers

Quality control · Risk management

Government Data Publication Preflight Agent

Datasets that are valid and safe to publish.

7 steps2 checks2 jobs
Quality control · Data cleanup

Insurance Bordereaux Validation Agent

Bordereaux that pass schema and reconcile to control totals.

7 steps1 check2 jobs
Quality control · Coding

Database Migration Rehearsal Agent

Migrations rehearsed safely with schema and data intact.

7 steps1 check4 jobs
Coding · IT and development

Data Pipeline Schema Adaptation Agent

Pipelines adapt to upstream changes without corrupting data.

8 steps2 checks2 jobs