AI Video Beyond the "Vibe Check": The Indispensable Role of Evals in AI's Next Frontier
The claim that a major AI coding agent business ran on "vibes" lit up the engineering community. In a Latent Space conversation with Ankur Goyal (Braintrust) and Malte Ubl (Vercel), the debate landed somewhere smarter than vibes vs. science. Product teams need a stack of feedback loops-each with a clear job, cost, and speed.
AI is "non-deterministic magic," as Ankur put it. That means product development needs stronger feedback than gut feel, but it also needs speed. The winning move: combine vibe checks, offline evals, and online experiments into a system that compounds insight every week.
Why you need multiple feedback loops
As Malte said, "I want to know if I'm doing well, and how fast can I find out." That's the product mandate. Instant signal to guide small iterations. Deeper signal to de-risk big bets. Both matter.
- Vibe checks: fast, subjective, high-signal for early drafts and UX.
- Offline evals: repeatable, cheap to run, great for gating regressions.
- A/B tests: truth under traffic, costlier but decisive for product impact.
Production-driven offline evals
Static "golden" datasets go stale. Top teams mine real failures from logs daily and turn them into eval cases. That keeps tests anchored to actual user pain, not guesswork.
- Automate failure harvesting from production.
- Convert each failure into a reusable test with expected outputs.
- Run the suite on every model or prompt change, block regressions.
This lets teams ship fast with confidence-because yesterday's bugs become today's guardrails.
Why coding agents are perfect for evals
Coding agents give you objective, binary signals: does it compile, do tests pass, does it render without errors. Vercel uses these signals to fine-tune models with RL and fix trivial errors faster than humans or heavy agent loops.
Lesson for product teams: build more verifiable checkpoints. The more binary your signals, the faster your iteration loop. Treat "green" states (pass/compile/render) as rewards and push them into training pipelines.
Evals as a product management tool
PMs are moving beyond PRDs. They're writing rubrics, designing LLM-as-judge prompts, and encoding domain nuance directly into evals. Finance? Healthcare? Compliance? Put the rules into the scoring functions and make them measurable.
- Define quality rubrics as code, not prose.
- Use staged scoring: correctness first, then usefulness, then UX polish.
- Version your rubrics and track drift like any other artifact.
The lab advantage (and what to do about it)
Large AI labs build private eval systems that double as competitive moats. Public benchmarks help with marketing but rarely reflect your product. Your answer: build in-house evals that mirror your customers, workflows, and constraints.
If it matters to your business, it should live inside your eval suite. Treat it as IP.
RL environments and the reward-hacking trap
RL offers scalable evaluation for computer-use agents without expensive human labels. But it's easy to optimize for the metric and miss the real goal. Known problem. Seen it before.
If you're going down this path, study specification gaming patterns and design checks that catch shortcuts early.
- Design composite rewards (correctness, latency, safety, UX) instead of one metric.
- Add "tripwire" tests that punish obvious shortcuts.
- Regularly audit agent behavior with human spot checks.
DeepMind's overview of specification gaming is a useful primer.
The practical playbook: build your eval ladder
- Define north-star outcomes: user success rate, time to value, cost per task, defect rate.
- Codify vibe checks: short live reviews for new flows; time-box to protect speed.
- Stand up offline evals: start small, update daily from production failures.
- Gate the CI: any model/prompt change must pass the suite before merge.
- Run A/Bs for material launches: predefine metrics, sample size, and stop rules.
- Instrument verifiable signals: compiles, tests pass, render OK, API success.
- Automate trivial fixes: RL or scripted "autopatchers" for known error classes.
- Put PMs in the loop: they own rubrics and LLM-as-judge prompts.
- Establish an eval review: weekly meeting to add cases, prune flaky tests, revisit weights.
Metrics that keep you honest
- Quality: task success rate, exact match, pass@k, hallucination rate.
- Speed: p50/p95 latencies, end-to-end time-to-answer.
- Cost: tokens per success, infra $ per task, re-run rates.
- Reliability: regression count per release, flaky-test rate, rollback frequency.
- User truth: retention lift, task completion lift, support ticket deltas.
Track them as a dashboard. Tie launches to movements in these numbers. No vanity metrics.
Org moves that make this stick
- Assign ownership: an "evals lead" with authority across product, eng, data.
- Budget for infra: eval runners, dataset storage, and log pipelines.
- Close the loop: every incident becomes a new eval within 24 hours.
- Educate the team: short docs and lunch-and-learns on writing good evals.
If you want a fast primer for your PM and eng teams, explore curated AI courses by job role here: Complete AI Training - Courses by Job.
The bottom line
Vibes aren't the enemy. They're a high-signal, expensive scoring function. Use them to steer early work, then lock in gains with offline evals and A/B tests.
The teams that win treat evals like product. They ship faster, break less, and learn on every cycle.
One more resource worth bookmarking on online experiments: Ron Kohavi's experimentation site.
Your membership also unlocks:
