A startup building runtime governance for AI agents today released run-assert-eval, a skill that automates the full cycle of discovering agent risks, measuring failure rates, generating policy, and confirming whether the fix worked. The release addresses two weak points in current safety practices: teams often miss the risks that matter because no one thought to write them down, and stitching together evaluation and enforcement still requires manual effort that few teams can sustain.
How it works
The new skill runs in four steps. First, it probes the agent to uncover failure modes that are not covered by existing written requirements. It then measures how often the agent fails on those discovered risks. From those findings, it generates runtime policy directly. Finally, it reruns the evaluation to verify that the policy actually reduced the failures.
This builds on foundations the company has been releasing since June. The earlier ASSERT tool converts written requirements into structured evaluations. A companion specification, Agent Control Specification, gives teams a portable way to enforce policy at the points where agents take action. In August, the company demonstrated how the two work together: a team evaluates its agent against requirements, applies a control, freezes the test set, and measures both safety and helpfulness before and after the change.
The assumptions that break in practice
That workflow depends on two conditions that often do not hold. As the company put it, "the most consequential failures are often the ones no one thought to write down." A team's documented requirements rarely capture every risk that will surface once an agent operates in the wild. The second assumption is that someone has the time and expertise to connect every step by hand, translating findings into policy and rebuilding the comparison. In operations teams already stretched thin, that manual handoff becomes a bottleneck.
Run-assert-eval removes both dependencies. By discovering risks automatically and generating policy from the results, it closes the gap between what teams can specify in advance and what actually goes wrong at runtime.
Where this fits in the operations workflow
For IT and development teams responsible for agent behavior in production, the release shifts governance left while keeping it tied to runtime evidence. Instead of writing policy based on assumptions, teams can generate it from observed failure patterns. The closed-loop design - discover, measure, enforce, retest - provides a repeatable process rather than a one-off review.
The approach also preserves the comparison structure from the earlier workflow. Teams still freeze the test set and measure safety versus helpfulness before and after applying controls. What changes is that the test set now includes risks the agent actually exhibits, not just risks someone anticipated.
Why this matters for IT and operations teams
Running AI agents in production means accepting that failures will occur in ways you did not predict. The operational question is how fast you can detect those failures, enforce a fix, and confirm it worked. Run-assert-eval compresses that loop into a single automated skill. For teams managing multiple agents or frequent updates, that automation reduces the manual toil of writing evaluations and policies for risks you have not yet seen. The output is not just a policy document - it is policy that has already been tested against the failures that triggered it.
Your membership also unlocks:
