A single AI coding step that's 98% accurate sounds reliable. Chain 34 such steps together without verification, and the overall probability of correctness drops to just 50.5% - barely better than a coin flip. That's the math of compound uncertainty, and it has direct consequences for developers building safety-critical systems where failure is not an option.
The calculation is simple: 0.98^34 = 0.505. Human intuition handles exponential decay poorly, so the risk often goes unnoticed. When an AI agent generates code, runs tests, or updates documentation across dozens of steps in an open-loop pipeline, small per-step errors accumulate silently. In most software contexts, a failing test will catch the mistake and the agent will retry. In safety-critical domains - medical devices, avionics, automotive braking - the cost of a missed error is measured in lives, not sprint velocity.
What verification can and cannot fix
Two recent projects show the promise and the limits. Andreas Kling used AI agents to port Ladybird's LibJS engine from C++ to Rust, producing 25,000 lines of Rust with zero regressions across 65,359 tests and byte-for-byte identical bytecode. The Bun JavaScript runtime went further: AI agents rewrote roughly one million lines of Zig to Rust across 6,755 commits, passing 99.8% of the test suite. Both projects kept a human in the loop and ran verification at every step.
Yet the Bun rewrite left 13,044 unsafe Rust blocks. A comparable handwritten project would have around 73. A passing test suite doesn't surface that safety debt - and it stops a safety-critical certification in its tracks. ISO 26262, DO-178C, and IEC 62304 do not accept self-generated test results as qualified verification evidence. The standards require deterministic tools, documented configurations, and traceable outputs that are generated as the work happens, not reconstructed afterward.
The emerging ISO/PAS 8800 standard, watched closely by industries beyond automotive, uses the term functional insufficiency to describe AI errors that arise because the training data didn't cover a specific condition. This is more precise than "hallucination." It means the system performed exactly as its training data suggested, and the gap was in the data, not a random invention. You can't fix it by improving the model alone - you have to bound it, monitor it, and prevent it from propagating into a safety decision unchecked.
Gates, not better models
Safety-critical engineering has always dealt with unreliable components by using gates: static analysis, unit tests, and coverage requirements that reset accumulated uncertainty. DO-178C demands 100% MC/DC coverage at the highest criticality level. ISO 26262 requires the same at ASIL D. Each gate verifies that the output meets a known specification, pushing the probability of error back toward zero before the next stage compounds it.
AI-generated code complicates this because the original intent is not recoverable. A human developer who writes an uninitialized variable made a mistake; a static analysis tool flags it, and the developer understands the context. An AI agent's output is statistically consistent with its training - often correct, occasionally wrong in ways that look right. When a gate catches a MISRA violation or a failed boundary condition, the reviewer is one step removed from any deliberate choice. That missing intent matters for certification audits that require a safety argument, not just a passing artifact.
The answer is not to abandon AI in safety-critical workflows. These tools already accelerate development, surface defects earlier, and handle repetitive verification work that exhausts engineers. The challenge is to position verification gates so they catch what an AI agent introduces, and to ensure the evidence trail remains intact. In the world of AI Agents & Automation, treating outputs as probabilistic rather than deterministic is the first step toward managing that risk.
Where the uncertainty lives
Many teams plan to add verification after the fact - a requirements review, then code generation, then documentation and traceability updates, with testing saved for the end. That's an open-loop pipeline with compounding uncertainty baked in. At 98% per-step accuracy across 34 stages, you've crossed the coin-flip line before running a single test. The fix isn't a better model; it's inserting gates early and often.
Static analysis enforces coding patterns that have caused failures before. Unit tests verify components under known conditions. Full structural coverage ensures every line, branch, and condition is exercised. These practices existed long before AI coding assistants, and they remain the most reliable way to manage Generative Code that arrives without a human author's deliberate reasoning.
Why this matters for IT and Development
Developers and engineering leads working on safety-critical systems need to treat AI-generated code as a probabilistic input, not a deterministic one. Compound uncertainty means that a 98%-accurate agent in a 34-step pipeline is more likely wrong than right unless verification gates reset the error at each stage. The standards that govern your industry - ISO 26262, DO-178C, IEC 62304 - already demand traceable evidence and complete coverage. AI doesn't change those requirements; it makes meeting them more urgent. Build your pipeline so that every AI step is followed by a gate that produces auditable proof, and never rely on self-verification alone.
Your membership also unlocks:
