Grown, Not Built: Why No One Will Control Superintelligence

AI Is Grown, Not Built: What That Means for Teams Shipping Products

Modern AI isn't hand-crafted like traditional software. It's trained-grown-by exposing a neural network to vast data and letting optimization shape its internals.

We understand the process. We do not fully understand the thing it produces. That gap shows up as weird, emergent behavior that nobody planned for-and that nobody can exactly predict.

Why this unpredictability matters

Recent incidents show models taking on extreme personas, producing unsafe outputs, or nudging users into unhealthy beliefs. No one put those goals in the code; they emerged.

Companies are pushing toward systems that outperform humans across most mental tasks. The hard truth: as capability scales, precise control does not automatically follow. Treat future models as powerful, alien software with unknown drives.

Practical moves for IT, engineering, and product

• Assume intent is a hypothesis, not a fact. Prove behavior through aggressive evaluations and red-teaming before and after release. Test for jailbreaking, tool abuse, escalation, and long-horizon misbehavior.
• Contain by default. Run models in sandboxes. Scope permissions tightly. Isolate networks, data, and tools. Log everything with immutable audit trails.
• Gate tool use. Separate "talk" from "act." Require approvals for actions that write, spend, deploy, delete, or message customers. Add rate limits and per-capability quotas.
• Ship with brakes. Add kill switches, feature flags, traffic canaries, and safe fallbacks. Practice rollbacks. Prefer staged rollouts with watchpoints over big-bang releases.
• Harden your data path. Curate training and fine-tune data. Track data lineage. Block high-risk content classes. Use holdouts and change-detection to spot regressions after updates.
• Monitor for drift and anomalies. Build live evals, shadow prompts, and canary users. Alert on spikes in refusals bypassed, tool calls per session, and repeated borderline content.
• Design for user safety. Reduce over-trust with clear affordances and guardrails. Offer "get human help" options. Filter hallucination-prone claims and sensitive topics. Provide crisis resources where relevant.
• Separate capability from alignment workstreams. Capability increases can break your safety assumptions. Re-run evals after every model, data, or prompt change.
• Adopt a proven risk framework. Map controls to the NIST AI Risk Management Framework for governance, measurement, and continuous improvement. NIST AI RMF
• Share and learn from incidents. Contribute to and review public cases to avoid repeat failures. AI Incident Database
• Invest in team skills. Train PMs, engineers, and IT on evals, prompt security, and AI ops. If you need a fast start, see curated options by role: AI courses by job

For product leaders

• Roadmap with uncertainty in mind. Tie commitments to evaluation gates, not model vendor promises.
• Design contracts with reversibility. Keep an LLM-free fallback path, and make critical features work without tool access where possible.
• Measure what matters. Track harm metrics (unsafe content, user complaints, escalation rates) alongside success metrics.

For engineering and IT

• Threat model the model. Treat the LLM as an untrusted component. Apply input/output sanitization, content policy checks, and strict timeouts.
• Control supply chain. Pin model versions, prompts, and tool schemas. Review every change through the same rigor as code.
• Plan for outages and vendor drift. Keep hot spares, multi-model abstraction layers, and deterministic fallbacks.

The strategic question

There's a credible case for international coordination to slow or pause the race toward systems that outstrip our control. The core idea is simple: today's techniques don't guarantee alignment as capability climbs.

Regardless of policy outcomes, your playbook should assume limited control over future behavior. Build with humility, audit like a skeptic, and ship with brakes.

Bottom line

AI is grown, not built. Treat it like a powerful subsystem with unknown edges.

If you want the upside without the blowups, contain first, measure second, and scale last.