Ops For Agents, Not Apps
Most operational models were built for stability. Keep the app up. Keep the dashboard green. That mindset breaks when your "workloads" appear, act for seconds, and vanish.
Agentic systems behave like pop-up shops. Think Spirit Halloween, not a year-round department store. Your job shifts from keeping a few services alive to coordinating swarms of short-lived specialists.
Why Yesterday's Playbooks Buckle
Kubernetes was a breakthrough for long-running containers. It assumes workloads should stay up. Agents don't care. They spin up in response to a prompt or another agent, perform a task, spawn more agents, and disappear.
Patterns emerge and dissolve too fast for static runbooks. Extending old tools leads to brittle glue: one-off pipelines, per-agent configs, dashboards that never see the process before it ends. The problem isn't Kubernetes. It's the assumption of persistence.
Capacity vs. Consumption: A Better Mental Model
Split operations into three pieces:
- Capacity: Compute, storage, networking, data services, and security controls.
- Consumption: Agents and models that use capacity.
- Inference interface: The broker that lets agents access capacity briefly and safely.
Agents don't need to know where they run. They need a fast, policy-aware interface to the right data, tools, and permissions the moment they spin up-and clean teardown when they're done.
What The Interface Must Guarantee
- Ephemeral identity: Short-lived credentials and scoped roles per agent/session.
- Context on demand: A shared memory layer that retrieves and writes state across agents without tight coupling.
- Policy-first access: Data residency, PII handling, and tool permissions enforced at request time.
- Sandboxed execution: Isolated runtimes with controlled egress and time-to-live per agent.
- Observable-by-default: Session-level traces, structured events, and tamper-evident audit logs emitted in real time.
- Composable tooling: Pluggable models, connectors, and tools behind stable contracts. No per-agent infrastructure snowflakes.
Early Signals
In one experiment, Reuven Cohen and the Agentics Foundation showed outcome-driven prompting that spun up swarms for research, design, coding, and testing-without a fixed workflow. The system self-organized, launched agents, handed off work, and shut them down.
It wasn't turnkey. Deployment choices, data access, and tool wiring took multiple attempts. Still, it proved that when agents coordinate around outcomes, they can get useful work done-if the interface and guardrails are there.
Enterprise Scenario: Global Support, Local Rules
Picture customer service agents spinning up in-region. Each needs the right customer data, the right language model, and the right tool access. All must meet regional privacy rules and then cleanly retire.
Without a clean split between capacity and consumption, every new agent becomes a one-off. With it, you can scale swarms without rebuilding the stack every time.
The Underestimated Challenges
- Memory and handoff: How do agents share context so work isn't repeated or lost when processes end seconds later?
- Monitoring the invisible: Traditional dashboards miss processes that complete before they're scraped. You need stream-first telemetry and session traces.
- Trust and compliance: Policies vary by region and use case. You need composability to swap models, tools, or data sources without breaking the system.
A Practical Roadmap For Ops Teams
- Define the planes: Document capacity (compute, storage, data, network) and consumption (agents, models, tools). Treat the interface as product, not plumbing.
- Build the inference broker: Centralize policy checks, routing, and short-lived credentials. Every agent request flows through it.
- Adopt ephemeral identity: Issue per-session identities and time-boxed tokens. Sign agent requests and tool invocations.
- Stand up shared context: Combine an event bus with a vector store or key-value cache. Tag context with TTLs and lineage so agents can read/write safely.
- Secure by default: Isolate runtimes, restrict egress, and scan tools. Enforce least privilege at the tool and data layer.
- Rework observability: Emit real-time events on agent start/stop, tool calls, data reads/writes, and policy decisions. Aggregate into session-level traces with sampling.
- Set SLOs for outcomes: Track inference latency, success rate, policy denials, cost per outcome, and context hit rate.
- Control cost: Enforce per-outcome budgets, default TTLs, and autoscaling policies. Kill idle agents fast.
- Automate compliance: Policy-as-code for residency, PII, and model use. Region-aware routing and redaction at the interface.
- Design for swap-ability: Standard contracts for models and tools. Use contract tests to prevent lock-in and speed change.
Standards And Trust
Ops needs common formats for session identity, policy decisions, and trace events. That makes swarms interoperable across platforms and vendors. It also makes audits repeatable.
For governance patterns, the NIST AI Risk Management Framework is a useful anchor. Map its controls to your interface, not to each agent.
Open Questions
- What standards will define ephemeral ops: identity, policy, and tracing for agent swarms?
- How do we prove repeatability when the actors vanish? Session artifacts, signed traces, and deterministic prompts can help-how far is enough?
- Where should context live, and for how long, to balance cost, privacy, and reuse?
Bottom Line
Ops is shifting from "keep it running" to "let the right thing appear, act, and disappear-safely and on budget." Separate capacity from consumption. Make inference the governed interface. Treat identity, context, and policy as first-class.
Do that, and swarms stop looking chaotic. They become manageable, auditable, and useful.
Further Learning
- AI courses by job for ops teams building agent-centric skills.
Your membership also unlocks:
