Is your agentic AI strategy increasing your cost?
Leadership commitments are necessary, but they are not enough. If you scale agentic AI without a grip on the economics, your token bill will grow faster than value, and the business case will stall.
The costs of AI are more complex than expected. Compute may be cheaper, but consumption is climbing. Tokens are becoming a loud, new line in operating expenditure. Treat them as such.
Every token has a cost
Tokens are the units models use to process and generate text. Every prompt, response, tool call, reasoning step, and connected task consumes them. The more agentic behavior you allow, the more tokens you burn.
Think of tokens like kilowatt hours. You pay for what you use. The job is to know how many you use, what they're worth, and which levers change the equation.
Why costs rise as you scale
Traditional TCO captured infrastructure, software, and implementation. Agentic AI introduces ongoing per-use spend. As adoption grows, usage grows daily - not just at go-live.
Background agents, retries, overlong context windows, and generous guardrails all compound consumption. Without visibility, these invisible flows become margin leaks.
The control points: where spend explodes or shrinks
- Model mix: Reserve premium models for high-value steps. Route routine steps to small or distilled models.
- Context discipline: Cap context length. Use retrieval to load only what's needed. Chunk, dedupe, and set strict top-k limits.
- Prompt hygiene: Shorten prompts. Remove verbose system text. Prefer function calls and structured outputs over free-form prose.
- Reasoning depth: Control chain length, max tokens, and tool-use loops. Gate "think" modes to cases that prove ROI.
- Caching and reuse: Cache frequent prompts/responses. Precompute embeddings for hot data. Share caches across teams.
- Quality gates: Run evals early to block low-probability runs. Stop bad chains before they spiral.
- Agent constraints: Enforce budgets, timeouts, and step limits per run. Log and review overages weekly.
- Use-case selection: Prioritize high-frequency, high-margin tasks. Kill low-value curiosities fast.
- Vendor terms: Negotiate volume tiers, reserved commits, and transparent metering. Push for audit-level token logs.
- Infrastructure choices: GPU-as-a-service gives speed; owning capacity can pay off at scale. Align this with projected token demand.
Build vs. buy: model the trade-offs
Embedding AI inside ERP/CRM/ITSM can speed adoption but may lock token usage behind vendor meters you don't control. Building outside those platforms increases control and may be cheaper over time, with higher upfront cost.
- Buy inside platforms: Fast start, less engineering; limited control of prompts, models, and caching; exposure to opaque pricing.
- Build in your stack: Full control of model mix and budgets; better unit economics at scale; higher initial investment and ops maturity needed.
The token P&L: metrics for your dashboard
- Tokens per task, per user, and per successful outcome
- Cost per task vs. value per task (dollar value or time saved)
- Model mix (% tokens by model tier) and effective blended rate
- Context efficiency: average context size, retrieval hit rate, cache hit rate
- Chain efficiency: average steps per run, abort rates, retry rates
- Quality and waste: eval pass rate, rework rate, hallucination incidents
- Latency vs. spend trade-offs by use case
- Budget burn: daily token budget, alerts, and circuit breakers
A simple 90-day plan
- Days 0-30: Instrument everything. Baseline token usage, costs, and outcomes. Set per-use-case budgets and quotas. Define ROI hypotheses with finance.
- Days 31-60: Optimize. Shorten prompts, cap context, enable caching, route to smaller models, add agent step limits. Remove or refactor the top 10% costliest chains. Start vendor renegotiations.
- Days 61-90: Govern at scale. Stand up AI FinOps, chargeback/showback, and approval gates. Publish a model and prompt "golden path." Lock in quarterly token targets tied to business outcomes.
Decision questions for executives
- What is our blended cost per 1,000 tokens this quarter, and what will it be in 12 months?
- Which 3 use cases generate the most value per token? Which 3 destroy it?
- Where do tokens flow we do not control today (inside SaaS, shadow projects, pilots)?
- What are our default models and fallbacks, and who can approve exceptions?
- What are our agent step limits and per-run budgets?
- Do we cache effectively? What is our cache hit rate target?
- What is our plan for model portability and vendor diversity?
- How do token budgets roll up to P&L outcomes by function?
Guardrails for sustainable scale
- Set monthly token budgets by product and by environment (dev, test, prod).
- Enable circuit breakers that halt runs on cost spikes or error bursts.
- Adopt a "cost-aware" prompt and agent template library.
- Require a unit-economics check at each phase gate of deployment.
- Publish a weekly cost and value report to product owners and finance.
Keep cost and value connected
Agentic AI will create an abundant digital workforce. It will also generate an abundant bill unless you control the levers. Tie consumption to outcomes, enforce budgets, and keep revisiting the build vs. buy mix as you scale.
This is part two of a three-part series on scaling agentic AI with clear value and controlled spend.
Learn more
- OpenAI pricing for current token rates and model tiers.
- Anthropic pricing for alternative cost structures and context sizes.
- Complete AI Training: courses by job to upskill leaders and teams on AI strategy and economics.
