Complete AI Training
Start for Free

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

AI news ·

Distributed KV caching with Amazon ElastiCache cuts self-hosted LLM inference costs by up to 40% and time to first token by up to 80%

Distributed KV caching with Amazon ElastiCache can cut inference costs up to 40% and time to first token up to 80% for repeated conversation contexts. A new Agent Toolkit skill lets coding agents diagnose, size, and deploy the caching layer in a few prompts.

Self-hosted large language models often waste GPU time recomputing attention states for conversation histories that repeat across sessions. Distributed KV caching with Amazon ElastiCache can cut inference costs by up to 40% and time to first token by up to 80%, according to AWS benchmarks. A new skill in the Agent Toolkit for AWS now lets coding agents diagnose, size, and deploy this caching layer in a few prompts.

The fix targets a common bottleneck. When a lawyer returns to a legal AI assistant after lunch, the GPU server handling the request holds none of the prior conversation's key-value attention states. It recomputes the full context - sometimes hundreds of thousands of tokens - before generating a single word. The same prefill penalty hits resumed coding agents and document Q&A assistants. A shared KV cache, sitting beside the GPU fleet on ElastiCache for Valkey, lets any server reuse those states for days across sessions and routing shifts.

What the distributed KV caching skill does

The skill, available in the Agent Toolkit for AWS, works with Kiro, Claude Code, OpenAI Codex, and other coding agents that support Agent Skills. It walks an agent through four stages: checking whether a workload benefits, sizing the cache with real traffic data, setting up the cluster, and troubleshooting failures.

Before spending money, the agent performs a fit check. It reads the GPU fleet with a read-only describe-instances call, counts Elastic Fabric Adapter interfaces, and asks only for what it cannot see - engine, model, tensor-parallel size, KV data type, server price, and prompt repeat frequency. If something does not fit, the agent stops and explains why.

For sizing, the skill uses KVTrace, a tool that records KV cache events from vLLM or SGLang engines and replays them offline against candidate cache sizes. The recorder never writes token IDs or prompt text. Traces contain block hashes and token counts. The agent estimates savings from the user's own traffic and stops if the numbers do not cover the cache's cost. It also flags a subtle reality: freed GPU time turns into dollars only if you run fewer servers or serve more traffic on the same ones.

Setup and configuration guardrails

The agent adds the LMCache connector to existing vLLM or SGLang engines, including vLLM workers under serving frameworks like llm-d and Ray Serve. It keeps the user's routing and engine settings intact and forces every server in the fleet to use one KV ID mechanism - matching hash and seed plus an explicit KV data type. Mismatched settings cause each server to write KV state the others cannot find.

Before production traffic hits the cache, the agent empties local tiers and checks that two cache-loaded runs produce the same output. A layout mismatch can show healthy hit counters while returning wrong answers. The skill also requires explicit approval for high-impact operations such as creating, changing, or deleting resources.

In one documented flow, a user told the agent: "My vLLM fleet serves a legal assistant with 64K-token contexts on p5 instances. Returning conversations take 13 seconds for the first token because the GPU recomputes the full conversation history every turn." The agent identified cold prefill as the cause and checked for no-infrastructure fixes first - turning prefix caching back on if it was disabled, for instance. It then identified constraints specific to legal workloads. A shared KV cache can let one tenant infer from response times whether another sent the same text. Cases requiring isolation get separate caches; tenants sharing a cache add their own ID to the KV hash.

For an example fleet of eight p5 instances serving 64,000-token prompts for a multi-turn RAG assistant, the agent looked up GPU and cache node prices, derived prefill rates and KV sizes from benchmarks, and recommended two cache.i8ge.48xlarge nodes. Because the user did not provide a cache hit rate, the agent presented scenarios and recommended a KVTrace run over a representative window - such as a full business day - to identify the real figure.

Troubleshooting RDMA transfer failures

The skill also handles post-deployment problems. When a user reported that BLOB.RDMA_HELLO succeeded but actual data transfers failed with climbing rdma_write_errors, the agent traced the symptom to a security group blocking EFA traffic between its own members and provided commands to check and fix the rule.

The prerequisites include a coding agent that supports Agent Skills, GPU instances with EFA RDMA write support such as p5, LMCache 0.5.8rc1 or later with vLLM 0.11.1 or later (or SGLang 0.5.13 or later), and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes.

Why this matters for IT and operations teams

Distributed KV caching changes the economics of self-hosted inference for workloads with long, repeating contexts. The skill removes the guesswork from sizing and setup by pulling numbers from your own traffic with KVTrace before any infrastructure changes. For operations teams managing GPU fleets, the agent enforces matching KV ID settings across servers - a configuration detail that, if missed, silently breaks cache sharing while metrics show healthy hit rates. The skill's approval gates on resource creation and its pre-production validation checks are designed to prevent costly misconfigurations before they reach users. Teams evaluating this approach can start with the distributed KV caching skill on GitHub or consult the KVTrace repository for offline savings estimates.

Share