# Amazon SageMaker HyperPod launches global inference gateway for multi-region routing

> Amazon SageMaker HyperPod's new Global Inference Gateway provides a single endpoint that routes requests across registered clusters using real-time signals, automatically failing over when a cluster goes down.

- Type: AI news
- Page: https://completeaitraining.com/news/amazon-sagemaker-hyperpod-launches-global-inference-gateway/
- Published: 2026-11-21
- Updated: 2026-11-21
- Author: Jeroen Erne
- Jobs: Finance, IT and development, Operations, Product development, Science and research
- Reading time: 3 min

Amazon SageMaker HyperPod now includes a Global Inference Gateway, a managed routing service that gives you one endpoint across every registered cluster, in one AWS Region or across several. The gateway reads real-time signals-KV cache headroom, queue depth, Region proximity, and error rate-to send each request to the best-suited cluster. It also fails over automatically when a cluster goes down, without any client-side changes.

During a traffic peak, a request can sit in a queue on a saturated HyperPod cluster while GPUs on another cluster, sometimes in another Region, sit idle. You have already paid for that capacity, but your requests can't reach it. The Global Inference Gateway solves that. It is the second tier of the architecture AWS introduced with the HyperPod Inference Gateway, which handles pod-level routing inside a single cluster.

#### How teams get spread thin across clusters

Models are getting larger, and the GPU capacity to serve them is harder to find. To get enough of it, teams spread workloads across more clusters and more Regions. Consider a team that serves four models across five HyperPod clusters in two Regions. Without the Inference Gateway, clients track 20 endpoints: one per model per cluster. The Inference Gateway cuts that to five, one per cluster. But each client still picks a cluster itself.

That choice has costs. Load distributes unevenly during peaks, with requests queuing in one Region while GPUs in the other sit idle. Failover is manual-an engineer changes configuration to move traffic away from a failed cluster. And spreading traffic across Regions means building and maintaining load balancers, DNS, and cross-Region networking yourself.

#### One endpoint, automatic failover

The Global Inference Gateway routes across every cluster you register. The gateway picks the cluster and carries the request over the AWS backbone, with authentication through SigV4 and IAM. Inside each cluster, the Inference Gateway keeps picking the pod, unchanged.

Key benefits include putting idle GPUs to work during peaks by overflowing requests to clusters with available capacity, and moving traffic off a failed cluster automatically. Clients call a single SageMaker-managed endpoint instead of tracking one per cluster or model. You skip building custom networking, and every request stays private-it reaches your clusters only over AWS PrivateLink.

Jinhyung Park, VP of AI Platform at Bagelcode, said: "In our beta tests, the Global Inference Gateway in Amazon SageMaker HyperPod routed requests across two AWS Regions through a single endpoint and kept serving them when one cluster could no longer serve the model. That gave our small platform team cross-Region failover without having to build or maintain our own routing layer."

#### How routing decisions get made

For each request, the gateway scores every healthy cluster on four signals and sends the request to the highest-scoring cluster. A service-managed metrics publisher on each cluster supplies the load signals. You control how much each signal counts with routing weights, each a value from 0.0 to 1.0, which must add up to 1.0 if you set them.

The defaults favor clusters close to the gateway's Region: RegionProximity gets a weight of 0.40, QueueDepth gets 0.25, KVHeadroom gets 0.20, and ErrorRate gets 0.15. If resilience matters more than proximity for your workload, you can raise ErrorRate and lower RegionProximity. You pass the weights in the **Routing** parameter when you create the gateway, and you can change them later with an update call.

#### Setting up and operating the gateway

Setup takes two decisions: which clusters to open up for routing, and which to group behind one gateway. You publish a cluster once by turning on the **spec.globalRouting** block in its InferenceGatewayConfig. The HyperPod Inference controller then creates a Network Load Balancer, publishes a VPC endpoint service, and uploads the listener's certificate to your TLS bucket-all in your account. You create the gateway once by calling **CreateGlobalInferenceGateway** with the clusters to route across, and SageMaker returns one endpoint.

Adding a cluster later does not affect the endpoint or other clusters. Detaching a cluster stops new requests and lets in-flight requests finish. Deleting the gateway stops its endpoint, while per-cluster endpoints keep serving. When a cluster fails, traffic moves to the next-best cluster automatically; a failed request is retried once on another cluster unless it has started streaming. If no healthy cluster remains, the gateway returns a 503 error.

The Global Inference Gateway is available in 13 AWS Regions: US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon), South America (São Paulo), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Stockholm), Asia Pacific (Tokyo), Asia Pacific (Mumbai), Asia Pacific (Singapore), and Asia Pacific (Sydney).

#### Why this matters for IT, operations, and development teams

For teams running inference across multiple clusters, the Global Inference Gateway removes the need to build and maintain cross-Region load balancers, DNS configurations, and custom failover logic. That directly reduces operational overhead for platform and infrastructure engineers. It also lets you get more utilization from GPU capacity you have already provisioned, which matters when capacity is tight and expensive. If your team is building expertise in managing AI infrastructure at scale, [AI Systems Admin Courses](https://completeaitraining.com/learningpath/ai-for-systems-administrators/) can help round out the skills needed to operate these environments efficiently.

---

This is the Markdown form of https://completeaitraining.com/news/amazon-sagemaker-hyperpod-launches-global-inference-gateway/. The whole site is mapped at https://completeaitraining.com/llms.txt.
