AI news ·
SageMaker inference adds Nvidia multi-instance GPU support to pack more models onto each endpoint
Amazon SageMaker Inference now supports NVIDIA MIG GPU partitioning, letting you carve a single GPU into up to seven isolated instances. Testing showed a 3/7 slice of an H100 delivered up to 89% more output tokens per second and cost 18-34% less per million tokens than a whole A100.
Amazon SageMaker Inference now supports GPU partitioning through NVIDIA Multi-Instance GPU (MIG), letting you divide a single physical GPU into independent, hardware-isolated partitions. This changes two constraints at once: you can host more models than you have accelerators, and you can choose a fraction of a newer GPU instead of paying for a whole one your model won't fully use.
Why the old options fall short
Before this, teams with more models than GPUs had three paths, each with a trade-off. Running fewer, larger endpoints and batching harder works only when models are interchangeable. Multi-model endpoints share a GPU without isolation, so a heavy request for one model affects the others and unloaded models pay a cold-start penalty. The third path, self-managed Kubernetes, gives you MIG but demands you handle driver versioning, partition configuration, a device plugin, scheduling, and consistency across a fleet.
MIG solves these by carving a GPU into as many as seven independent instances, each with dedicated compute, cache, memory, and memory bandwidth. Because the split is enforced in hardware, you get predictable latency, fault isolation, and tenant isolation. A crash stays inside its partition. A traffic spike on one model doesn't show up in another's p99. And every model stays loaded, with no eviction and no load penalty.
How it works on SageMaker
GPU partitioning adds two optional fields to API calls you already make. AcceleratorPartitionLayouts goes on a production variant in CreateEndpointConfig and declares how each accelerator is divided. AcceleratorPartitionType goes on an inference component and selects which partition the model runs on. There are no new API operations, no new resource types, and an endpoint that omits both fields behaves exactly as it does today.
A layout entry describes how one accelerator is split, and every accelerator on every instance in the variant is divided the same way. On an A100, which has seven compute slices to divide, a layout like mig-4g.20gb, mig-2g.10gb, mig-1g.5gb uses all seven slices. Each of the eight A100s on a p4d instance then carries three partitions, giving you 24 total. The carve applies uniformly and isn't positional, so an entry doesn't map to a specific GPU.
Invocation doesn't change. You address a model by inference component name, and the partition it occupies never appears in the request. Partitioning happens while the endpoint is provisioned, and in testing, carving accelerators added roughly 17 seconds to provisioning time.
Partition profiles by GPU
The profiles available are a property of the GPU. An A100 and an H100 divide into sevenths on compute and eighths on memory, while G7 divides into halves and G7e into quarters. Profile names differ between GPUs even where the slice counts match, so don't carry a layout from one instance family to another without checking.
Read a profile name as compute units followed by GPU memory: mig-3g.40gb takes three compute units and 40 GB. The 3g profile is worth knowing because its two dimensions don't match - it takes three of seven compute units but four of eight memory units. Two of them fill the GPU and nothing else fits beside them.
Within these limits you have real choice. On a P4d or P5 you can carve seven of the smallest profile, two 3g partitions, or a mixed layout. On a G7e you can take four of the smallest, two 1g beside a 2g, or two 2g. On a G7, either two partitions or one. The high-level rule: every profile must be one your GPU offers, and the compute and memory units your profiles add up to must not exceed what one device provides.
Choosing a layout for your workload
Compute units govern prefill - the compute-heavy phase that reads the prompt and builds its representation. Memory units govern decode and capacity, because generating tokens re-reads model weights on every step and memory caps how much KV cache you can hold. Almost no real workload exercises only one of these.
Memory is the easier of the two to underestimate. A partition holds the model weights, the KV cache sized for your context length and concurrency, and 1-2 GB of runtime overhead. Add those up before choosing a profile. For a concrete reference point, an 8B model at FP16 comfortably serves an 8,000-token context on a 40 GB partition.
Salesforce has been involved from the start, helping define requirements and validating the implementation. Richard Chang, Software Engineering Architect at Salesforce, said: "NVIDIA MIG partitions a single GPU into multiple isolated instances, each with dedicated compute and memory resources. Salesforce sees this as an important capability because many inference workloads do not require an entire GPU. MIG would allow us to consolidate workloads safely, significantly improve GPU utilization, reduce inference costs, and maintain predictable performance and strong isolation."
Performance: a slice of a newer GPU against a whole older one
To see whether a partition of a newer GPU pays off, testing compared one mig-3g.40gb partition of an H100 against one whole, uncarved A100-40GB. Both ran Llama-3.1-8B-Instruct on production SageMaker real-time endpoints, with identical containers, tokenizers, and client configurations. Five traffic shapes were tested, from short prompts to long-context summarization, across a concurrency ladder from 2 to 512.
Four results stood out. The partition was faster everywhere, delivering 53 to 89 percent more output tokens per second at peak load and lower inter-token latency in every case. Time to first token was lower by 5.6 to 47.5 percent. On cost, the partition was 18 to 34 percent cheaper per million output tokens above a crossover point that moved earlier as prompts lengthened. And at the top of the concurrency ladder on long-context traffic, the A100 failed a third to half of its requests against a serving-path deadline, while the partition failed none.
Treat this as a case-by-case comparison, not a verdict. The useful mental model is to regard available GPU fractions as GPU types in their own right, list them alongside whole accelerators, and compare them on your own workload. For those working in AI for Developers Courses, this granularity is worth factoring into capacity planning.
What to know before you commit
A component's partition type is fixed for its life. Moving a model to a different profile means creating a new component. Changing the endpoint's carve means a new endpoint and a traffic migration, because updating an endpoint configuration doesn't re-carve hosts that are already running. This makes partitioning a good fit for a stable portfolio and a poor one for a mix you reshuffle daily.
Partitioning applies to real-time inference on inference component endpoints. Serverless endpoints and heterogeneous instance pools aren't supported, and asynchronous invocation cannot address an inference component. You're billed for the full GPU instance, as before - partitioning raises how much of that instance you can use, not its hourly price. And a model that genuinely needs a full accelerator, or more than one, should keep having one. No partition spans accelerators.
GPU partitioning is available on P4d, P5, G7, and G7e instance families in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland).
Why this matters for IT and operations teams
If you manage inference infrastructure, this removes the platform work that previously came with GPU partitioning. You pick a partition profile, and SageMaker handles GPU configuration, placement, and scaling. This is especially relevant for teams following an AI Systems Admin Courses learning path, where understanding managed partitioning changes how you size endpoints. The direct operational takeaway: every supported accelerator is now several sizes rather than one. Whether a partition of a newer GPU or a whole older one is the right unit for a given model is no longer a question you answer by rounding up - it's one you measure against your actual memory and compute demands.