Foundry Local brings agentic coding to developer hardware with Qwen3.8-27B

Microsoft has released Foundry Local, a tool that runs the Qwen3.8-27B coding model directly on developer hardware. The move lets developers use an AI coding assistant through GitHub Copilot without sending code or context to the cloud, cutting recurring API costs and removing the requirement for a constant internet connection.

How local inference changes the developer workflow

Cloud-based coding assistants transmit your codebase and prompts to remote servers for processing. Foundry Local keeps everything on the machine in front of you. The model, your context, and the generated output never leave the device. For teams handling proprietary code, internal APIs, or regulated data, that architectural difference removes a compliance headache. It also means predictable costs: the inference runs on hardware you already own, so there is no per-token billing to track.

The tool is built around Qwen3.8-27B, a model that balances strong coding capability with a size practical for consumer GPUs. Published benchmarks show it performs well on agentic terminal tasks, software engineering problems, and repository-level code generation. With 4-bit quantization, the model shrinks to roughly 18 GB, fitting on GPUs with 24 GB of VRAM and supporting context windows up to 64,000 tokens. GPUs with 32 GB or more can push that to the model's maximum of 262,000 tokens.

Optimizations that keep local inference responsive

Running a 27-billion-parameter model locally creates real engineering challenges around speed and memory. Foundry Local uses ONNX Runtime with a stack of hardware-aware techniques tuned for Nvidia GPUs. DFlash2 speculative decoding uses a lightweight drafter model to predict multiple tokens at once, breaking through the memory-bandwidth ceiling on decode throughput. CUDA graph fusions and kernel optimizations target the latest Nvidia SM121 GPU architecture to reduce compute bottlenecks.

Memory management combines INT4 weight handling, KV-cache quantization, embedding memory sharing, and CPU offloading. The serving layer adds prefix caching, which reuses previously computed context so repeated prompts do not start from scratch, and continuous batching to handle concurrent requests. These matter most in repository-scale work, where the assistant processes large codebases across multiple simultaneous tasks.

In testing against llama.cpp using the 4-bit Qwen3.8-27B model on an Nvidia GeForce RTX 5090 and an Nvidia DGX Spark, Foundry Local delivered higher generation speed at every prompt length. The performance gap widened beyond 131,000 tokens, exactly where long-context coding workflows need sustained throughput.

Setting up Foundry Local with GitHub Copilot

The setup requires an Nvidia GPU with at least 24 GB of VRAM and Foundry Local CLI version 0.13.0 or later. Users install the CLI through WinGet, load the Qwen3.8-27B model, and point GitHub Copilot's Bring Your Own Key feature at the local API endpoint. The model download triggers automatically on first load. Nvidia CUDA and cuDNN runtimes must be installed separately, with library paths configured correctly.

Once connected, developers select Foundry Local as the model provider in GitHub Copilot's model picker and choose Qwen3.8-27B. The experience is designed for everyday coding and targeted subtasks. Microsoft is clear about the trade-off: local models have more limited reasoning capabilities than advanced cloud models. For complex, high-reasoning tasks, cloud models remain the stronger choice. The vision is a hybrid workflow where developers route work to the right model for the job.

What comes next

Microsoft plans to expand Qwen3.8-27B support to Macs and AMD GPUs, grow its lineup of coding-optimized local models, and deepen the integration between Foundry Local and GitHub Copilot. The current release focuses on Nvidia hardware running Windows, with the optimization recipe published for teams that want to tune the model further.

Why this matters for IT and development teams

For developers and IT leads managing code security, infrastructure costs, or remote work policies, local inference removes three persistent friction points at once. Sensitive code stays on-device, there is no per-request billing to track against a cloud budget, and work continues when the network drops. The 24 GB VRAM requirement puts this within reach of many workstation-class GPUs. The main constraint is reasoning depth on complex problems, which means teams will likely adopt a split workflow: local models for routine coding tasks, cloud models for architecture decisions and hard debugging. Professionals looking to build skills with these tools can explore AI Coding Courses that cover both local and cloud-assisted development patterns.