How Much GPU Do You Really Need for Running Local AI With Ollama?

What GPU Do You Need to Run Local AI with Ollama? It’s More Affordable Than You Think

Running AI locally means you need some serious hardware, especially a dedicated GPU if you’re using Ollama. But don’t worry — you don’t have to break the bank to get a card that delivers solid performance. While the latest RTX 5090 might be tempting, it’s neither necessary nor practical for most users.

Whether you’re a developer, hobbyist, or simply curious about AI, local AI setups like Ollama offer a great way to explore large language models (LLMs) on your own machine. Unlike cloud-based tools like ChatGPT, local AI demands more from your hardware, particularly your GPU. Ollama currently supports only dedicated NVIDIA GPUs, though other tools like LM Studio can work with integrated GPUs but still need decent specs to run smoothly.

VRAM Is the Real MVP for Ollama

When choosing a GPU for AI, the key spec isn’t the number of CUDA cores or the latest architecture—it’s the amount of VRAM. For gaming, you want the fastest chip for high frame rates and slick graphics, but for AI, memory capacity matters most.

If your model doesn’t fit entirely in the GPU’s VRAM, your CPU and system RAM have to pitch in—and that kills performance. VRAM is much faster than system memory, so keeping the whole model loaded on the GPU is critical.

For example, running the Deepseek-r1:14b model on an RTX 5080 with 16GB VRAM yields about 70 tokens per second with a 16k context window. Push beyond that, and the model spills into system RAM, dropping performance to 19 tokens per second—even though the GPU still handles most of the work. This slowdown also applies to laptops. If local AI matters to you, invest in a GPU with as much VRAM as possible.

How Much VRAM Do You Actually Need?

The simple answer: as much as you can afford. But it really depends on the models you plan to run. Ollama’s models page lists each model’s size, which gives a baseline for VRAM requirements.

Keep in mind, you’ll need more VRAM than just the model size. The context window—the amount of information fed into the model—also consumes memory (known as the KV cache). A good rule of thumb is to multiply the model size by 1.2 to estimate VRAM needs.

Take OpenAI’s gpt-oss:20b model, which is 14GB on disk. Multiplying by 1.2 suggests you need roughly 16.8GB of VRAM. From experience, running this on a 16GB RTX 5080 with context windows above 8k causes performance drops as the system RAM and CPU get involved.

For flexibility and comfort, a 24GB VRAM GPU is ideal. Larger models or bigger context windows will force your system to use slower memory, harming speed and responsiveness.

Older and Cheaper GPUs Can Still Deliver

If you’re on a budget, don’t overlook older cards. The RTX 3090, for example, remains a favorite for AI enthusiasts. It offers 24GB VRAM, comparable CUDA cores to the RTX 5080, and better memory bandwidth—all at a lower price than the newest GPUs. Plus, its 350W power draw is more manageable compared to some of the latest, power-hungry cards.

The RTX 3060 is another popular choice due to its 12GB VRAM and relatively low 170W power draw. Running two RTX 3060s in tandem can match the VRAM and power consumption of a single RTX 3090, often at a fraction of the cost.

The takeaway: you don’t need to buy the most expensive GPU. Balance your budget with VRAM capacity and speed. Ollama supports multiple GPUs, so combining cards can be a smart move depending on your setup and workload.

Stick with NVIDIA for Now

While Ollama does support some AMD GPUs, NVIDIA remains the best choice for local AI today. NVIDIA’s CUDA ecosystem is more mature and widely supported. AMD is making strides, especially with mobile chips and integrated GPUs, but for dedicated desktop AI tasks, NVIDIA leads.

For those interested in alternatives, AMD-powered mini PCs with large unified memory pools offer interesting options for AI workloads—especially with software like LM Studio that supports Vulkan. However, Ollama currently doesn’t support AMD integrated GPUs.

Final Thoughts

• Focus on VRAM over raw GPU power for local AI with Ollama.
• Check model sizes and context window needs before buying hardware.
• Older GPUs like the RTX 3090 or dual RTX 3060 setups offer great value.
• Stick with NVIDIA GPUs for the broadest support and best performance.

Local AI with Ollama doesn’t require an ultra-expensive GPU. Do your research, match your hardware to your models, and you'll find a setup that fits your budget and delivers solid performance.

For those interested in expanding AI skills and exploring more about AI hardware and software, check out Complete AI Training for practical courses and resources.