Skip to content

How much VRAM you need to run a local model

Short answer

Model file size plus about 1.1 GB for context and runtime. A 5.3 GB model wants roughly 6.4 GB, so 8 GB of VRAM is comfortable and 6 GB is not.

The rule

Take the GGUF file size and add about 1.1 GB. That covers the KV cache for the context window and the runtime's own overhead.

Compare the result with your card's VRAM. If it fits with 2 GB to spare you are comfortable; if it fits by a little, expect it to fall back under pressure; if it does not fit, some layers run on the CPU and the speed drops sharply.

What each tier gets you

  • No card: the small models only. Qwen 3.5 2B managed 15 tokens per second on CPU alone.
  • 8 GB: everything up to about a 5 GB file comfortably.
  • 12 GB: every everyday model, up to Gemma 4 12B at 6.6 GB. This is our test card.
  • 16 GB: headroom for longer context windows.
  • 24 GB and up: the large models, from Gemma 4 26B-A4B and Qwen 3.5 27B at 24 GB to Gemma 4 31B at 40 GB.

Why the cache matters more than people expect

The KV cache grows with the conversation. A model that loaded fine can run out of room thirty messages later, which looks like a random failure and is not. Starting a fresh conversation for a new document fixes it.

Other ways people ask this

  • how much vram do i need for a 7b model
  • Is an 8 GB graphics card enough for local AI in 2026?

Related