Skip to content

VRAM

Short answer

VRAM is the memory physically on your graphics card. A model has to fit in it to run on the GPU at full speed.

This is the number that decides which models you can run. The rule of thumb: the model file size plus about 1.1 GB for the context window and runtime overhead has to be less than your VRAM. A 5.3 GB model wants roughly 6.4 GB, so an 8 GB card manages it and a 6 GB card does not.

When a model does not fit, llama.cpp will put some of its layers in system RAM and run those on the CPU. It works, and it is much slower — the GPU spends its time waiting for the slowest layer.

Related terms

Other ways people ask this

  • what is vram

Related