GPU layers
Short answer
GPU layers is the number of a model's transformer layers held in video memory. The rest run on the CPU, which is slower.
A model is a stack of identical layers, and they do not all have to live in the same place. If 28 of 32 layers fit in VRAM, the other 4 run on the CPU and the data crosses the bus twice per token.
Partial offload is much better than not running at all and much worse than fitting entirely. It is why a model that is 0.5 GB too big for a card can be dramatically slower than one that is 0.5 GB under.
Related terms
Other ways people ask this
- what is gpu layers