Skip to content

KV cache

Short answer

The KV cache stores the intermediate attention values for tokens the model has already processed, so it does not have to recompute the whole conversation for every new word.

Without it, generating the hundredth token would mean redoing the work for the first ninety-nine. With it, each new token is cheap. This is why generation speed is roughly constant rather than slowing to a crawl mid-answer.

The cache lives in the same memory as the weights and grows with the conversation, which is where the extra gigabyte over the model's file size goes. A long conversation on a small card can run out of room even though the model itself loaded fine.

Related terms

Other ways people ask this

  • what is kv cache

Related