Inference
Short answer
Inference is running a trained model to get an answer, as opposed to training, which is building the model in the first place.
Training a model of any size needs a cluster and weeks. Inference needs a laptop and a second. This asymmetry is the whole reason local AI is possible at all: you are not building anything, you are reading a file that somebody else spent millions building.
In practice inference has two phases with very different speeds — reading your prompt, which is fast and parallel, and producing the answer one token at a time, which is not.
Related terms
Other ways people ask this
- what is inference