Skip to content

Evaluation (eval)

Short answer

An evaluation is a fixed set of questions with known answers, run against each model so that results can be compared rather than asserted.

Nightjar's is 16 questions covering reading comprehension over a page, table lookup, and whether a model will refuse to answer when the source does not contain the answer. Qwen 3.5 2B and Gemma 4 12B both scored 16; Llama 3.2 1B scored 10.

Sixteen questions is a small set and we say so. It is enough to catch a model that cannot read a table and not enough to rank two models that both can.

Related terms

Other ways people ask this

  • what is benchmark eval

Related