Nightjar and llama.cpp
Short answer
Nightjar's assistant runs on llama.cpp, which is why GGUF files, quantisation labels and GPU layer offloading behave exactly as they do in every other tool built on it.
What this inherits
GGUF as the model format, quantised execution on CPU and GPU, partial offload when a model does not fit in VRAM, and the context sizing behaviour that comes with it.
If you know llama.cpp, you already know how Nightjar will behave when a model is slightly too big for your card.
What Nightjar adds
The sizing decision, so you are not choosing a quantisation level by hand; retrieval over the page or document; and citations back into the source.
The inference itself is llama.cpp's, which is also why the Vulkan and CUDA trade-offs described on the benchmarks pages apply here.
Models you built yourself
Anything you quantised or converted yourself loads from disk like any other GGUF. A file built for a much newer llama.cpp than the bundled runtime may not load.
Other ways people ask this
- llama.cpp browser
- use llama.cpp models in browser
- I already have models downloaded in llama.cpp. Can a browser use them directly?
Related
- Up a levelNightjar and the local AI tools you already use
- Closely relatedUsing Hugging Face models with Nightjar
- Closely relatedNightjar and KeePassXC
- Closely relatedUsing your LM Studio models in Nightjar
- Closely relatedUsing your Ollama models in Nightjar
- Elsewhere on the siteWhat is a local AI browser?