Llama 3.1 8B
on NVIDIA GTX 1080 Ti · text-generation-webui
Use cases
rp
Summary
User recounts their AI tooling history, starting with Ollama, LM Studio, Msty and text-generation-webui on a mini PC with an i5 6400T and 24GB DDR4 using integrated graphics, where Llama 3.1 8B generated about 1 tok/s.
They later moved to a Xeon E3-1270 V2 with 32GB DDR3 and a GTX 1080 Ti, then SillyTavern with KoboldCpp, Llamafile and finally llama.cpp.
The 1 tok/s figure is from the earlier integrated-graphics mini PC, not the 1080 Ti; the user says they enjoyed it despite the limits and that roleplay started there.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.
Llama 3.1 8B at 4-bit by hardware
- 1.5 t/s · one run
Bars are medians on one scale, 0 to 1.5 tokens/s. Press ▶ to watch text arrive at a row's speed.
Models most run on the NVIDIA GTX 1080 Ti
- Llama 2 7B4-bit62.0 t/s · one run
Bars are medians on one scale, 0 to 62.0 tokens/s. Press ▶ to watch text arrive at a row's speed.
Everything reported on the NVIDIA GTX 1080 Ti →Every Llama 3.1 report →Estimate Llama 3.1 on this card →