Llama 3.1 8B
NVIDIA GTX 1080 Ti · text-generation-webui
- generation:
- 1.0 tokens/s
User recounts their AI tooling history, starting with Ollama, LM Studio, Msty and text-generation-webui on a mini PC with an i5 6400T and 24GB DDR4 using integrated graphics, where Llama 3.1 8B generated about 1 tok/s. They later moved to a Xeon E3-1270 V2 with 32GB DDR3 and a GTX 1080 Ti, then SillyTavern with KoboldCpp, Llamafile and finally llama.cpp. The 1 tok/s figure is from the earlier integrated-graphics mini PC, not the 1080 Ti; the user says they enjoyed it despite the limits and that roleplay started there.