Local model performance on your hardware

generation:

10.0 tokens/s

quant:

Q5 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at around 10 t/s on an RTX 4070 Ti Super in a Threadripper 3975WX system with 256 GB RAM. Setup is llama.cpp server with the Unsloth Q5 GGUF, run with only --ctx-size and --jinja flags. User asks whether 10 t/s is the best achievable at this quant without a better GPU.

generation:

54.1 tokens/s

prompt processing (prefill):

1522.4 tokens/s

quant:

IQ2_XXS (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a full 256k context offloaded to VRAM with no CPU spillover, using n-gram speculative decoding. Generation speed degrades with context depth: 43.69 t/s at 32k, 36.59 t/s at 64k, 27.55 t/s at 128k, 22.15 t/s at 192k, and 18.48 t/s at 256k. Prompt processing also degrades similarly.

generation:

5.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a 4070 Super run that was slow because only 38 of 66 layers were on GPU, with the rest running from system RAM. The same model achieved 81.5 tok/s on a 5090. The user introduces a tool called Picchio for diagnosing layer placement and speeds.

generation:

7.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B Q4_K_M running on an RTX 4070 Ti Super 12GB and a MacBook M5 Air over RPC with llama.cpp, completing a coding game clone task in 117 minutes. Setup uses llama.cpp over RPC between the two machines. LM Studio is also mentioned. The 117 minutes is the total time for the task, not a throughput figure.