Local model performance on your hardware

generation:

65.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 65.07 tok/s on a single Tesla V100-SXM2-32GB at 1K context. Setup is llama.cpp (pinned tag b10793) with Q4_K_M GGUF and MTP speculative decoding enabled, 128K context window, single stream. Decode falls to 62.41 tok/s at 8K, 61.24 at 32K, 52.51 at 64K and 42.22 at 128K. With MTP off on a T2-001 packed-attention build the same card gives 35.22 tok/s at 1K and 23.83 at 128K; stock llama.cpp gives 35.95 at 1K and 16.49 at 128K. Prompt caching cuts multi-turn wall time from 22.94 s to 3.08 s per turn.

generation:

57.3 tokens/s

prompt processing (prefill):

720.0 tokens/s

quant:

UD-Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B UD-Q4_K_M at 57.25 t/s decode (depth 0) on a single Tesla V100-PCIE-32GB. Setup is llama.cpp with llama-split-bench --profile, MTP enabled, measured up to 262k context. Prefill is about 720 t/s on new prompts. Decode falls to 20.49 t/s at depth ~260k. An initial run suffered thermal throttling (15.68 t/s at 64k depth) due to BMC fan control; values shown are with fan mode set to Full.

generation:

50.0 tokens/s

quant:

UD-Q4-K-M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at about 50 t/s decode on a V100 32GB. Setup is llama.cpp with UD-Q4-K-M (MTP) and batch size 4096, ubatch 2048, on a PG500-216 card at 185W with a custom compile. User notes prefill drops off quickly and asks for pointers on improving it.

generation:

10.0 tokens/s

prompt processing (prefill):

52.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next Q8_0 at ~52 tokens per second prompt processing and ~10 tokens per second generation on a V100 32GB. Setup is llama.cpp with a 262144-token context, --cpu-moe and -ngl 64, on a machine with 128GB system RAM; the GPU drew 95.85 W at 48% utilization with 14206 MiB of its 32768 MiB in use. Both throughput figures are given as approximate.

generation:

47.2 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 47.2 t/s decode on one Tesla V100-SXM2-32GB (250 W) in an X99 box with 2x Xeon E5-2690 v4 and 251 GB RAM. Setup is llama.cpp (V100 tensor-parallel fork) with IQ3_S GGUF weights, q8_0 KV cache, 128k context, and MTP self-speculation with up to 3 draft tokens per step. A 64k context run with f16 KV reached 52.5 t/s with 91.9% draft acceptance; the 128k run had 88.0% acceptance. Per 8k-token stretches the rate ranged 46-66 t/s at 64k and 41-57 t/s at 128k.

generation:

43.3 tokens/s

prompt processing (prefill):

833.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-Uncensored at 833.37 prompt tok/s and 43.34 decode tok/s on two NVIDIA Tesla V100 GPUs. Setup is llama.cpp with Q8_0 weights, F16 KV cache, 262,144-token context, tensor split 1,1, and the built-in MTP head with draft length 3. The matching DFlash2 Q8 candidate measured 607.82 prompt tok/s and 33.00 decode tok/s, so MTP remains the production default. A single-V100 Q4_K_M profile with Q4_0 KV and MTP draft length 1 reached 44.201 tok/s on a 512-token short test and 13.001 tok/s on a 255,000-token capacity run.

generation:

32.2 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 27B at 32.17 t/s generation on a single NVIDIA Tesla V100 SXM2 32GB. Setup is llama.cpp (build 6eab47181) with Q4_K_M GGUF weights, run via llama-bench with -p 0 and -n 128,256,512,1024,2048, steady across those generation lengths. Other quants measured on the same card: Q5_K_M 29.14 t/s, Q6_K 25.11 t/s, Q8_0 22.80 t/s. BF16 required an added RTX 3090 24GB and ran 15.27 t/s.

generation:

65.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 65.07 t/s decode at 1K context on a Tesla V100-SXM2-32GB. Setup is llama.cpp pinned at tag b10793 with the repo's patches, Q4_K_M GGUF, 1 parallel slot, multi-token prediction speculative decoding enabled. Decode falls to 42.22 t/s at 128K context; stock llama.cpp gives 35.95 t/s at 1K and 16.49 t/s at 128K, and the patched build without MTP gives 35.22 t/s at 1K and 23.83 t/s at 128K. Multi-turn agentic sessions with prompt caching take 3.08 s per turn versus 22.94 s without.

generation:

33.4 tokens/s

prompt processing (prefill):

826.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 33.41 tok/s generation and 826.03 tok/s prefill on a single Tesla V100 32GB. Setup is llama.cpp with UD-Q4_K_XL GGUF at 131,072 context; generation falls to 27.95 tok/s at 32k depth and 15.84 tok/s at 131k depth. Peak VRAM was 25,264 MiB of 32,768 MiB. KV cache quantization (-ctk q8_0 -ctv q8_0) silently fell back to CPU with 0% GPU utilization; CUDA 13 dropped Volta support.

generation:

27.7 tokens/s

prompt processing (prefill):

433.2 tokens/s

quant:

UD-Q5_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Flash-Next at 433.19 t/s prompt processing and 27.72 t/s generation on a Tesla V100 32GB, comparing a custom llama.cpp fork against vanilla llama.cpp. Setup is llama.cpp (v100-optimized fork) with UD-Q5_K_XL GGUF, q8_0 K/V cache, 262144 context, FlashAttention on, MTP n-max=3, 63 GPU layers, 100000 cached tokens plus a 1000-token appended prompt. The fork gains +43.94% prompt processing at 100k cached + 1k appended (300.94 to 433.19 t/s) and +32.19% generation with the MTP3 decode stack (27.591 to 36.471 t/s), while standard llama-bench tg128 is effectively unchanged (27.89 to 27.72 t/s).

generation:

107.0 tokens/s

prompt processing (prefill):

4445.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.5 4B Q8_0 at 107 t/s generation and 4,445 t/s prompt processing on a Tesla V100 32GB. Setup is llama.cpp (PrismML fork 3ae4f51, CUDA 12.9) with flash attention on, -ngl 99, tested at 0, 16K and 32K context; generation drops to 93 t/s at 32K. An RTX 4070 control card reaches 91 t/s generation and 5,860 t/s prompt processing on the same model. The V100 draws 154 W while generating (0.69 tokens/joule) against the 4070's 173 W (0.53 tokens/joule).

generation:

42.2 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 42.2 t/s on 4x Tesla V100-SXM2-32GB (128 GiB VRAM total). Setup is a llama.cpp fork with tensor parallelism, CUDA_VISIBLE_DEVICES=0,1,2,3, GGML_CUDA_ALLREDUCE=nccl and GGML_CUDA_P2P=1, at 16k context. The 156 GiB model runs on 128 GiB of VRAM. At full 256k context with --ctx-size 262144 -ncmoe 12 decode is about 21.8 t/s. An expert-cache pool sweep on qwen35moe (256 experts, 8 used, 40 layers, -ncmoe 99) gives 16.1 t/s off, 20.4 t/s at N=8, 30.3 t/s at N=32, 39.2 t/s at N=64 and 46.7 t/s at N=128, with hit rates of 47%, 62%, 71%, 77%, 85% and 89%. Gains are sublinear in bytes moved: a run that copies almost nothing still tops out near 76 t/s against 102 t/s with no offload. All decode figures are single-stream.

generation:

40.3 tokens/s

prompt processing (prefill):

527.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwopus3.6 27B Coder-Compat at 40.3 t/s decode on a Tesla V100 32GB, with 527 t/s prompt throughput. Setup is llama.cpp v9836 (upstream-dflash branch) with the Q4_K_M GGUF at 8K context, flash attention on, and MTP n=3 speculative decoding; two decode runs averaged. Baseline without speculative decoding was 31.2 t/s decode. On an RTX 3060 12GB, Qwopus3.6 35B-A3B Coder-MTP ran 25.5 t/s baseline but dropped to 16.9 t/s with MTP n=3, a regression.

generation:

23.2 tokens/s

prompt processing (prefill):

433.1 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B on a V100 at 433.1 t/s prompt processing and 23.19 t/s generation with a Volta-optimized llama.cpp fork. Benchmark uses 100k cached tokens plus 1k new prompt tokens and 64 generated tokens; the fork improves the FlashAttention path for roughly 40% higher prompt processing. Upstream llama.cpp gave 306.7 t/s prompt and 23.29 t/s generation on the same setup. The user notes the gain is mostly limited to Qwen-like model shapes and also tested Qwen3.5 122B A10B, Laguna-S-2.1, and GLM-5.2.

generation:

20.0 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 20 t/s generation with a Q8 quant on 2x V100 32GB GPUs, 64 GB of VRAM in total, using llama.cpp. The model family is not specified. User asks about running multiple concurrent instances or agents in parallel to raise aggregate throughput.

generation:

82.0 tokens/s

quant:

Q8 (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-122B-A10B at 50 t/s and Gemma-4-26B-A4B at 113 t/s on a system with 16 GPUs. The system runs multiple models across the 16 GPUs. User notes hallucinations and pipeline issues.