Local model performance on your hardware

generation:

201.0 tokens/s

quant:

AWQ (AWQ)

kv:

FP8

User reports Qwen3.8 Flash-Next at 201 tok/s on 4x V100 32GB PCIe cards with no NVLink, for a single request of English code. Setup is ReVolta 0.1.2, a vLLM fork, with an AWQ quant, 512K context, FP8 KV cache with CPU offload, TP4 plus DCP2, and DFlash2 speculative decoding drafting 8-token blocks from incoai's DFlash2 checkpoint. The user says this may be the first DFlash run on Volta (SM70); the table also lists a 163 figure beside the 201 tok/s row.

generation:

69.3 tokens/s

prompt processing (prefill):

622.0 tokens/s

quant:

IQ2_XXS (GGUF)

User reports Maya-S-v2-IQ2_XXS at 69.3 tok/s decode on four Tesla V100-PCIE-32GB cards, with prefill of 622 tok/s at 8192 tokens and 466 tok/s at 2048. Setup is engine v1.1.19 at 32K context, with MTP speculative decoding on (589 of 750 drafts accepted, 78.5%) and the whole model resident in VRAM across 128 GB. The 69.3 tok/s is the mean of 3 greedy runs with thinking off; the engine's own stat logged 14.52 ms/token (68.86 tok/s) at a 100% VRAM hit rate. The split leaves about 33 GB of VRAM unused, and the 96.5 GB model loads from a SATA SSD in about 180 s. These are the engine's own split numbers, before --calibrate.

generation:

16.7 tokens/s

prompt processing (prefill):

61.7 tokens/s

quant:

IQ3_XXS (GGUF)

User reports Qwen3.8-Flash-Next at 16.7 t/s steady on a Tesla V100-PCIE-32GB, with prompt processing at 61.7 t/s warm. Setup is Strata v0.1.38 compiled for sm_70 with IQ3_XXS, 131,072 context, KV streamed to RAM, MTP draft (q2_0), and 15,553 of 24,576 experts (63%) cached in VRAM (25.2 GiB). Speed stays flat at 16.7-16.9 t/s through a 31k-token session; MTP acceptance is 97.8% on repetitive output and ~65% on free prose. The user also reports a thinking-budget pitfall where hard prompts can exhaust max_tokens on reasoning and return empty content, fixed by sending reasoning_budget_tokens: 4000 per request. A comparison against Qwen3.8-27B on llama.cpp on the same hardware found the 125B stronger on long-tail knowledge and proof depth, while the 27B was 1.7-2x faster in wall clock and better at fine-grained format obedience.

generation:

65.1 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.8-27B at 65.07 tok/s on a single Tesla V100-SXM2-32GB at 1K context. Setup is llama.cpp (pinned tag b10793) with Q4_K_M GGUF and MTP speculative decoding enabled, 128K context window, single stream. Decode falls to 62.41 tok/s at 8K, 61.24 at 32K, 52.51 at 64K and 42.22 at 128K. With MTP off on a T2-001 packed-attention build the same card gives 35.22 tok/s at 1K and 23.83 at 128K; stock llama.cpp gives 35.95 at 1K and 16.49 at 128K. Prompt caching cuts multi-turn wall time from 22.94 s to 3.08 s per turn.

generation:

29.3 tokens/s

prompt processing (prefill):

1055.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

User benchmarks Qwen3.8-27B NVFP4 at 29.3 tok/s single-stream decode on one Tesla V100-PCIE-32GB, comparing 1Cat-vLLM v1.5.1 against llama.cpp. Setup is 1Cat-vLLM with NVFP4 weights and fp8_e4m3 KV cache at 32K context, tensor parallel 1, 29.9 GB VRAM; llama.cpp ran the same model as UD-Q4_K_M GGUF with q8_0 KV cache at 131072 context, 20.6 GB VRAM. 1Cat is about 10% behind llama.cpp on single-stream decode (32.7 tok/s) but 45% ahead with 4 concurrent requests (110 tok/s aggregate vs 75.7) and reads long prompts 25-42% faster (1055 tok/s at 2.9K down to 847 at 29.4K). A separate Qwen3-8B FP16 run on the same card gave 40.7 tok/s single-stream and 141 tok/s at 4 concurrent for 1Cat, against 49.4 and 155 for llama.cpp.

generation:

57.3 tokens/s

prompt processing (prefill):

720.0 tokens/s

quant:

UD-Q4_K_M (GGUF)

User reports Qwen3.8 27B UD-Q4_K_M at 57.25 t/s decode (depth 0) on a single Tesla V100-PCIE-32GB. Setup is llama.cpp with llama-split-bench --profile, MTP enabled, measured up to 262k context. Prefill is about 720 t/s on new prompts. Decode falls to 20.49 t/s at depth ~260k. An initial run suffered thermal throttling (15.68 t/s at 64k depth) due to BMC fan control; values shown are with fan mode set to Full.

generation:

14.7 tokens/s

quant:

EXL3 6.00bpw mixed K 5-8 (EXL3)

User reports 14.7 tok/s end-to-end on a 27B EXL3 SC pack on a V100-DGXS-32GB, matching the vLLM-plugin path within noise. Setup is ExLlamaV3 with EXL3 6.00bpw mixed K 5-8 weights on a single device. The PR ports EXL3 quantized-matrix kernels to Volta (sm_70), adding a GEMV decode kernel, arch gating, and reference tests. Single-device GEMV decode is 0.98-2.12x vs the hgemm floor across K2-4 shapes, with 664 GB/s aggregate weight-read at K=6 decode.

generation:

50.0 tokens/s

quant:

UD-Q4-K-M (GGUF)

User reports Qwen3.8 27B at about 50 t/s decode on a V100 32GB. Setup is llama.cpp with UD-Q4-K-M (MTP) and batch size 4096, ubatch 2048, on a PG500-216 card at 185W with a custom compile. User notes prefill drops off quickly and asks for pointers on improving it.

generation:

10.0 tokens/s

prompt processing (prefill):

52.0 tokens/s

quant:

Q8_0 (GGUF)

User reports Qwen3.8-Flash-Next Q8_0 at ~52 tokens per second prompt processing and ~10 tokens per second generation on a V100 32GB. Setup is llama.cpp with a 262144-token context, --cpu-moe and -ngl 64, on a machine with 128GB system RAM; the GPU drew 95.85 W at 48% utilization with 14206 MiB of its 32768 MiB in use. Both throughput figures are given as approximate.

generation:

50-55 tokens/s

prompt processing (prefill):

850-900 tokens/s

quant:

IQ4_XS (GGUF)

kv:

fp16

User reports Qwen3.8-Flash-Next Uncensored at 50-55 t/s decode and 850-900 t/s prefill on a Tesla V100 32GB, with general tasks at 40-45 t/s. Setup is Strata with IQ4_XS and fp16 KV cache at 262K context, single card with 180W power cap. User also ran Qwen3.8-Flash-Next IQ3_S on an RTX 3090 at 60-65 t/s decode and 1200-1300 t/s prefill, and notes the uncensored IQ4_XS variant handles non-English writing better.

generation:

47.2 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0

User reports Qwen3.8-27B at 47.2 t/s decode on one Tesla V100-SXM2-32GB (250 W) in an X99 box with 2x Xeon E5-2690 v4 and 251 GB RAM. Setup is llama.cpp (V100 tensor-parallel fork) with IQ3_S GGUF weights, q8_0 KV cache, 128k context, and MTP self-speculation with up to 3 draft tokens per step. A 64k context run with f16 KV reached 52.5 t/s with 91.9% draft acceptance; the 128k run had 88.0% acceptance. Per 8k-token stretches the rate ranged 46-66 t/s at 64k and 41-57 t/s at 128k.

generation:

43.3 tokens/s

prompt processing (prefill):

833.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Qwen3.8-27B-Uncensored at 833.37 prompt tok/s and 43.34 decode tok/s on two NVIDIA Tesla V100 GPUs. Setup is llama.cpp with Q8_0 weights, F16 KV cache, 262,144-token context, tensor split 1,1, and the built-in MTP head with draft length 3. The matching DFlash2 Q8 candidate measured 607.82 prompt tok/s and 33.00 decode tok/s, so MTP remains the production default. A single-V100 Q4_K_M profile with Q4_0 KV and MTP draft length 1 reached 44.201 tok/s on a 512-token short test and 13.001 tok/s on a 255,000-token capacity run.

generation:

32.2 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.6 27B at 32.17 t/s generation on a single NVIDIA Tesla V100 SXM2 32GB. Setup is llama.cpp (build 6eab47181) with Q4_K_M GGUF weights, run via llama-bench with -p 0 and -n 128,256,512,1024,2048, steady across those generation lengths. Other quants measured on the same card: Q5_K_M 29.14 t/s, Q6_K 25.11 t/s, Q8_0 22.80 t/s. BF16 required an added RTX 3090 24GB and ran 15.27 t/s.

generation:

65.1 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.8 27B at 65.07 t/s decode at 1K context on a Tesla V100-SXM2-32GB. Setup is llama.cpp pinned at tag b10793 with the repo's patches, Q4_K_M GGUF, 1 parallel slot, multi-token prediction speculative decoding enabled. Decode falls to 42.22 t/s at 128K context; stock llama.cpp gives 35.95 t/s at 1K and 16.49 t/s at 128K, and the patched build without MTP gives 35.22 t/s at 1K and 23.83 t/s at 128K. Multi-turn agentic sessions with prompt caching take 3.08 s per turn versus 22.94 s without.

generation:

33.4 tokens/s

prompt processing (prefill):

826.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8-27B at 33.41 tok/s generation and 826.03 tok/s prefill on a single Tesla V100 32GB. Setup is llama.cpp with UD-Q4_K_XL GGUF at 131,072 context; generation falls to 27.95 tok/s at 32k depth and 15.84 tok/s at 131k depth. Peak VRAM was 25,264 MiB of 32,768 MiB. KV cache quantization (-ctk q8_0 -ctv q8_0) silently fell back to CPU with 0% GPU utilization; CUDA 13 dropped Volta support.

generation:

27.7 tokens/s

prompt processing (prefill):

433.2 tokens/s

quant:

UD-Q5_K_XL (GGUF)

kv:

q8_0

User reports Qwen3.8-27B Flash-Next at 433.19 t/s prompt processing and 27.72 t/s generation on a Tesla V100 32GB, comparing a custom llama.cpp fork against vanilla llama.cpp. Setup is llama.cpp (v100-optimized fork) with UD-Q5_K_XL GGUF, q8_0 K/V cache, 262144 context, FlashAttention on, MTP n-max=3, 63 GPU layers, 100000 cached tokens plus a 1000-token appended prompt. The fork gains +43.94% prompt processing at 100k cached + 1k appended (300.94 to 433.19 t/s) and +32.19% generation with the MTP3 decode stack (27.591 to 36.471 t/s), while standard llama-bench tg128 is effectively unchanged (27.89 to 27.72 t/s).

generation:

107.0 tokens/s

prompt processing (prefill):

4445.0 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks Qwen3.5 4B Q8_0 at 107 t/s generation and 4,445 t/s prompt processing on a Tesla V100 32GB. Setup is llama.cpp (PrismML fork 3ae4f51, CUDA 12.9) with flash attention on, -ngl 99, tested at 0, 16K and 32K context; generation drops to 93 t/s at 32K. An RTX 4070 control card reaches 91 t/s generation and 5,860 t/s prompt processing on the same model. The V100 draws 154 W while generating (0.69 tokens/joule) against the 4070's 173 W (0.53 tokens/joule).

generation:

42.2 tokens/s

User reports Qwen3.8-Flash-Next at 42.2 t/s on 4x Tesla V100-SXM2-32GB (128 GiB VRAM total). Setup is a llama.cpp fork with tensor parallelism, CUDA_VISIBLE_DEVICES=0,1,2,3, GGML_CUDA_ALLREDUCE=nccl and GGML_CUDA_P2P=1, at 16k context. The 156 GiB model runs on 128 GiB of VRAM. At full 256k context with --ctx-size 262144 -ncmoe 12 decode is about 21.8 t/s. An expert-cache pool sweep on qwen35moe (256 experts, 8 used, 40 layers, -ncmoe 99) gives 16.1 t/s off, 20.4 t/s at N=8, 30.3 t/s at N=32, 39.2 t/s at N=64 and 46.7 t/s at N=128, with hit rates of 47%, 62%, 71%, 77%, 85% and 89%. Gains are sublinear in bytes moved: a run that copies almost nothing still tops out near 76 t/s against 102 t/s with no offload. All decode figures are single-stream.

generation:

40.3 tokens/s

prompt processing (prefill):

527.0 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Qwopus3.6 27B Coder-Compat at 40.3 t/s decode on a Tesla V100 32GB, with 527 t/s prompt throughput. Setup is llama.cpp v9836 (upstream-dflash branch) with the Q4_K_M GGUF at 8K context, flash attention on, and MTP n=3 speculative decoding; two decode runs averaged. Baseline without speculative decoding was 31.2 t/s decode. On an RTX 3060 12GB, Qwopus3.6 35B-A3B Coder-MTP ran 25.5 t/s baseline but dropped to 16.9 t/s with MTP n=3, a regression.

generation:

23.2 tokens/s

prompt processing (prefill):

433.1 tokens/s

User reports Qwen3.8 27B on a V100 at 433.1 t/s prompt processing and 23.19 t/s generation with a Volta-optimized llama.cpp fork. Benchmark uses 100k cached tokens plus 1k new prompt tokens and 64 generated tokens; the fork improves the FlashAttention path for roughly 40% higher prompt processing. Upstream llama.cpp gave 306.7 t/s prompt and 23.29 t/s generation on the same setup. The user notes the gain is mostly limited to Qwen-like model shapes and also tested Qwen3.5 122B A10B, Laguna-S-2.1, and GLM-5.2.