Local model performance on your hardware

generation:

50.4 tokens/s

prompt processing (prefill):

815.6 tokens/s

quant:

NVFP4 hybrid 5.01 bpw (GGUF)

kv:

Q4_0

User reports 50.44 t/s generation and 815.64 t/s prompt processing. Source evidence (verbatim): Source: https://hackernoon.com/how-i-got-qwen38-27b-running-at-50-toks-with-256k-context # How I got Qwen3.8 27B running at 50 tok/s with 256K context Two GPUs: an NVIDIA RTX PRO 4000 Blackwell SFF 24 GB (432 GB/s) holds the target model, the MTP head and the 256K KV cache, and an RTX 2000 Ada 16 GB holds the F16 vision projector. Debian 13, CUDA 12.9. Custom llama.cpp build with pinned PRs. Model Qwen3.8 27B, custom 5.01 bits-per-weight iMatrix/NVFP4 hybrid GGUF (16.3 GB), context 262144, Q4_0 KV cache. Target-only at about 40K context: 22.40 tokens/s generation. With MTP (n_max 3) at about 40K context: 44.95 tokens/s. Production ten-run mean with MTP n=8 plus n-gram: 50.44 tokens/s. Greedy target-only 21.19 tokens/s, greedy MTP n=8 59.46 tokens/s. Filled 261.5K cache: 12.06 tokens/s. Prompt processing at 32K context with a flash attention patch: 815.64 tokens/s (759.38 before the patch).

generation:

84.6 tokens/s

quant:

Q5_K_M

User reports speculative decoding with DFlash reaching 84.64 t/s at 38.67% acceptance on a best tuned 256K run, against 17.98 t/s for regular decoding. An NVFP4 hybrid reached 94.36 t/s but with worse perplexity, so Q5_K_M was chosen as best. A context fill of 262,116 tokens completed without OOM using 22,920/24,467 MiB VRAM, with decode at 21.56 t/s.

generation:

50.4 tokens/s

prompt processing (prefill):

226.8 tokens/s

quant:

NVFP4 (GGUF)

User reports a custom iMatrix quant with mixed precision: NVFP4 for bulk matrices, Q5_K/Q6_K for attention, DeltaNet and FFN, Q6_K embeddings, Q8_0 output head and NVFP4 MTP. WikiText-2 PPL is 6.1197 versus 6.1127 for Q4_1. MTP target-only decode runs at 21.189 t/s, with MTP at 59.456 t/s. At full 261.5K context decode is 12.606 t/s. The GGUF is published on HF.