Qwen3.8 27B
on NVIDIA L4 · llama.cpp · 226,048 ctx
Use cases
codingsummarizationmathlong-context
Summary
User reports Qwen3.8-27B at 32.96 tok/s average decode (24.18 stock) on an NVIDIA L4 24 GB, with 39.4 tok/s on prose and 37.4 tok/s across seven workloads.
Setup is llama.cpp (PR #27342 plus a MMVQ crossover patch) with UD-Q4_K_XL GGUF, q4_0 KV cache, 226,048-token context, DFlash 2 Q4_K_M drafter at n-max 7, and GGML_MMVQ_MAX=2; 23.0 of 24.0 GiB resident, ECC off, 72 W cap.
Prefill is 579 tok/s at a 7k prompt, 514 at 33k and 387 at 79k; mean accepted length is 3.04. The user notes the card cannot reach 100 tok/s because the 15.75 GiB weight read would need 6.7 accepted tokens per pass, and that a 3090-class card would land near 101 tok/s.