llamaperf

Qwen3.8 27B

on NVIDIA L4 · llama.cpp · 226,048 ctx

NVIDIA hardware
Oct 4, 2026
Throughput
33.0 t/s gen · 579.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
UD-Q4_K_XL (GGUF)
KV cache
q4_0
VRAM reported
24 GB

Use cases

codingsummarizationmathlong-context

Summary

User reports Qwen3.8-27B at 32.96 tok/s average decode (24.18 stock) on an NVIDIA L4 24 GB, with 39.4 tok/s on prose and 37.4 tok/s across seven workloads. Setup is llama.cpp (PR #27342 plus a MMVQ crossover patch) with UD-Q4_K_XL GGUF, q4_0 KV cache, 226,048-token context, DFlash 2 Q4_K_M drafter at n-max 7, and GGML_MMVQ_MAX=2; 23.0 of 24.0 GiB resident, ECC off, 72 W cap. Prefill is 579 tok/s at a 7k prompt, 514 at 33k and 387 at 79k; mean accepted length is 3.04. The user notes the card cannot reach 100 tok/s because the 15.75 GiB weight read would need 6.7 accepted tokens per pass, and that a 3090-class card would land near 101 tok/s.