Qwen3.8 27B
NVIDIA RTX A5000 · llama.cpp · 262,144 ctx
- generation:
- 57-58 tokens/s
- quant:
- IQ3_S (GGUF)
- kv:
- q4_0
- flash attention:
- on
User reports Qwen3.8-27B running on a single 24 GB RTX A5000 with llama.cpp b11491 (CUDA 13.4), using the IQ3_S-mtp quant (12.1 GB) with q4_0 KV cache at 256K context and MTP speculative decoding. Decode is 57-58 t/s at short context, falling to 27.5 t/s at 250K; prefill is ~900 t/s up to 32K and 478 t/s at 250K. Setup uses --parallel 1 (requests serialized), -ngl all, -ub 2048, and the mmproj vision encoder offloaded to GPU. A second recipe with UD-Q4_K_XL (17.5 GB) is also provided, reaching ~38-40 t/s short-context decode and 950-973 t/s prompt eval. MTP acceptance is 55-75% at temp 0 and 48-62% at temp 1.0. The user notes asymmetric KV quantization collapses prefill to ~34 t/s, and q8_0 KV above 128K is unstable on this architecture.