NVIDIA RTX A5000 reports

generation:

57-58 tokens/s

quant:

IQ3_S (GGUF)

kv:

q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B running on a single 24 GB RTX A5000 with llama.cpp b11491 (CUDA 13.4), using the IQ3_S-mtp quant (12.1 GB) with q4_0 KV cache at 256K context and MTP speculative decoding. Decode is 57-58 t/s at short context, falling to 27.5 t/s at 250K; prefill is ~900 t/s up to 32K and 478 t/s at 250K. Setup uses --parallel 1 (requests serialized), -ngl all, -ub 2048, and the mmproj vision encoder offloaded to GPU. A second recipe with UD-Q4_K_XL (17.5 GB) is also provided, reaching ~38-40 t/s short-context decode and 950-973 t/s prompt eval. MTP acceptance is 55-75% at temp 0 and 48-62% at temp 1.0. The user notes asymmetric KV quantization collapses prefill to ~34 t/s, and q8_0 KV above 128K is unstable on this architecture.