NVIDIA RTX A4000 16GB reports

generation:

30.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at about 30 t/s on an RTX A4000 VM. The model is the DavidAU TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP GGUF. The VM costs $0.15/hour and is shared with a friend; the user offers to move to an RTX A6000 if more people join.

generation:

23.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at ~23 t/s on 3x RTX A4000 16GB. Setup is llama.cpp with Q8_0 quantization, MTP draft mode and ngram speculative decoding. User asks for optimization advice.