NVIDIA A100 80GB

generation:

50-70 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 50-70 t/s single-user on dual A100 80GB, dropping to 10-15 t/s with 2-3 concurrent sessions. Setup is llama.cpp with UD-Q4_K_XL and Q8_0 KV cache, 7 parallel slots at 262k context each (1835008 total), Q8_0 MTP head with draft max 2, mmproj enabled, flash attention on, layer split, 32GB cache RAM. User is frustrated with multi-user scaling and asks whether to switch to vLLM or run Qwen3.6-35B-A3B per card; use case is long-session agentic coding.

generation:

18.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~18 tokens/s on an NVIDIA A100 80GB, compared with ~62 tokens/s for Qwen3.6-27B on the same hardware. Setup is vLLM with unquantized (BF16/FP16) weights, same context length and prompt for both models. User asks whether the 3.4x gap is expected and whether vLLM version, FlashAttention kernels, or Qwen3.8-specific configuration could explain it.

generation:

56.8 tokens/s

prompt processing (prefill):

1019.1 tokens/s

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks TensorSharp against llama.cpp on Qwen 3.8 Flash Next, reporting prompt processing and generation speeds for 1 GPU and 2 GPU layer split configurations. The promptTps and generationTps fields capture the 1 GPU pp512 and tg64 values respectively.