NVIDIA V100 32GB reports

generation:

218.0 tokens/s

quant:

NVFP4

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports best-case MTP decode of 218 tok/s with 99.2% acceptance on a Volta V100. Setup is the custom NInfer inference engine with software NVFP4. Throughput is about 200 tok/s without context-copy speculation.

generation:

65.7 tokens/s

prompt processing (prefill):

2273.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports PXA, a fork of ik_llama.cpp with a vLLM plugin, at 65.7 t/s plain decode with TP4 on an 8x V100 SXM2 NVLink system. Setup is PXA as shipped with tensor parallel across four cards and no speculative decoding, measured on 16 GSM8K questions with 192 greedy tokens. With speculative decoding at k=7 the same TP4 setup reaches 159.6 t/s, and prefill at 3k is 2273 t/s with a TTFT of 1.4 s. 1Cat vLLM on the same system reads 61.0 t/s plain decode at TP4.

generation:

10.0 tokens/s

prompt processing (prefill):

200.0 tokens/s

quant:

Q3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Laguna S2.1 at about 10 t/s decode and about 200 t/s prefill on a V100 32GB with CPU offload and 50GB DDR4. Setup is the Q3 quant at 262k context. User finds it too verbose for planning but excellent for complex debugging, fixing issues that Qwen and Claude could not, and compares its reasoning style to GLM 5.2. It is not a generalist replacement but is useful as a specialized debugger.

generation:

60.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports prefill of about 4000 t/s and decode of about 60 t/s at 256k context on 4x V100 32GB. Setup offloads 50GB of ngram data to system RAM. MTP improves decode at low concurrency.

generation:

20.0 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 20 t/s generation with a Q8 quant on 2x V100 32GB GPUs, 64 GB of VRAM in total, using llama.cpp. The model family is not specified. User asks about running multiple concurrent instances or agents in parallel to raise aggregate throughput.

generation:

82.0 tokens/s

quant:

Q8 (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-122B-A10B at 50 t/s and Gemma-4-26B-A4B at 113 t/s on a system with 16 GPUs. The system runs multiple models across the 16 GPUs. User notes hallucinations and pipeline issues.

generation:

54-55 tokens/s

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 29-30 t/s without MTP and 54-55 t/s with MTP on the MTP branch of llama.cpp by am17an, at a 150W power limit. The model is used as a VSCode Copilot backend. Throughput falls to 40-45 t/s after 50k tokens.