V100 32GB

reported speed:

65.7 tokens/s generation · 2273.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post describes PXA engine, a fork of ik_llama.cpp with vLLM plugin, for old datacenter cards. Benchmarks on 8x V100 SXM2 NVLink system. Prefill @3k: 2273 t/s (TTFT 1.4s), decode TP4 plain: 65.7 t/s. Also mentions running Qwen3.8 Flash-Next on four P100s with 150k context. Speculative decoding with k=7 gives 159.6 t/s (TP4).

reported speed:

10.0 tokens/s generation · 200.0 tokens/s prompt processing

quant:

Q3

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Model is Laguna S2.1 (not in canonical list). Ran on V100 32GB with CPU offload, 50GB DDR4. Q3 quant, 262k context. Decode ~10 tps, prefill ~200 tps. Found it too verbose for planning but excellent for complex debugging, fixing issues that Qwen and Claude couldn't. Compares reasoning style to GLM 5.2. Not a generalist replacement but useful as a specialized debugger.

reported speed:

60.0 tokens/s generation

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

4xV100 32GB, 50GB ngram offloaded to system RAM. Prefill ~4000 tks, decode ~60 tks at 256k context. MTP improves decode at low concurrency.

reported speed:

20.0 tokens/s generation

quant:

Q8 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User is running a model with Q8 quantization on 2x V100 32GB GPUs (64GB total VRAM) using llama.cpp, achieving ~20 tok/s generation. They are asking about parallelizing multiple concurrent instances/agents to increase aggregate throughput. The model family is not specified.

reported speed:

82.0 tokens/s generation

quant:

Q8 (gguf)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Workhorse drafting model on Board A. Also mentions Qwen3.5-122B-A10B at 50 tok/s and Gemma-4-26B-A4B at 113 tok/s. System uses multiple models across 16 GPUs. User notes hallucinations and pipeline issues.