NVIDIA V100 16GB

generation:

353.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 353 t/s on four V100 16GB cards. Setup is NInfer with NVFP4 weights, Q8 KV cache, 256k max context and MTP 4/0 speculation, tensor parallel across 4 GPUs. The user previously ran 2 V100s at 181 t/s and notes the repo now supports tp4.

generation:

93.0 tokens/s

quant:

mix_q3gud_im (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MiniCPM-V 4.7 35B-A3B at about 93 tok/s on a Tesla V100 16GB under Windows 11, with vision and 128k context. Setup is stock llama.cpp with OpenBMB's official mix_q3gud_im quant (15.5 GB), q4_0 KV cache, flash attention on, and the vision encoder kept on CPU so all MoE experts fit on the GPU. A 120k-token prompt was answered correctly without a crash, using 16.17 of 16.38 GB VRAM. At 262k context generation drops to 70 tok/s. DSpark n-gram speculative decoding was slower because the drafter's VRAM pushed experts back to RAM. The user also benchmarked 15 models on a 72-task code-graded suite, where this model scored 87.5 overall.

generation:

50.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks whether NInfer can run on a V100 16GB plus RTX 4070 12GB, reporting 50 t/s on llama.cpp and 35 t/s on coding at long context. Setup is llama.cpp with an IQ3_XXS Flash-Next quant; the machine has 32GB of system RAM. The user also gets a sustained 40 t/s on Strata with the same IQ3_XXS Flash-Next, which they find performs better than a Q4 27B, but the 32GB of RAM causes slow prefill and occasional freezing.

generation:

143.5 tokens/s

prompt processing (prefill):

1310.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

INT8 group-64

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B NVFP4 at 143.5 committed output tok/s on two Tesla V100-SXM2 16GB cards with an 85,000-token coding input. Setup is NInfer-V100X2 with NVFP4 weights, INT8 group-64 KV cache, MTP4, 180,000-token context capacity, PCIe P2P and no NVLink. Prefill averaged about 1,310 tok/s over the 85K input, with peaks around 2,500 tok/s possible under PP2 pipeline parallelism. Short structured JSONL generation with DFlash7 reached 253+ tok/s, averaging 255.3 tok/s over the final two runs with 118 input tokens and 99.05% draft acceptance. The 143 tok/s route uses an experimental attention implementation that can change outputs and is off by default; on 104 GPQA-Diamond questions with 85K padding it scored 90/104 versus 88/104 for default attention.

generation:

180.0 tokens/s

quant:

NVFP4

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 180 t/s on a pair of V100 16GB cards without NVLink. Setup uses NInfer with NVFP4 weights, Q8 KV cache and MTP 4/0 speculative decoding. The user calls it the legend of V100 and links the tuxKOH/ninfer-V100X2 repo.

generation:

57-60 tokens/s

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 57-60 tok/s on a rented V100 16GB. Setup is AgrillaMoE, a fork of llama.cpp's server, with the 2-bit UD-Q2_K_XL quant and a MoE-expansion patch raising active experts to 20 on layers 25-39. The user says the expansion patch improves GPQA-Diamond from 81.82% to 84.34% with Q8_0, and that the server speaks both OpenAI and Anthropic APIs.

generation:

96.7 tokens/s

quant:

PXQN2

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 96.7 t/s decode on two V100s with PXA v3. Setup is PXA engine with PXQN2 quant, 4,096-token prompt, speculative decoding on. Same model runs 85.8 t/s on one V100, 60.1 t/s on two P100s and 48.1 t/s on one P100. Gemma 4 26B-A4B reaches 168.6 t/s on one V100 and 71.1 t/s on one P100. A Flash-Next MoE runs 36.3 t/s on four P100s and 20.4 t/s on one P100 with system RAM.

generation:

52.0 tokens/s

prompt processing (prefill):

398.6 tokens/s

quant:

Q2_0 (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 52.0 t/s decode and 398.6 t/s prefill on a single Tesla V100-PCIE-16GB (PCIe Gen3 x16). Setup is the Strata-V100 fork with Q2_0 weights, int8 KV cache, 262,144-token context, MTP with --spec 8 and draft floor 0.70, a 20/28 layer split, and 700 MiB vision reserve; the machine has a Ryzen 5 3600 and 48 GB DDR4-3200. Each value is the median of three fresh uncached requests with exactly 256 output tokens. Decode falls from 52.0 t/s at 1K to 36.8 t/s at 256K; the 128K and 256K prompts hit 84 °C and thermally throttled during prefill. A separate before/after run of the integrated build measured prefill gains of +2.44% at 2K, +3.93% at 8K and +0.94% at 32K.

prompt processing (prefill):

7090.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next UD-Q4_K_XL at up to 7,350 tok/s prefill and ~113 tok/s decode on an IBM AC922 with 4x Tesla V100 16GB SXM2 GPUs. Setup is a forked Strata engine with MTP speculative decoding, 72 GiB of experts page-locked in RAM across both sockets, and GPUs pulling from NVLink 2.0 at ~70 GB/s each. The 252K-token prompt prefilled at 7,090 tok/s in 35 s; follow-up at that depth gave first token after 0.26 s and 60 tok/s. Generation varied by workload: ~113 tok/s peak on JSON, ~100 on code, ~84 on prose. Stock llama.cpp on the same machine produced 130 tok/s prefill and 15 tok/s decode.

prompt processing (prefill):

2500-3500 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 27B at over 200 tok/s generation on 4x V100 16GB GPUs. Setup is vLLM with NVFP4 Nvidia checkpoints and DFlash speculative decoding, tensor parallel 4, with image support enabled and KV cache around 120k tokens. Prefill is reported as 2.5-3.5k t/s. The user notes the server cost $5400 on eBay and used a custom repo to unpack NVFP4 into FP16 on the fly.

generation:

4.6 tokens/s

prompt processing (prefill):

53.0 tokens/s

quant:

Q6_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User asks for help building an offline LLM rig and reports a test of Qwen3.5 9B at Q6_K on their main rig, getting 53 t/s prompt processing and 4.6 t/s generation with ik_llama.cpp. Setup is ik_llama.cpp with Q6_K; the test rig's hardware is not specified. The planned build is dual Xeon 2683v4 with 256GB DDR4 ECC and a V100 16GB, targeting DeepSeek V4 Flash and MiMo V2.5 at UD-IQ4_XS. User estimates 6-7 t/s generation and 50-60 t/s prompt processing for DeepSeek V4 Flash on CPU alone, hoping a GPU raises prompt processing to 200-300 t/s, and expects MiMo V2.5 to be 1-2 t/s slower. ik_llama.cpp doubled prompt processing and added 1 t/s generation over llama.cpp in their test.

generation:

68.6 tokens/s

quant:

PXQ4 (PXQ)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 68.6 t/s on 2× V100 16GB with the PXA engine's new tensor split, up from 51.0 t/s with the layer split it replaces. Setup is PXA v2026.09.20 with the PXQ4 quant format on PCIe x4 risers; the same run hit 107.4 t/s on repetitive text and 43.6 t/s on a ~15k-token prompt. A head-to-head against stock llama.cpp on the same cards gives 80.2 vs 58.9 t/s on prose, 139.8 vs 80.5 on repetitive text, and 47.8 vs 46.1 on the ~15k-token prompt. The user notes the long-context row is a narrow win and that prompt processing under the tensor split is slower than their own layer split on x4 risers (850 vs 1,020 t/s).

generation:

43.4 tokens/s

prompt processing (prefill):

4679.0 tokens/s

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Next running on a 6x V100 16GB setup with TP2 PP3, reaching up to 4,679 t/s prefill at 16K input and 43.38 t/s generation with MTP enabled. Setup uses the 1cat-vllm engine with speculative decoding limited to speculative=1; KV cache is 8.78 GiB, 531,288 tokens, supporting 2.03x concurrency at 262,144 tokens per request. MTP nearly doubles generation throughput versus base (22.59 t/s average without MTP). A 20-minute gpu-burn stress test peaked at 65°C and stabilized at 64°C with fans at 76%.

generation:

39.9 tokens/s

prompt processing (prefill):

1376.9 tokens/s

quant:

Q6_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 39.88 t/s decode and 1,376.87 t/s prompt processing on a mismatched pair of Tesla V100 PCIe cards (16 GB + 32 GB, 48 GB total). Setup is mainline llama.cpp with the Q6_K_M GGUF, tensor split, Flash Attention, Q8 KV cache, batch/ubatch 2,048, 20 threads, and NUMA distribute, with the 32 GB V100 as main GPU. Prompt processing measured 1,324.34 t/s at 4k and 1,221.49 t/s at 16k. A Qwen3.8 Flash Next 177B/6B-active MoE Q4_K_XL GGUF ran at 26.29 t/s decode and 380.67 t/s prompt on the same two cards.

generation:

30-40 tokens/s

prompt processing (prefill):

600-700 tokens/s

quant:

Q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 30-40 t/s decode and 600-700 t/s prefill on 3x V100 16GB. Setup uses Q8 quantization with speculative decoding, MTP, and prefix caching, tuned by an automated agent. On heavy agentic work at 256k context, decode dropped to around 20 t/s. Qwen3.8 Flash Next at Q4 ran at 20 t/s decode and 90 t/s prefill. The build cost around $1500 and uses a 3D printed cooling block; GPU temperatures stay under 55C.

generation:

219.1 tokens/s

quant:

NVFP4

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 NVFP4 on four Tesla V100s (2017) via a custom v100-skinny engine matching RTX 5090 decode throughput at ~220 tok/s. Setup uses QPN kernels to translate FP4/FP8 to FP16 for Volta tensor cores. Long-context tests show MTP depth k=3 better at ~65K context. The figure is for 4 GPUs versus 1, is not power efficient, and prefill is slower.