Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.
User reports about 38 t/s average generation on 2x P100, with 60+ t/s on some tasks when MTP is enabled.
Prefill is about 200 t/s, which the user calls painfully slow; the system draws 500W under load.
The user says it works but is basically unusable for anything interactive and asks whether prefill can be sped up on this old hardware.
User reports Qwen3.8-27B at about 45 tok/s in English and about 32 tok/s in Polish on 2x Tesla P100 16GB.
Setup is llama.cpp with UD-Q5_K_XL at 262k context, row/tensor split with MTP, on an X79 platform (i7-3930K, ~44 GB DDR3) under Proxmox with GPU passthrough.
The P100s serve as the main LLM backend for an agent with long prompts and many tool calls; a GTX 1080 handles vision, Whisper, embeddings and image generation. The user asks whether to replace the 1080 with a Titan V or 2080 Ti, and reports a Quadro GP100 was not detected at all on the board.
User reports about 30 tok/s with the Swift-1.5 version of Qwen3.8 Flash-Next on a Tesla P100 16GB in a PCIe3 x16 slot, after switching from llama.cpp to Strata.
Setup is Strata with 64GB of ECC DDR4 at 2100 and a 131072-token context; the logs show expert cache hits of 63.5% to 77.8% with routed experts going over PCIe, so part of the model ran from system RAM.
Individual runs in the pasted logs range from 28.7 to 33.3 tok/s on prompts of roughly 78k tokens, which were read in 2 to 3 s. The user says it serves their Hermes agent and calls it impeccable for a local LLM.
User reports Qwen3.8-27B Q4_0 at 33.28 t/s decode and 406.9 t/s prefill on 2x Tesla P100 16GB with patched llama.cpp, up from 23.43 t/s decode and 233.6 t/s prefill on stock.
Setup is llama.cpp v0.7.0 with 41 sm_60 patches, tensor split across two P100s, 5 interleaved rounds per config.
The patch set also gives 77.6 t/s decode on Ornith-1.0-35B-A3B UD-Q4_K_M with tensor split, and layer split gives 20.68 t/s decode on the 27B. Tested only on 2x P100; some patches are generic for all cards.
User reports Qwen3.8 27B at 10.01 t/s on a Tesla P100 16GB.
Setup uses the Q3_K_M GGUF quant with a 2,180-token prompt that took 3 minutes 37 seconds.
The user asks whether this speed is normal and whether a 14B model would be faster, noting the P100 is old hardware.
User reports Qwen3.8-Flash-Next at 5.84 t/s generation and 56.10 t/s prompt processing on an Nvidia P100 16GB.
Setup is llama.cpp with IQ3_XXS quant and 80K context, running on an HP Z440 workstation with Intel Xeon E5-1650 v3 and 64GB RAM.
The model is a 125B MoE with 6B active parameters, partially offloaded to CPU. User notes it is slow but works for non-time-critical tasks like email sorting and calendar management.
User reports Qwen3.8 35B at Q6 running at 25 t/s with about 128k context on 2x Tesla P100 16GB.
User says the model cheats heavily on their personal benchmark (building a clock in Lua for Conky) and still has trouble getting it working.
User asks whether their prompting is the problem or if they are jaded by frontier models, and whether to have Qwen push work to Opus/Sol.
User reports Qwen3.8 Flash-Next at 31.2 t/s decode on a single Tesla P100 16GB with PXA v3.1 Overdrive mode, using a 32 GB quant.
Setup is PXA with a 32 GB quant, experts cached on card and rest streamed, single request.
Overdrive off gives 20.0 t/s; two P100s reach 38.2 t/s, one V100 35.4 t/s, two V100s 55.7 t/s. Against Strata on the same P100, prose decode 31.1 vs 27.0 t/s, code 34.3 vs 34.1, prompt processing 4k 326 vs 226 t/s and 8k 338 vs 246 t/s.
User benchmarks llama.cpp CUDA patches on two Tesla P100-PCIE-16GB cards, reporting Qwen3.8-27B Q4_K_XL at 24.5 tok/s decode with -sm tensor, up from 22.3 tok/s stock.
Setup is llama.cpp b10660 with patches 0001-0012, CUDA 12.4, F16 KV cache, tensor split across two cards.
The README also reports Llama-3.1-8B Q4_K_M at 79.57 tok/s on one card (2.03x stock), Qwen3.8-27B pp8192 at 378.7 tok/s, and a real 74,919-token request at 20.92 tok/s with F16 KV versus 16.85 with q8_0. Four concurrent agents reach 20.3 tok/s each (72.9 aggregate).
User reports Qwen3.6 35B-A3B at 120-140 t/s generation on two Tesla P100s, with prefill around 1000 t/s at 0 context and 700 t/s at 13k context.
Setup uses Unsloth's UD_Q4_K_XL quant; the engine is not named.
The user praises the model for broad agentic tasks and notes it is fast enough to be conversational, but criticizes it for coding, persona holding, and hallucination. They also mention running Qwen3.8 Flash-Next at 30-40 t/s with Strata at Q3.
User reports Qwen3.8-27B Q6_K at 32.6 t/s decode (tg256, no MTP) on 2x Tesla P100 16GB with tensor split, versus 17.51 t/s upstream at the fork point.
Setup is a llama.cpp fork with CUDA work for Pascal (sm_60), q4_0 KV cache, and fp16 math with fp32 accumulation. Prefill pp2048 at 0 context is 493 t/s versus ~250 t/s upstream.
With MTP speculative decoding the fork reaches 54 t/s at 2k context and 29-35 t/s at 260k context. Prefill at 260k context is 123 t/s filling and 153 t/s for a question on a loaded context. Perplexity on the gate corpus at -c 4096 is 2.6101.
User reports Nemotron-3.5-Lightning-30B-A3B at 110 tok/s writing code on two Tesla P100 16GB cards.
Setup is llama.cpp b10970 with Q4_0 weights, F16 KV cache, tensor split across both cards, and the model's built-in MTP draft head at n-max 2. The same model reaches 95 tok/s on prose, 50 tok/s at 128k context, 39 tok/s at 256k, and 16 tok/s at 1M tokens.
The study covers 545 speed measurements of 29 models from 2B to 122B parameters, all weights and KV cache in VRAM with no system RAM offload. A 119B MoE model runs 39 tok/s against 4.3 tok/s for a 70B dense model on the same cards. Tensor split makes dense models from 8B up 21-44% faster. A single P100 throttles to 906 MHz and loses 25% under sustained load, while two cards share the heat and lose 5.5%.
User reports Qwen3.8 27B at 50-60 t/s generation and 350 t/s prefill on 2x Tesla P100 16GB with a custom llama.cpp fork.
Setup is llama.cpp with Q6_K quant, 262144 context, batch 32768, ubatch 1024, and MTP speculative decoding.
At 260k context, generation is 30-35 t/s and prefill is 110 t/s. GPUs are capped at 175W/250W each and run at 79C with minor thermal throttling; user estimates 5-10% higher numbers with better cooling.
User reports Qwen3.6 35B A3B at 54-60 t/s generation (66-72 t/s on code) on a single Tesla P100 16GB, up from 30-35 t/s prose on an RX 6600 XT.
Setup is llama.cpp with shinbunbun patches, UD_Q4_K_XL quant, 16-bit KV cache, MTP speculative decoding, and --n-cpu-moe 22, running 32k context.
Prefill is 600 t/s at 0 ctx dropping to 440-500 t/s by 10k. The user notes the P100 is underrated for the price and has a second card coming for full offload.
User reports Qwen3.6-35B-A3B at ~85 t/s generation and ~300 t/s prompt processing on 2x Tesla P100.
Setup is llama.cpp with the Q4_K_XL GGUF, MTP speculative decoding, and community P100 patches that added about 50% decode; a single P100 on Q2_K_XL reached ~76 t/s generation and ~250 t/s prompt processing.
Card count barely affects single-stream speed, and PCIe lane width (x16/x16, x16/x8, x8/x8) made no difference. The user corrects an earlier 3-card result that was invalidated by stuck 405 MHz core clocks.
User reports almost 70 t/s with Qwen3.6-35B-A3B Q4_K_XL on a budget build using three Nvidia Tesla P100 16GB GPUs.
The GPUs cost $80 each and are split across three nodes to manage thermals; the motherboard required a patched BIOS to enable Above 4G Decoding.
User is still testing and optimizing, and has published a GitHub repository for the build.
User reports Qwen3.8 27B at 40+ t/s on two Tesla P100s, with a roughly 30 t/s average across long contexts and up to 55 t/s at 0 context.
Setup is a custom llama.cpp fork with P100 kernel optimizations, Q6_K quant, both cards capped at 175W and communicating over PCIe gen 3.
Speeds vary by about ±2 t/s; the user notes fp16 math saves 40% or more on prefill with negligible accuracy loss.