3× RTX PRO 6000 64GB DDR5, RTX 5090 64GB DDR5, MacBook Pro 48GB · llama.cpp
generation:
6.4 tokens/s
prompt processing (prefill):
16.5 tokens/s
quant:
BF16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports full BF16 at 330 GiB with no quantization, running stock llama.cpp master b10680 over RPC on a LAN across 4 devices on 3 boxes: 94 GiB on the RTX PRO, 40 on the Mac's Metal, 30 on the 5090, plus the 5090's DDR5 as a 4th RPC device.
About 39 GiB still spills to local RTX PRO RAM, as the device pool cannot hold all 234 GiB of non-PLE weights.
The user notes a reported single 2×3090 box at 32 t/s on this model at UD-Q4_K_XL, which is 1.5× better byte-for-byte than what the user is achieving, so it is not purely a bytes problem. The next step is GGML_SCHED_DEBUG=2 plus GGML_RPC_STATS to see how much of the 155 ms token is RPC round-trips versus actual work.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 51.56 tokens per second over 5369 generated tokens with MTP on 2x R9700, writing a binary tree in javascript.
Setup is llama.cpp built in the ROCm 10 docker image with unsloth's Q8_K_XL GGUF, tensor split across both cards, 255000 context, f16 KV cache and MTP drafting up to 3 tokens, using 27GB on each card.
Draft acceptance is 0.72147. The user sees 37-50 tg/s generating text, spiking over 60tg/s when writing code.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports hybrid quantization to fit MTP and long context in 16 GB VRAM.
Setup uses MTP draft with spec-type draft-mtp and spec-draft-n-max 3, at 85,000 context, with q4_0 KV cache quantization for both cache types.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Next-Flash at ~240 t/s decode on a single RTX Pro 6000 Max-Q at 300W.
Setup is a patched sglang with the RadixArk/Qwen3.8-Flash-Next-NVFP4 checkpoint, using FP8 quantization of the lm head and layers, kernel tuning, and MTP config tuning.
The user notes a theoretical bandwidth limit of ~280 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B MTP quants on an Apple M5 Max.
The oQ4e-mtp quant reaches 36.6 t/s with a quality score of 86.8.
User calls oQ4e-mtp the best balance of speed and quality and recommends it as the sweet spot.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 7-8 t/s with 2x3060 and RAM offloading, but 4.5-5.5 t/s with 3x3060.
User asks why adding a GPU slows down generation, and notes there is no tensor parallelism in both chats.
User asks about enabling ngram or mlock.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks about adding RAM to run a larger quant of Qwen 3.8.
User mentions Q4_K_XS ran fine at 16 tps.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 Flash Next across VRAM tiers from 8 GB to 96 GB with llama.cpp b10666. CPU-only decode reaches 8.34 tok/s and full 96 GB decode reaches 109.07 tok/s. At 245K context, decode ranges from 14.89 tok/s on 24 GB to 21.61 tok/s on 96 GB.
The model activates 6B params per token (MoE). RAM-resident loading gives 1.87x prefill versus mmap. Non-unified KV reaches 92.0 tok/s total at concurrency 16.
A PLE table on CUDA causes a 55.6x decode slowdown, from 108.5 to 1.95 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-UD-IQ3_XXS.gguf at 40.68 t/s generation and 778.91 t/s prompt processing at 131,072 context, against 13.36 t/s and 137.87 t/s at 65,536 context.
Setup is llama-server with flash-attn, a q8_0 KV cache, Vulkan0, and speculative decoding with draft-mtp and ngram-mod.
User asks why the smaller context degrades performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks TensorSharp against llama.cpp on Qwen 3.8 Flash Next, reporting prompt processing and generation speeds for 1 GPU and 2 GPU layer split configurations.
The promptTps and generationTps fields capture the 1 GPU pp512 and tg64 values respectively.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen 3.8 27B Q4_K_M running on an RTX 4070 Ti Super 12GB and a MacBook M5 Air over RPC with llama.cpp, completing a coding game clone task in 117 minutes.
Setup uses llama.cpp over RPC between the two machines. LM Studio is also mentioned.
The 117 minutes is the total time for the task, not a throughput figure.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a custom kernel achieving fp32 quality with 0.99997 similarity at int8.
Setup includes optimized prefill and decode speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a fork optimized for AMD gfx906 (MI50/MI60/Radeon VII) reaching 379.2 t/s prefill versus 332.3 t/s (+14.1%), 252.6 t/s versus 231.1 t/s on a 120k-context fill (+9.3%), and 13.6 t/s versus 13.5 t/s deep-context TG.
DFlash acceptance is 0.691.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports prompt processing around 4000 t/s and decode above 80 t/s on a single stream on a DIY CUDA box with two CMP 170HX cards.
Setup runs Qwen Flash Next, with Qwen 27B and Gemma 26B also mentioned.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 280 t/s decode on dual R9700s.
Setup is MXFP4 quantization. Prompt processing at 128k context runs at 3831 t/s.
The user notes the performance came from collaboration.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 84 t/s with 4 streams at 196K context on a model split across an iGPU (Vulkan) and an RTX 3090 Ti (CUDA).
The model has 512 experts per layer, 36 layers of gated DeltaNet, 12 layers of top-k sparse attention, a 26.8 GiB n-gram table, and a built-in MTP draft head, at 103.69 GiB.
Baseline on the iGPU alone is 22.2 tok/s. HumanEval+ scores 155/164, against 156/164 for Qwen3.8-27B on dual 3090 vLLM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 20 t/s with Qwen3.8-27b-ud-q4-k-xl on an MI100, and later 40 t/s with a dflash2 model.
User also reports Qwen3.6-35b-a3b-ud-q5_k_m at 60 t/s max at 64k context.
User is frustrated with the performance compared to expectations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports a first tuning attempt with selective FFN offload to CPU improving speed over whole-layer offload.
True Q4_K_M runs at 13.2 tok/s near 50-61K context.
MTP is worse at deep context. Coding quality improved over IQ3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Hermes with Qwen 3.8 27B fp8 at 262k context on a quad R9700 setup, with prefill at 17636 t/s and TG at 36.6 t/s.
Setup is vLLM-Radiance, with one GPU on PCIe 4x8 and three on 4x16.
Best TG reached 106 t/s with 80% MTP 4 acceptance. vLLM-Radiance works well with the quad setup despite official dual support.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Next at ~10.1 t/s on an RTX 3090 capped at 250W, with a Xeon e5 2690v4 and 96GB DDR4 RAM.
Setup replaces the N-gram layer with Q8 precision on top of an IQ4_XS quant.
The user reports no meaningful speed penalty from the change.