llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 3× RTX PRO 6000 64GB DDR5, RTX 5090 64GB DDR5, MacBook Pro 48GB · llama.cpp

Sep 7, 2026
Throughput
6.4 t/s gen · 16.5 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
BF16

Summary

User reports full BF16 at 330 GiB with no quantization, running stock llama.cpp master b10680 over RPC on a LAN across 4 devices on 3 boxes: 94 GiB on the RTX PRO, 40 on the Mac's Metal, 30 on the 5090, plus the 5090's DDR5 as a 4th RPC device. About 39 GiB still spills to local RTX PRO RAM, as the device pool cannot hold all 234 GiB of non-PLE weights. The user notes a reported single 2×3090 box at 32 t/s on this model at UD-Q4_K_XL, which is 1.5× better byte-for-byte than what the user is achieving, so it is not purely a bytes problem. The next step is GGML_SCHED_DEBUG=2 plus GGML_RPC_STATS to see how much of the 155 ms token is RPC round-trips versus actual work.