Local model performance on your hardware

generation:

36.5 tokens/s

prompt processing (prefill):

909.0 tokens/s

quant:

PTQ1_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Ternary-Bonsai-2-27B at PTQ1_0 running on a 32 GB RTX PRO 4500 Blackwell via llama.cpp, with 36.5 tok/s decode and 909 tok/s prefill at 32,768 tokens of context in a single stream. Setup uses the 6.58 GB ternary GGUF with a 131,072-token context window; the Q8_0 mmproj vision tower still works in the ternary GGUF. At 131,072 tokens of context the single stream drops to 14.1 tok/s decode, 630 tok/s prefill and 207 s time to first token. Two and four concurrent streams at 32,768 tokens give 12.4 and 4.6 tok/s decode per stream, and the concurrent cells at 131,072 tokens returned HTTP 500. The user notes decode collapses with concurrency, unlike EXL3 rows which hold a flat decode rate.

generation:

27.9 tokens/s

prompt processing (prefill):

150.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 125B MoE at 27.9 t/s token-weighted decode on an RTX PRO 4500 Blackwell 32GB with 64GB system RAM. Setup is a llama.cpp fork with UD-Q3_K_XL GGUF, 240,000 context, F16 KV cache, 16 of 48 routed-expert layers on GPU and 32 on CPU, disk-backed PLE, and MTP disabled. Real prompt chunks prefill around 150 t/s with a peak of 201. Decode stayed flat from 116K to 148K context (27.7, 28.5, 27.5 t/s across the three bands). The user notes the sparse-QSA CUDA patch is experimental and unreviewed, and that these are warm-cache workload numbers rather than a controlled benchmark.

generation:

45.2 tokens/s

prompt processing (prefill):

2022.5 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 27B IQ4_XS on an RTX Pro 4500 Blackwell 32GB with llama.cpp b9007, reaching 2022.54 t/s prompt processing and 45.19 t/s generation. The same card also runs Qwen3.6 35B-A3B MXFP4 at 5507.10 t/s prompt processing and 159.81 t/s generation, along with Gemma4 26B-A4B MXFP4, Ernie 4.5 21B-A3B MXFP4, Nemotron Cascade 2 30B-A3B MXFP4, Tesselate OmniCoder 9B Q8, Qwen3.5 4B Q4_K, Qwen3.5 9B UD Q4_K_XL and GLM 4.7 Flash MXFP4. Compared with an RTX 5090, the 5090 is 60-70% faster at 2-3x power. User is happy with the card for 24/7 use.