Local model performance on your hardware

generation:

27.0 tokens/s

User reports Qwen3.8 27B at about 27 t/s on dual RTX 5060 Ti 16GB with no speculation (no MTP, DFlash, n-gram or DSpark). Setup is vLLM nightly 0.32.1rc1 under WSL2 on Windows 11, with a Ryzen 5 3600 and 32GB DDR4; the second card sits in the first M.2 slot via an adapter. On a second machine (Ryzen 7 7800X3D, 64GB DDR5) the same run gave about 40 t/s, rising to 44.4 t/s after NCCL tuning flags; the user says NCCL was effectively disabled without them, since disabling it entirely reproduced the 27 t/s.

generation:

62.2 tokens/s

User reports DeepSeek V4.1-Flash decoding at a median 62.2 t/s on four DGX Spark (GX10) units in the original IKEA Kallax setup, with prefill of 4,255 t/s at 8K and 4,268 t/s at 250K. Decode uses speculative decoding, which swings about ±10 t/s between runs, so decode was left out of the cooling comparison. Each run is a single 8-minute sweep from 1K to 250K tokens with the model already loaded. Across six cooling setups the hottest GPU peaked at 82°C with no throttling; clocks stepped down about 3% from roughly 2525 MHz at 53°C to 2450 MHz at 82°C, and 250K prefill stayed between 4,225 and 4,353 t/s. The user's V3 open-faced air duct enclosure was the coolest setup at 65.1°C average under load.

generation:

84.2 tokens/s

quant:

NVFP4 (GGUF)

User benchmarks Qwen3.8 27B across six engines on one RTX 5090 32 GB, with llama.cpp reaching the highest plain single-request decode at 84.2 t/s. Setup is llama.cpp with an NVFP4 GGUF target and Q4_K_M DFlash2 drafter, 40,960-token context, greedy decoding with thinking off; the plain figure has no speculation. Other engines used different checkpoints and drafters, so results describe whole configurations. With DFlash2, NInfer led at 280.5 t/s single-request and 488 t/s aggregate across four requests; llama.cpp measured 212.9 t/s and 259 t/s aggregate. The author treats differences below roughly 5% as inconclusive.

generation:

55.0 tokens/s

prompt processing (prefill):

448.0 tokens/s

quant:

UD-IQ3_XXS (GGUF)

User reports Qwen3.8-27B at about 55 t/s generation and 448 t/s prefill on an M5 Pro MacBook Pro with 24GB unified memory. Setup is the Inco Splash engine (v1.4.1) with the UD-IQ3_XXS GGUF (10.9GB), capped at 18GB memory for around 120k context. User's own KLD measurements give 89.5% top-1 agreement with mean KLD 0.0542, and they say Safari and Discord still run comfortably alongside it.

generation:

80.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports running Qwen3.8 27B at an average generation speed near 80 t/s on a workstation with 2x RTX 4090 and 128 GB DDR5. Setup uses a K_M quant with a 128k context size; the harness is pi, dsh or qwen code, with dsh judged worse than the other two. The user compares it unfavourably with a free hosted DeepSeek Flash, which solved their data collection, figure and table preparation and LaTeX assembly tasks much faster and with better results, and asks whether local models are still worth it.

generation:

28.8 tokens/s

prompt processing (prefill):

757.0 tokens/s

quant:

4-bit (MLX)

User benchmarks Nanbeige4.3-33B (dense, 4-bit) on a MacBook Pro M5 Max with 128 GB, comparing mlx-lm, llama.cpp and Kapitsa at prompt lengths from 512 to 32K tokens. At 512 tokens mlx-lm prefills at 757 t/s and decodes at 28.8 t/s, llama.cpp at 691 t/s and 27.3 t/s, and Kapitsa at 693 t/s and 34.9 t/s. mlx-lm and Kapitsa read the same 4-bit MLX AWQ pack (group 64); llama.cpp reads Q4_K_M. Kapitsa's decode figures use KFlash, its tree-based speculative decoding with a small drafter, while mlx-lm and llama.cpp run plain decoding. On a 512-token Python coding task Kapitsa reaches 36.6 tok/s greedy and 32.3 tok/s sampled, against 27.1 and 26.7 for mlx-lm and 24.9 and 24.8 for llama.cpp. At 29.5K context mlx-lm has the fastest cold TTFT at 59.4 s, Kapitsa the fastest follow-up TTFT at 300 ms and the lowest peak memory at 28.1 GB. The author works on Kapitsa and notes the KV cache is large, 26 GB for 128K tokens.

prompt processing (prefill):

3700.0 tokens/s

quant:

fp8-iq4r

kv:

fp8_v4

User reports ~3700 t/s prefill with Qwen3.8 Flash-Next fp8-iq4r on a single RX 9070 XT, with experts placed in RAM and a ~57GB host pool. KV cache uses fp8_v4 and --max-num-batched-tokens is set to 4096, which gave the best prefill results. The card is power limited during prefill at over 300W, and PCIe5 x16 is nearly saturated; gen 4 drops prefill to ~2000 t/s. Prefill stays flat across context while decode decays 18-19% over 100k tokens, with a worst case of ~45 t/s at 240k context. Vision is fully working. The user has 96GB of DDR5 and is still working on decode speed.

generation:

40.0 tokens/s

quant:

Q8_0 (GGUF)

User reports Qwen3.8 27B at about 40 tokens/second on an RTX 5080 plus RTX 5070, with MTP disabled. Setup is Ollama with Q8_0 weights and 128k of context; the vision encoder and MTP were turned off to fit the split across the two cards. With MTP enabled at 64k context the user sees about 54 tokens/second. They ask whether Ollama's per-GPU buffer can be lowered, having had no luck with LLAMA_ARG_FIT_TARGET, or whether to switch to llama.cpp.

generation:

23.3 tokens/s

prompt processing (prefill):

1077.0 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4.1 Flash at 23.3 t/s decode and 1,077 t/s prefill on a Strix Halo 128GB (Bosgame M5), fully resident in GPU memory with no SSD streaming. Setup is the user's Gufo fork with IQ1_M GGUF weights, a 112 GiB backbone in GPU memory, IQ4_NL Engram tables (103 GiB) read from NVMe, and a 4.6 GiB Q2_K/Q4_K drafter. The reported figure is plain decode; DSpark speculative decoding with prompt lookup reaches 28.0 t/s on mixed text and 65.3 t/s on repetitive text at depth 0. At 32K depth decode is 22.9 t/s with 993 t/s prefill, and at 128K 21.7 t/s with 700 t/s prefill. Eight concurrent users total 78.5 t/s, and editing a source file through prompt lookup gives 67.6 t/s. The user warns the setup needs the whole machine, leaving about 1.75 GiB of RAM with the drafter loaded.

generation:

125.5 tokens/s

prompt processing (prefill):

2993.2 tokens/s

quant:

IQ2_XS (GGUF)

User reports Qwen3.8 Swift-1.5 Flash-Next at 125.5 tok/s generation and 2,993.2 tok/s prefill on an RTX 5090 Laptop 24 GB. Setup is PolyStrata with an IQ2_XS GGUF of about 63.5 GiB, configured for 131,072 tokens of context, with 128 GB of system RAM. The model is far larger than the card's 24 GB, so weights ran partly from system memory.

generation:

20.0 tokens/s

prompt processing (prefill):

1100.0 tokens/s

quant:

QAT

kv:

Q8

User reports Gemma 4 12B QAT at 20 t/s decode and 1100 t/s prefill on a single Intel Arc A770 16GB. Setup is llama.cpp with QAT weights, Q8 KV cache and a Q4 MTP head, giving 128k context with about 5GB VRAM headroom; the card is in a 13-year-old Haswell desktop with under 6GB RAM headroom. Four models run together at 14-15GB VRAM via llama-swap, with LiteLLM routing prompts through Laya to thinking and non-thinking aliases of the 12B. The user notes SYCL/llama.cpp lacks Alchemist optimizations, making the A770 the bottleneck, and that GPU-resident models still use 4-5GB of system RAM as compute buffer.

generation:

56.0 tokens/s

quant:

8-bit (MLX)

User reports Granite Switch 4.1 8B at 56 t/s chat on an M5 Max 128GB, with the 30B at 17 t/s. Setup is mlx-lm with a custom granite_switch package, 8-bit MLX weights, greedy decoding, one request at a time, medians of 3 runs. The 4-bit builds reach 98 t/s (8B) and 32 t/s (30B). Decision steps take 0.62 s on the 8B and 2.3 s on the 30B at a ~1.2K-token prompt. Accuracy on 500 labelled claims is 0.814 (8B 8-bit), 0.844 (30B 8-bit) and 0.840 (30B 4-bit), against 0.812 and 0.846 for the bf16 conversions. The user picks the 8B 8-bit for routing and classification.

generation:

18.0 tokens/s

User reports Qwen3-14B writing at 18 tok/s on a MacBook Pro M2 Pro with 16GB, where the model fits entirely in unified memory. Setup is llama.cpp with the Metal build, run through the MAFDET app, which reasons about the message while the user is still typing. The app cuts the wait from Enter to the first word from 32.7 s to 12.1 s on 6 word problems, with 6/6 accuracy both ways; on Qwen3-4B the mean/median wait drops from 18.1s/16.7s to 5.6s/4.1s over 24 problems. The same 14B on the user's RTX 2060 is split with CPU at 4.5-5 tok/s. Tested on one Mac only.

generation:

200.0 tokens/s

prompt processing (prefill):

8000.0 tokens/s

quant:

Q6

User reports Qwen3.8 Flash-Next at 200 tok/s decode and 8k tok/s prefill on a two-machine M5 Ultra 256GB (80-core) cluster. Setup is Tensorfold with Q6 weights, connected over two Thunderbolt 5 cables; the user copied Tensorfold's guide and had DeepSeek set it up. GLM-5.3 Flash on the same cluster gave 50 tok/s decode and 4.9k tok/s prefill, which the user found too slow. A previous Qwen3.8 27B Q4 run under NInfer gave around 1.4k prefill and 100-120 tok/s decode. The user runs Hermes as the harness and asks whether a lighter harness would speed things up.

generation:

39.6 tokens/s

prompt processing (prefill):

926.0 tokens/s

quant:

INT4

User reports Qwen3.8-27B at 39.6 tok/s decode on a Jetson AGX Thor, with 926 tok/s prompt processing and 23.6 tok/s end-to-end. Setup is TensorRT Edge-LLM 0.10.1 with INT4 weights, a FlashHead LM head and MTP speculative decoding, at batch 1 with 2,048-token prompts and 128 tokens out, averaged over 20 requests. Peak memory was 24.8 GiB and the weights download is about 20 GiB. Accuracy against the FP16 model: GSM8K 96.4 vs 96.1, IFEval 83.5 vs 81.9, MMLU-Pro 79.1 vs 79.5. The user notes Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M plus MTP, without stating prompt length or accuracy.

generation:

50.0 tokens/s

prompt processing (prefill):

1400.0 tokens/s

User reports Qwen3.8 Flash-Next on a Strix Halo 128GB (Framework Desktop) at 1400 t/s prefill and about 50 t/s generation, which they call good enough for their usage. Setup is Gufo, serving the model alongside embedding and STT/TTS models; the 50 t/s is described as an average. The user is weighing selling their RTX 5090 to scale out for three concurrent users with 2-3 sessions each at 200k context, and is leaning toward buying a second Strix Halo.

generation:

106.0 tokens/s

quant:

EXL3 3.5bpw (EXL3)

User reports DeepSeek V4.1 Flash at 106 t/s per stream at concurrency 1 on four DGX Sparks, averaging 93 t/s aggregate across eight categories. Setup is TensorFold's Zig engine with an EXL3 3.5bpw pack, 1M context, a 2M-position KV pool, DSpark speculative drafts on and tensor parallel across all four GB10 nodes over RoCE v2. The sweep runs to 16 streams: per-stream decode falls to 20 t/s while aggregate rises to 250 t/s at C16. Coding reaches 139 t/s per stream at C1 and 343 t/s aggregate at C16; prose 81 and 224. Cold prefill was skipped, previously measured at 1.9-2.3K tok/s.

generation:

112.0 tokens/s

User reports the Flint model at around 112 tok/s on an M5 Pro 48GB Mac. Setup is Ollama, with the speed read from Ollama's own eval_count / eval_duration and the model unloaded after each run. The figure comes from the author's own speed-test tool, which streams the answer and counts reasoning tokens toward the speed.

generation:

57.2 tokens/s

prompt processing (prefill):

1637.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User benchmarks Qwen3.8-Flash-Next on a Strix Halo 128GB under Gufo v0.3.0 on Windows 11, reporting 57.16 t/s MTP decode and 1,637 t/s prefill at 8,192 tokens of context. Setup is Gufo with UD-Q4_K_XL weights plus an MTP Q8_0 draft, greedy decoding, two repetitions, one process at a time. MTP decode holds 87.71 t/s at 262,144 tokens with 1,464 t/s prefill; without MTP, autoregressive decode is 33.52 t/s at 8K and 27.35 t/s at 256K. MTP acceptance is 69% at 8K and 100% at longer contexts. The user compares v0.3.0 against v0.2.0 and notes the release build showed heavy disk activity.

generation:

130.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User compares an RTX 5080 desktop against a rented DGX Spark on Qwen3-8B Q4_K_M, where the PC decodes at 130.55 t/s versus 42.43 t/s on Spark at 512 input tokens, roughly three times faster. Both ran llama.cpp release b11889 with Q8 KV cache, Flash Attention and a 40,960-token context; the PC kept the whole model on its 16 GB card. Decode falls to 79.09 t/s on the PC and 25.64 t/s on Spark at 32,768 input tokens. With Qwen3-32B the PC needed CPU offload (40 GPU layers) and managed 4.92 t/s against Spark's 10.50 t/s. Aggregate batched throughput at 8 concurrent requests was 471.75 t/s on the PC and 209.02 t/s on Spark.