Local model performance on your hardware

generation:

21-25 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User reports Ornith-1.5-122B decoding at ~22 tok/s (21-25 across runs) on real coding output on a single RX 7900 XTX 24 GB, with 32 GB DDR4 and an i7-7700K. Setup is a patched Strata on ROCm with unsloth's UD-IQ2_XXS (42 GB), experts split between VRAM, an 18 GB RAM tier and the SSD, so weights stream from system RAM and disk. Prompt processing on an 8.9K-token prompt ran at 407 tok/s (22 s), and decode at 9K context fell to 12.2 tok/s. The model is Qwen3.5-MoE based with KLD vs BF16 of 0.1225.

generation:

37.8 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

turbo6

User reports Qwen3.8-27B at 37.8 tok/s on a Ryzen AI MAX+ 392 laptop with Radeon 8060S and 32 GB, against 29.6 tok/s for upstream gufo on the same machine. Setup is the user's tubo fork of gufo with IQ3_XXS weights and speculative decoding; the 512-token answer figure is a 10-run average. The apex quant with turbo6 6-bit KV cache reaches 45.1 tok/s on the same test and 53.3 tok/s over the HumanEval run, with 17.6 GiB RAM at 4k context and 128k context tested on 32 GB. Prefill is no faster than gufo up to about 1k tokens; splitting work onto the NPU gives 484 vs 440 tok/s at 4k and 472 vs 428 tok/s at 8k prompt tokens.

generation:

0.6 tokens/s

quant:

Q4_0 (GGUF)

User reports Qwen2.5-0.5B-Instruct at 0.63 tok/s on a 2006 Dell Latitude D620 laptop with no GPU. Setup is llama.cpp on CPU with Q4_0 GGUF, 512 context, 2 threads, 128 tokens generated; the machine has an Intel Core Duo T2300 and 2 GB DDR2 RAM. A single run took 203.2 s with 41.2 s to first token, and 1024 context crashed with OOM. The user does not recommend it and calls the results not generalizable.

generation:

27.0 tokens/s

User reports Qwen3.8 27B at about 27 t/s on dual RTX 5060 Ti 16GB with no speculation (no MTP, DFlash, n-gram or DSpark). Setup is vLLM nightly 0.32.1rc1 under WSL2 on Windows 11, with a Ryzen 5 3600 and 32GB DDR4; the second card sits in the first M.2 slot via an adapter. On a second machine (Ryzen 7 7800X3D, 64GB DDR5) the same run gave about 40 t/s, rising to 44.4 t/s after NCCL tuning flags; the user says NCCL was effectively disabled without them, since disabling it entirely reproduced the 27 t/s.

generation:

62.2 tokens/s

User reports DeepSeek V4.1-Flash decoding at a median 62.2 t/s on four DGX Spark (GX10) units in the original IKEA Kallax setup, with prefill of 4,255 t/s at 8K and 4,268 t/s at 250K. Decode uses speculative decoding, which swings about ±10 t/s between runs, so decode was left out of the cooling comparison. Each run is a single 8-minute sweep from 1K to 250K tokens with the model already loaded. Across six cooling setups the hottest GPU peaked at 82°C with no throttling; clocks stepped down about 3% from roughly 2525 MHz at 53°C to 2450 MHz at 82°C, and 250K prefill stayed between 4,225 and 4,353 t/s. The user's V3 open-faced air duct enclosure was the coolest setup at 65.1°C average under load.

generation:

84.2 tokens/s

quant:

NVFP4 (GGUF)

User benchmarks Qwen3.8 27B across six engines on one RTX 5090 32 GB, with llama.cpp reaching the highest plain single-request decode at 84.2 t/s. Setup is llama.cpp with an NVFP4 GGUF target and Q4_K_M DFlash2 drafter, 40,960-token context, greedy decoding with thinking off; the plain figure has no speculation. Other engines used different checkpoints and drafters, so results describe whole configurations. With DFlash2, NInfer led at 280.5 t/s single-request and 488 t/s aggregate across four requests; llama.cpp measured 212.9 t/s and 259 t/s aggregate. The author treats differences below roughly 5% as inconclusive.

generation:

55.0 tokens/s

prompt processing (prefill):

448.0 tokens/s

quant:

UD-IQ3_XXS (GGUF)

User reports Qwen3.8-27B at about 55 t/s generation and 448 t/s prefill on an M5 Pro MacBook Pro with 24GB unified memory. Setup is the Inco Splash engine (v1.4.1) with the UD-IQ3_XXS GGUF (10.9GB), capped at 18GB memory for around 120k context. User's own KLD measurements give 89.5% top-1 agreement with mean KLD 0.0542, and they say Safari and Discord still run comfortably alongside it.

generation:

20.1 tokens/s

User reports DeepSeek V4.1 Flash at 20.1 tok/s on an unnamed machine, served through Maya. The run read 223 new prompt tokens in 3.9 s with 65 reused, over a 1,155-token conversation; no quant, context length or KV cache setting is given. The user's point is behavioural: the model insisted it is Claude and refused to repeat its prompt verbatim, and the user asks whether Maya diverges from upstream in the scheduler or offload path.

generation:

80.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports running Qwen3.8 27B at an average generation speed near 80 t/s on a workstation with 2x RTX 4090 and 128 GB DDR5. Setup uses a K_M quant with a 128k context size; the harness is pi, dsh or qwen code, with dsh judged worse than the other two. The user compares it unfavourably with a free hosted DeepSeek Flash, which solved their data collection, figure and table preparation and LaTeX assembly tasks much faster and with better results, and asks whether local models are still worth it.

generation:

28.8 tokens/s

prompt processing (prefill):

757.0 tokens/s

quant:

4-bit (MLX)

User benchmarks Nanbeige4.3-33B (dense, 4-bit) on a MacBook Pro M5 Max with 128 GB, comparing mlx-lm, llama.cpp and Kapitsa at prompt lengths from 512 to 32K tokens. At 512 tokens mlx-lm prefills at 757 t/s and decodes at 28.8 t/s, llama.cpp at 691 t/s and 27.3 t/s, and Kapitsa at 693 t/s and 34.9 t/s. mlx-lm and Kapitsa read the same 4-bit MLX AWQ pack (group 64); llama.cpp reads Q4_K_M. Kapitsa's decode figures use KFlash, its tree-based speculative decoding with a small drafter, while mlx-lm and llama.cpp run plain decoding. On a 512-token Python coding task Kapitsa reaches 36.6 tok/s greedy and 32.3 tok/s sampled, against 27.1 and 26.7 for mlx-lm and 24.9 and 24.8 for llama.cpp. At 29.5K context mlx-lm has the fastest cold TTFT at 59.4 s, Kapitsa the fastest follow-up TTFT at 300 ms and the lowest peak memory at 28.1 GB. The author works on Kapitsa and notes the KV cache is large, 26 GB for 128K tokens.

prompt processing (prefill):

3700.0 tokens/s

quant:

fp8-iq4r

kv:

fp8_v4

User reports ~3700 t/s prefill with Qwen3.8 Flash-Next fp8-iq4r on a single RX 9070 XT, with experts placed in RAM and a ~57GB host pool. KV cache uses fp8_v4 and --max-num-batched-tokens is set to 4096, which gave the best prefill results. The card is power limited during prefill at over 300W, and PCIe5 x16 is nearly saturated; gen 4 drops prefill to ~2000 t/s. Prefill stays flat across context while decode decays 18-19% over 100k tokens, with a worst case of ~45 t/s at 240k context. Vision is fully working. The user has 96GB of DDR5 and is still working on decode speed.

generation:

40.0 tokens/s

quant:

Q8_0 (GGUF)

User reports Qwen3.8 27B at about 40 tokens/second on an RTX 5080 plus RTX 5070, with MTP disabled. Setup is Ollama with Q8_0 weights and 128k of context; the vision encoder and MTP were turned off to fit the split across the two cards. With MTP enabled at 64k context the user sees about 54 tokens/second. They ask whether Ollama's per-GPU buffer can be lowered, having had no luck with LLAMA_ARG_FIT_TARGET, or whether to switch to llama.cpp.

generation:

23.3 tokens/s

prompt processing (prefill):

1077.0 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4.1 Flash at 23.3 t/s decode and 1,077 t/s prefill on a Strix Halo 128GB (Bosgame M5), fully resident in GPU memory with no SSD streaming. Setup is the user's Gufo fork with IQ1_M GGUF weights, a 112 GiB backbone in GPU memory, IQ4_NL Engram tables (103 GiB) read from NVMe, and a 4.6 GiB Q2_K/Q4_K drafter. The reported figure is plain decode; DSpark speculative decoding with prompt lookup reaches 28.0 t/s on mixed text and 65.3 t/s on repetitive text at depth 0. At 32K depth decode is 22.9 t/s with 993 t/s prefill, and at 128K 21.7 t/s with 700 t/s prefill. Eight concurrent users total 78.5 t/s, and editing a source file through prompt lookup gives 67.6 t/s. The user warns the setup needs the whole machine, leaving about 1.75 GiB of RAM with the drafter loaded.

generation:

125.5 tokens/s

prompt processing (prefill):

2993.2 tokens/s

quant:

IQ2_XS (GGUF)

User reports Qwen3.8 Swift-1.5 Flash-Next at 125.5 tok/s generation and 2,993.2 tok/s prefill on an RTX 5090 Laptop 24 GB. Setup is PolyStrata with an IQ2_XS GGUF of about 63.5 GiB, configured for 131,072 tokens of context, with 128 GB of system RAM. The model is far larger than the card's 24 GB, so weights ran partly from system memory.

generation:

20.0 tokens/s

prompt processing (prefill):

1100.0 tokens/s

quant:

QAT

kv:

Q8

User reports Gemma 4 12B QAT at 20 t/s decode and 1100 t/s prefill on a single Intel Arc A770 16GB. Setup is llama.cpp with QAT weights, Q8 KV cache and a Q4 MTP head, giving 128k context with about 5GB VRAM headroom; the card is in a 13-year-old Haswell desktop with under 6GB RAM headroom. Four models run together at 14-15GB VRAM via llama-swap, with LiteLLM routing prompts through Laya to thinking and non-thinking aliases of the 12B. The user notes SYCL/llama.cpp lacks Alchemist optimizations, making the A770 the bottleneck, and that GPU-resident models still use 4-5GB of system RAM as compute buffer.

generation:

56.0 tokens/s

quant:

8-bit (MLX)

User reports Granite Switch 4.1 8B at 56 t/s chat on an M5 Max 128GB, with the 30B at 17 t/s. Setup is mlx-lm with a custom granite_switch package, 8-bit MLX weights, greedy decoding, one request at a time, medians of 3 runs. The 4-bit builds reach 98 t/s (8B) and 32 t/s (30B). Decision steps take 0.62 s on the 8B and 2.3 s on the 30B at a ~1.2K-token prompt. Accuracy on 500 labelled claims is 0.814 (8B 8-bit), 0.844 (30B 8-bit) and 0.840 (30B 4-bit), against 0.812 and 0.846 for the bf16 conversions. The user picks the 8B 8-bit for routing and classification.

generation:

18.0 tokens/s

User reports Qwen3-14B writing at 18 tok/s on a MacBook Pro M2 Pro with 16GB, where the model fits entirely in unified memory. Setup is llama.cpp with the Metal build, run through the MAFDET app, which reasons about the message while the user is still typing. The app cuts the wait from Enter to the first word from 32.7 s to 12.1 s on 6 word problems, with 6/6 accuracy both ways; on Qwen3-4B the mean/median wait drops from 18.1s/16.7s to 5.6s/4.1s over 24 problems. The same 14B on the user's RTX 2060 is split with CPU at 4.5-5 tok/s. Tested on one Mac only.

generation:

200.0 tokens/s

prompt processing (prefill):

8000.0 tokens/s

quant:

Q6

User reports Qwen3.8 Flash-Next at 200 tok/s decode and 8k tok/s prefill on a two-machine M5 Ultra 256GB (80-core) cluster. Setup is Tensorfold with Q6 weights, connected over two Thunderbolt 5 cables; the user copied Tensorfold's guide and had DeepSeek set it up. GLM-5.3 Flash on the same cluster gave 50 tok/s decode and 4.9k tok/s prefill, which the user found too slow. A previous Qwen3.8 27B Q4 run under NInfer gave around 1.4k prefill and 100-120 tok/s decode. The user runs Hermes as the harness and asks whether a lighter harness would speed things up.

generation:

39.6 tokens/s

prompt processing (prefill):

926.0 tokens/s

quant:

INT4

User reports Qwen3.8-27B at 39.6 tok/s decode on a Jetson AGX Thor, with 926 tok/s prompt processing and 23.6 tok/s end-to-end. Setup is TensorRT Edge-LLM 0.10.1 with INT4 weights, a FlashHead LM head and MTP speculative decoding, at batch 1 with 2,048-token prompts and 128 tokens out, averaged over 20 requests. Peak memory was 24.8 GiB and the weights download is about 20 GiB. Accuracy against the FP16 model: GSM8K 96.4 vs 96.1, IFEval 83.5 vs 81.9, MMLU-Pro 79.1 vs 79.5. The user notes Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M plus MTP, without stating prompt length or accuracy.

generation:

50.0 tokens/s

prompt processing (prefill):

1400.0 tokens/s

User reports Qwen3.8 Flash-Next on a Strix Halo 128GB (Framework Desktop) at 1400 t/s prefill and about 50 t/s generation, which they call good enough for their usage. Setup is Gufo, serving the model alongside embedding and STT/TTS models; the 50 t/s is described as an average. The user is weighing selling their RTX 5090 to scale out for three concurrent users with 2-3 sessions each at 200k context, and is leaning toward buying a second Strix Halo.