Local model performance on your hardware

reported speed:

15.8 tokens/s generation · 81.8 tokens/s prompt processing

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark compares MLX vs llama.cpp on Qwen3.8-27B. MLX: mlx-community/Qwen3.8-27B-4bit, ~16.1GB, prompt 81.76 tok/s, generation 15.81 tok/s, peak memory 16.39GB. llama.cpp: unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M, 15.32 GiB / 27.32B params, full Metal offload, Flash Attention enabled, prompt 99.61 ± 0.44 tok/s, generation 9.69 ± 0.34 tok/s. llama.cpp ~22% faster prompt processing, MLX ~63% faster generation.

reported speed:

17.3 tokens/s generation

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post describes a bug in oMLX 0.6.4 distributed clustering where the coordinator (M3 Ultra 256GB) fails to release RAM/GPU after a crash. Model: mlx-community/MiniMax-M3-4bit (236GB). Cluster: rank 0 Mac Studio M3 Ultra 256GB, rank 1 Mac Studio M2 Ultra 192GB. First completion: 17 tokens, prompt 7,693, 17.3 tok/s. Crash triggered by overlapping requests with pipeline_prefill_overlap and coalesced batching. Post-crash: ~116GB wired memory with no owning process, GPU pinned at 100%, requires reboot. Worker (M2 Ultra) released memory cleanly. User also mentions MiniMax-M3 crashes the cluster after first prompt. No subjective rating given.

reported speed:

18.3 tokens/s generation · 235.7 tokens/s prompt processing

quant:

oQ4e (MLX)

kv:

8.0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports slow performance with Qwen3.8-Flash-Next on M2 Max 96GB using oMLX engine. Benchmark results show pp1024/tg128: 124.1 pp TPS, 20.4 tg TPS; pp4096/tg128: 170.7 pp TPS, 16.7 tg TPS; pp8192/tg128: 214.3 pp TPS, 18.5 tg TPS; pp16384/tg128: 235.7 pp TPS, 18.3 tg TPS. Also tried oQ4e-fp16-mtp, oQ3-fp16-mtp, oQ3-MTP variants. fp16 variants even slower (~150pp, 8tg). User notes llama.cpp achieves 350-400 pp TPS. Model is MoE with 6B active parameters. Context length set to 32768. KV cache quant is turboquant_kv_bits 8.0. Engine is oMLX (custom MLX-based).

reported speed:

1.8 tokens/s generation

quant:

JANG

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark of MoE streaming PR for oMLX on M4 Pro 48GB. Four models tested: GLM-5.3-Flash-JANG-MTP (10.52 GiB after load, 14.68 GiB peak, 12.36s TTFT, 1.82 tok/s), Qwen3.8-JANG 4S (7.04 GiB, 11.19 GiB, 8.56s, 3.63 tok/s), Qwen3.8-JANG 4M (7.05 GiB, 11.11 GiB, 11.23s, 3.19 tok/s), DeepSeek-V4-Flash-0731-JANG (8.25 GiB, 16.87 GiB, 6.52s, 2.71 tok/s). Primary record uses GLM-5.3-Flash-JANG-MTP. Other models: Qwen3.8 (JANG 4S and 4M quants), DeepSeek V4 Flash (0731 variant, JANG quant). MoE streaming allows running larger MoE models with lower memory footprint at the cost of speed.

reported speed:

38.0 tokens/s generation

quant:

T5 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Runs on a custom oMLX fork with a Metal kernel for ternary experts; stock oMLX/mlx-lm won't load it. ~35.6 GiB resident: routed expert gate/up as ternary (Bonsai T5 packing, ~1.875 bpw), expert down projections as Q3, 53 GB n-gram table left on SSD as Q8 and mmapped per token. All low-bit tensors fitted with Unsloth's imatrix. Prefill 210-380 tok/s. 64K context confirmed (65,536-token prompt + 256 output at 30.7 tok/s, 42.3 GiB physical peak); 96K trips the prefill guard. Physical peak at 8K context ~41.5 GiB; swaps ~2 GiB once at load. Setup: oMLX memory guard 'safe' profile, limit 48 GB, one model, one request at a time. Quality vs Unsloth UD-Q4_K_XL (doesn't fit in 48 GB): KLD vs Q8_0 0.49 vs 0.036; MMLU 83.0% vs 89.7%; GSM8K 90.0% vs 92.0%; HumanEval 92.7% vs 95.7%. Known wart: sometimes ignores 'answer with just the letter' in Chinese.

reported speed:

45.0 tokens/s generation · 300.0 tokens/s prompt processing

quant:

IQ3_XXS (GGUF)

kv:

kvarn4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post by u/FerLuisxd (title mentions u/rss.app). Config for Qwen3.8-27B on RTX 5060 Ti 16GB with vision and 85K context, 1.5GB VRAM headroom. Uses beellama (llama.cpp fork) with MTP speculative decoding. Quant IQ3_XXS-mtp from ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF. KV cache quant kvarn4. Author notes mmproj could be moved to CPU for more VRAM.

reported speed:

21.5 tokens/s generation · 199.0 tokens/s prompt processing

quant:

Q5_K_M (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark run on Bosgame M5 (128GB/2TB) with Fedora 44, Vulkan driver 26.1.7. Used a llama.cpp fork (strix-halo-qwen4exp-b10685). MTP acceptance rate stays at 80% even at context >200K. llama-benchy results show prompt processing (pp) and generation (tg) at various context lengths: pp2048 460.38 t/s, tg512 34.86 t/s; pp4096 465.96 t/s, tg512 33.04 t/s; pp8192 437.14 t/s, tg512 31.18 t/s; pp16384 440.89 t/s, tg512 31.21 t/s; pp32768 407.48 t/s, tg512 29.65 t/s; pp65536 351.55 t/s, tg512 27.40 t/s; pp131072 265.54 t/s, tg512 23.87 t/s; pp200000 198.99 t/s, tg512 21.53 t/s. The reported promptTps and generationTps are from the pp200000 and tg512 at 200K context, respectively. The model is Qwen3.8-Flash-Next-Uncensored with Q5_K_M quant, and MTP draft model is shared-Q8_0.

reported speed:

36.8 tokens/s generation · 545.8 tokens/s prompt processing

quant:

UD-Q6_K_XL (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post also mentions a 27B 8-bit XL model that was too slow to be workable, but the benchmarked/served model is Qwen3-Coder-Next UD-Q6_K_XL. Bench run with llama-benchy 0.4.1 API latency mode at -c 262144. tg32 peak 37.94 t/s.

reported speed:

76.9 tokens/s generation

quant:

ROCmFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post benchmarks Nex-N2.5-mini (ROCmFP4 GGUF) on AMD Strix Halo against Qwen3.8-27B. Decode 76.9 tok/s vs Qwen3.8-27B's 14-34 tok/s. Terminal-Bench 2.1: 73.4 vs 73.0; WebArena: 63.4 vs 64.8; SWE-Bench: 43.8 vs 61.7. Weights at huggingface.co/julianmb/Nex-N2.5-mini-ROCmFP4-GGUF; engine HaloFPX (github.com/julianmb/halofpx).

reported speed:

1.9 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

27B model (~16GB) sharded across 4 devices via RAMDeck: old 12GB Windows laptop (primary, 3.4GB), Mini PC with RTX 3060 (20GB), Mac mini (3.7GB), Android phone (1GB). Model family not named. 1.92 tok/s at ~25ms latency. Author notes it's slower than their prior 13B run but emphasizes feasibility over speed.

reported speed:

34.3 tokens/s generation · 223.7 tokens/s prompt processing

quant:

UD-Q4_K_XL (GGUF)

kv:

f16

flash attention:

on

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Part 4 of a series on running Qwen3.8-Flash-Next on 2x RTX 3090 with dual Broadwell Xeon and DDR4. Focus is on prefill optimization by moving expert cache off GPU during prompt processing. Reports prefill improvements of 2.2-2.5x across 8k, 37k, and 119k contexts. Decode performance unchanged. Uses llama.cpp with custom branch flashnext-e06. Hardware changed from 4 DIMMs to 6x32GB DDR4-2133 ECC. Prefill numbers: 8k 223.7 t/s (was 99.9), 37k 212.6 t/s (was 88.1), 119k 206.5 t/s (was 81.3). Decode: 8k 34.3 t/s, 37k 41.2 t/s, 119k 33.9 t/s. Time to first token improved from 82s to 37s at 8k, 424s to 176s at 37k, 1461s to 575s at 119k. Quality screen showed no regression. MTP acceptance 0.79-0.83. Uses 150-slot expert cache, 261k context, f16 KV cache. Code available at github.com/Inovello/llama.cpp/tree/flashnext-e06.

reported speed:

11.4 tokens/s generation · 337.9 tokens/s prompt processing

quant:

IQ3_XXS (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post title says IQ3 XXS but body also mentions 'IQ3_S - 3.4375 bpw'; model file is Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf. Generation t/s ranged 10-20 tps; detailed log shows tg=11.44 t/s with MTP speculative decoding (draft acceptance 0.4125). Prompt processing ~334-338 t/s. KV cache q4_0 for both K and V. Context 65536. More than 1GB VRAM left after loading.

reported speed:

20.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User runs Qwen3.8 27B split between RTX 3060 12GB and RX 9070 XT at 20 t/s, or on 780M iGPU with 5400MHz DDR5 at 5 t/s. Prefers 5 t/s for system usability. The 20 t/s is for the split configuration; the 5 t/s is for the iGPU. The post mentions two GPUs, but the primary benchmark is the split setup.

reported speed:

37.2 tokens/s generation

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark of multiple models on 16GB VRAM. Qwen3.6-35B-A3B-APEX-I-Quality achieved 96.7% HumanEval pass@1, 100% agentic success, 37.2 tok/s. Ornith-1.0-35B IQ4_NL also tested with 38 tok/s. Other models: Qwen3.8-27B variants, gemma-4-26B-A4B, KAT-Coder-V2.5-Dev, Qwen3.5-9B. Ornith 1.0 35B A3B won overall.

reported speed:

38.4 tokens/s generation

quant:

IQ3_XXS

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 38.4 t/s for Qwen3.8 27B on RTX 3060 with tuned llama.cpp, and 55.9 t/s for Qwen3.6 35B-A3B (MoE) with same tuning. Also mentions editing speeds ~188 t/s and context lengths 16K/12K for Ubuntu/WSL2. The post includes a link to a GitHub repo. The user expresses frustration about not reaching 50-60 t/s for the 27B model.

reported speed:

28.0 tokens/s generation · 315.0 tokens/s prompt processing

quant:

IQ2_M

kv:

Q8

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Model is Qwen3.8 Flash Next, quantized IQ2_M, fits in 52GB VRAM across 4 GPUs. Generation 27-29 t/s, prefill 290-340 t/s. Context offloaded at Q8. N-gram cache on SSD. User reports ~98% clean Finnish output, but safety guardrails cause freezes in grey areas.

reported speed:

40.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports running 3 local sessions of GLM 4.7 at about 40 tok/s each on an AMD Strix Halo mini PC with 128GB unified RAM. The model family is inferred as GLM-4.7 from the post title and text. The user is positive about the setup for budget local AI.

reported speed:

38.8 tokens/s generation · 200.0 tokens/s prompt processing

quant:

Q4_K_M (AP)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.8 Flash/Next 125B MoE running on RTX 5090 with 64GB system RAM, AP quantized to Q4_K_M, using ik_llama.cpp. Decode ~38.8 tok/s, prefill ~200 tok/s, VRAM usage ~29.7 GiB. User notes it's slower and dependent on system RAM bandwidth/CPU offload. Also mentions Qwen 3.8 27B is extremely fast but weaker. Seeking advice on models and optimizations.

reported speed:

14.2 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Winner of a 25-task coding-agent benchmark on RTX 3060 12GB. Also tested Qwen3.8-27B (14.73 tps, 13/25), Nemotron 3.5 Lightning 30B-A3B (29.96 tps, 8/25), Tiel-Coder 35B-A3B (20.57 tps, 3/25 strict), Ternary-Bonsai-27B (59.21 tps, 0/25). KAT-Coder completed 21/25 tasks. User plans to tweak settings to reach 18 tps.

reported speed:

15.0 tokens/s generation

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 15 t/s generation and 100-200 t/s prompt processing at context 130k. Uses ROCm, mentions Vulkan similar speed. Settings include flash-attn, split-mode layer, fit on.