Local model performance on your hardware

reported speed:

15.8 tokens/s generation · 81.8 tokens/s prompt processing

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark compares MLX vs llama.cpp on Qwen3.8-27B. MLX: mlx-community/Qwen3.8-27B-4bit, ~16.1GB, prompt 81.76 tok/s, generation 15.81 tok/s, peak memory 16.39GB. llama.cpp: unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M, 15.32 GiB / 27.32B params, full Metal offload, Flash Attention enabled, prompt 99.61 ± 0.44 tok/s, generation 9.69 ± 0.34 tok/s. llama.cpp ~22% faster prompt processing, MLX ~63% faster generation.

reported speed:

17.3 tokens/s generation

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post describes a bug in oMLX 0.6.4 distributed clustering where the coordinator (M3 Ultra 256GB) fails to release RAM/GPU after a crash. Model: mlx-community/MiniMax-M3-4bit (236GB). Cluster: rank 0 Mac Studio M3 Ultra 256GB, rank 1 Mac Studio M2 Ultra 192GB. First completion: 17 tokens, prompt 7,693, 17.3 tok/s. Crash triggered by overlapping requests with pipeline_prefill_overlap and coalesced batching. Post-crash: ~116GB wired memory with no owning process, GPU pinned at 100%, requires reboot. Worker (M2 Ultra) released memory cleanly. User also mentions MiniMax-M3 crashes the cluster after first prompt. No subjective rating given.

reported speed:

18.3 tokens/s generation · 235.7 tokens/s prompt processing

quant:

oQ4e (MLX)

kv:

8.0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports slow performance with Qwen3.8-Flash-Next on M2 Max 96GB using oMLX engine. Benchmark results show pp1024/tg128: 124.1 pp TPS, 20.4 tg TPS; pp4096/tg128: 170.7 pp TPS, 16.7 tg TPS; pp8192/tg128: 214.3 pp TPS, 18.5 tg TPS; pp16384/tg128: 235.7 pp TPS, 18.3 tg TPS. Also tried oQ4e-fp16-mtp, oQ3-fp16-mtp, oQ3-MTP variants. fp16 variants even slower (~150pp, 8tg). User notes llama.cpp achieves 350-400 pp TPS. Model is MoE with 6B active parameters. Context length set to 32768. KV cache quant is turboquant_kv_bits 8.0. Engine is oMLX (custom MLX-based).

reported speed:

1.8 tokens/s generation

quant:

JANG

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark of MoE streaming PR for oMLX on M4 Pro 48GB. Four models tested: GLM-5.3-Flash-JANG-MTP (10.52 GiB after load, 14.68 GiB peak, 12.36s TTFT, 1.82 tok/s), Qwen3.8-JANG 4S (7.04 GiB, 11.19 GiB, 8.56s, 3.63 tok/s), Qwen3.8-JANG 4M (7.05 GiB, 11.11 GiB, 11.23s, 3.19 tok/s), DeepSeek-V4-Flash-0731-JANG (8.25 GiB, 16.87 GiB, 6.52s, 2.71 tok/s). Primary record uses GLM-5.3-Flash-JANG-MTP. Other models: Qwen3.8 (JANG 4S and 4M quants), DeepSeek V4 Flash (0731 variant, JANG quant). MoE streaming allows running larger MoE models with lower memory footprint at the cost of speed.

reported speed:

38.0 tokens/s generation

quant:

T5 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Runs on a custom oMLX fork with a Metal kernel for ternary experts; stock oMLX/mlx-lm won't load it. ~35.6 GiB resident: routed expert gate/up as ternary (Bonsai T5 packing, ~1.875 bpw), expert down projections as Q3, 53 GB n-gram table left on SSD as Q8 and mmapped per token. All low-bit tensors fitted with Unsloth's imatrix. Prefill 210-380 tok/s. 64K context confirmed (65,536-token prompt + 256 output at 30.7 tok/s, 42.3 GiB physical peak); 96K trips the prefill guard. Physical peak at 8K context ~41.5 GiB; swaps ~2 GiB once at load. Setup: oMLX memory guard 'safe' profile, limit 48 GB, one model, one request at a time. Quality vs Unsloth UD-Q4_K_XL (doesn't fit in 48 GB): KLD vs Q8_0 0.49 vs 0.036; MMLU 83.0% vs 89.7%; GSM8K 90.0% vs 92.0%; HumanEval 92.7% vs 95.7%. Known wart: sometimes ignores 'answer with just the letter' in Chinese.

reported speed:

45.0 tokens/s generation · 300.0 tokens/s prompt processing

quant:

IQ3_XXS (GGUF)

kv:

kvarn4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post by u/FerLuisxd (title mentions u/rss.app). Config for Qwen3.8-27B on RTX 5060 Ti 16GB with vision and 85K context, 1.5GB VRAM headroom. Uses beellama (llama.cpp fork) with MTP speculative decoding. Quant IQ3_XXS-mtp from ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF. KV cache quant kvarn4. Author notes mmproj could be moved to CPU for more VRAM.

reported speed:

15.0 tokens/s generation · 50.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post reports stock llama.cpp on AMD Ryzen AI 9 HX 470 96GB: Qwen 3.8 27B 10-12 t/s gen, Qwen 3.8 Flash same, ~50 t/s prompt. After using strix-halo-llamacpp fork: 15-18 t/s for Flash and 12-15 t/s for 27B. generationTps set to 15 (midpoint of 12-15 for 27B after fork).

reported speed:

21.5 tokens/s generation · 199.0 tokens/s prompt processing

quant:

Q5_K_M (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark run on Bosgame M5 (128GB/2TB) with Fedora 44, Vulkan driver 26.1.7. Used a llama.cpp fork (strix-halo-qwen4exp-b10685). MTP acceptance rate stays at 80% even at context >200K. llama-benchy results show prompt processing (pp) and generation (tg) at various context lengths: pp2048 460.38 t/s, tg512 34.86 t/s; pp4096 465.96 t/s, tg512 33.04 t/s; pp8192 437.14 t/s, tg512 31.18 t/s; pp16384 440.89 t/s, tg512 31.21 t/s; pp32768 407.48 t/s, tg512 29.65 t/s; pp65536 351.55 t/s, tg512 27.40 t/s; pp131072 265.54 t/s, tg512 23.87 t/s; pp200000 198.99 t/s, tg512 21.53 t/s. The reported promptTps and generationTps are from the pp200000 and tg512 at 200K context, respectively. The model is Qwen3.8-Flash-Next-Uncensored with Q5_K_M quant, and MTP draft model is shared-Q8_0.

reported speed:

36.8 tokens/s generation · 545.8 tokens/s prompt processing

quant:

UD-Q6_K_XL (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post also mentions a 27B 8-bit XL model that was too slow to be workable, but the benchmarked/served model is Qwen3-Coder-Next UD-Q6_K_XL. Bench run with llama-benchy 0.4.1 API latency mode at -c 262144. tg32 peak 37.94 t/s.

reported speed:

76.9 tokens/s generation

quant:

ROCmFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post benchmarks Nex-N2.5-mini (ROCmFP4 GGUF) on AMD Strix Halo against Qwen3.8-27B. Decode 76.9 tok/s vs Qwen3.8-27B's 14-34 tok/s. Terminal-Bench 2.1: 73.4 vs 73.0; WebArena: 63.4 vs 64.8; SWE-Bench: 43.8 vs 61.7. Weights at huggingface.co/julianmb/Nex-N2.5-mini-ROCmFP4-GGUF; engine HaloFPX (github.com/julianmb/halofpx).

reported speed:

22.0 tokens/s generation · 261.0 tokens/s prompt processing

quant:

ternary

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Millie is a series of compressed agentic models derived from Agents-A1 (a Qwen 3.5 35B-A3B finetune). The ternary expert model got 56% on SWE-bench Verified and runs at 22 tokens/s decode and 261 tokens/s pre-fill on an iPhone 17 Pro (12 GB RAM). The 2-bit expert version got 60%. The software targets Macs with 16 GB+ memory and Linux gaming PCs with 16 GB+ system RAM and as little as 4 GB VRAM. The harness is forked from OpenAI Codex. The user is seeking testers for AMD GPUs and small 4-8 GB cards.

reported speed:

1.9 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

27B model (~16GB) sharded across 4 devices via RAMDeck: old 12GB Windows laptop (primary, 3.4GB), Mini PC with RTX 3060 (20GB), Mac mini (3.7GB), Android phone (1GB). Model family not named. 1.92 tok/s at ~25ms latency. Author notes it's slower than their prior 13B run but emphasizes feasibility over speed.

reported speed:

34.3 tokens/s generation · 223.7 tokens/s prompt processing

quant:

UD-Q4_K_XL (GGUF)

kv:

f16

flash attention:

on

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Part 4 of a series on running Qwen3.8-Flash-Next on 2x RTX 3090 with dual Broadwell Xeon and DDR4. Focus is on prefill optimization by moving expert cache off GPU during prompt processing. Reports prefill improvements of 2.2-2.5x across 8k, 37k, and 119k contexts. Decode performance unchanged. Uses llama.cpp with custom branch flashnext-e06. Hardware changed from 4 DIMMs to 6x32GB DDR4-2133 ECC. Prefill numbers: 8k 223.7 t/s (was 99.9), 37k 212.6 t/s (was 88.1), 119k 206.5 t/s (was 81.3). Decode: 8k 34.3 t/s, 37k 41.2 t/s, 119k 33.9 t/s. Time to first token improved from 82s to 37s at 8k, 424s to 176s at 37k, 1461s to 575s at 119k. Quality screen showed no regression. MTP acceptance 0.79-0.83. Uses 150-slot expert cache, 261k context, f16 KV cache. Code available at github.com/Inovello/llama.cpp/tree/flashnext-e06.

reported speed:

11.4 tokens/s generation · 337.9 tokens/s prompt processing

quant:

IQ3_XXS (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Post title says IQ3 XXS but body also mentions 'IQ3_S - 3.4375 bpw'; model file is Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf. Generation t/s ranged 10-20 tps; detailed log shows tg=11.44 t/s with MTP speculative decoding (draft acceptance 0.4125). Prompt processing ~334-338 t/s. KV cache q4_0 for both K and V. Context 65536. More than 1GB VRAM left after loading.

reported speed:

20.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User runs Qwen3.8 27B split between RTX 3060 12GB and RX 9070 XT at 20 t/s, or on 780M iGPU with 5400MHz DDR5 at 5 t/s. Prefers 5 t/s for system usability. The 20 t/s is for the split configuration; the 5 t/s is for the iGPU. The post mentions two GPUs, but the primary benchmark is the split setup.

reported speed:

37.2 tokens/s generation

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Benchmark of multiple models on 16GB VRAM. Qwen3.6-35B-A3B-APEX-I-Quality achieved 96.7% HumanEval pass@1, 100% agentic success, 37.2 tok/s. Ornith-1.0-35B IQ4_NL also tested with 38 tok/s. Other models: Qwen3.8-27B variants, gemma-4-26B-A4B, KAT-Coder-V2.5-Dev, Qwen3.5-9B. Ornith 1.0 35B A3B won overall.

reported speed:

38.4 tokens/s generation

quant:

IQ3_XXS

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 38.4 t/s for Qwen3.8 27B on RTX 3060 with tuned llama.cpp, and 55.9 t/s for Qwen3.6 35B-A3B (MoE) with same tuning. Also mentions editing speeds ~188 t/s and context lengths 16K/12K for Ubuntu/WSL2. The post includes a link to a GitHub repo. The user expresses frustration about not reaching 50-60 t/s for the 27B model.

reported speed:

3.1 tokens/s generation

quant:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

CPU-only single-threaded inference on 7800x3D with 32GB DDR5. User reports ~3-3.2 tps for Qwen3.5 0.8B BF16. TTFT growth non-monotonic, tied to prompt token count.

reported speed:

30.0 tokens/s generation

quant:

1-bit

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Browser inference engine in WebGPU. Decode 25-30 tok/s in chat UI, raw 32 tok/s. Prompt processing 1489 tokens in ~25s. Context 3072 tokens on 6GB card. Model is natively 1-bit, 27B params in 3.8GB. Also mentions smaller tiers (Qwen3.5 0.8B, 2B, 4B) and vision tower.

reported speed:

28.0 tokens/s generation · 315.0 tokens/s prompt processing

quant:

IQ2_M

kv:

Q8

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

Model is Qwen3.8 Flash Next, quantized IQ2_M, fits in 52GB VRAM across 4 GPUs. Generation 27-29 t/s, prefill 290-340 t/s. Context offloaded at Q8. N-gram cache on SSD. User reports ~98% clean Finnish output, but safety guardrails cause freezes in grey areas.