Local model performance on your hardware

generation:

18.6-21 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next UD-Q4_K_XL (104 GB GGUF) at 18.6 to 21 tok/s, with peaks of 24.8, split across an RTX 3090 desktop and an RTX 4060 Laptop over a direct 1 Gbps Ethernet cable. The split is by layers, not tensor parallelism: the laptop runs layers 0 to 13 and the desktop 14 to 47 plus the output head, with each machine filling VRAM, then pinned RAM, then a RAM cache, then NVMe. The desktop alone gets 13 tok/s on the same model. A 9K-token prompt with nothing cached took about 103 s with the laptop in the pool against about 41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. With IQ3_XXS (71 GB), which fits the 3090, the desktop alone reaches 53 to 79 tok/s and the laptop does not help.

generation:

50-70 tokens/s

User reports running Qwen3.8 9B on an RTX 3060 8GB with 16GB of system RAM, getting 50-70 tok/s but only 20-30k tokens of usable context. No engine, quant or KV cache setting is stated. User finds the speed acceptable but the memory too tight for practical work, and asks whether a second GPU or a dedicated cheap machine is the better upgrade for coding and research use.

generation:

2-3 tokens/s

User reports running DeepSeek 4.1 on a single RTX 3090 in a 32GB machine at 2-3 t/s. No engine, quant or context length is stated; the user notes it is only a 2-bit quant and that the run is not really useful at the moment. The user plans to add 128GB of RAM, a spare RTX 3060 and possibly another 3090.

generation:

20.0 tokens/s

User reports running DeepSeek V4.1 Flash at about 20 tok/s and calls it an insanely great model. No hardware, engine, quant or context length is stated for the run. The user asks how others use frontier models to optimize their local inference setups and plans to try the same with Fable 5.2.

generation:

12.0 tokens/s

quant:

Q4 (GGUF)

User asks what model fits a 16GB DDR3 system (i7 3770, 1600MT dual-channel, 5400RPM HDD) and how to run it faster as an agent. They ran LFM3 1.7B Q4 (about 1GB) in llama.cpp at around 12 tok/s, slowing as the chat grows, and Nanbeige 4.2 3B starting near 3.5 tok/s and dropping below 1 tok/s; they also tried Qwen 3 8B and found the models poor as agents and wasteful of tokens. No GPU is named, so the runs are CPU-only; the user notes their RAM is theoretically 25.6GB/s and asks about 2-bit Qwen 4 27B and upcoming Nanbeige 4.5.

generation:

34.1 tokens/s

User reports Qwen3.8-Flash-Next at 34.1 tok/s decode on a Ryzen AI Max+ 395 128GB mini PC. Setup is Gufo at 16K context, with the model weights taking around 103.7 GiB. With MTP decode rises to 53.7 tok/s, and adding the NPU gives 34.2 tok/s decode with about 10% faster prefill. The user questions whether the largest model is the best use of the memory.

generation:

30-62 tokens/s

prompt processing (prefill):

915-1800 tokens/s

quant:

STQ-1.75bpw

User reports DeepSeek V4.1 Flash running at 1M-token context on a 96 GB Apple M5 Ultra, with decode of ~30–62 tok/s and prefill of ~915–1,800 tok/s. Setup is Sushi 1.5.1 on MLX 0.32.3 with a 1.75bpw STQ pack (~310 GB on disk), streaming routed experts from SSD with an 88 GB SSD budget; the 189 GB Engram tables are read from SSD and never loaded into GPU memory. First token takes 1.2–70 s from 1k to 128k context, and ~17 min cold at 1M where decode falls to ~23 tok/s. User calls it clearly smarter than the Qwen 3.8 Flash Next 2.75 quant they run daily, but too slow for daily chat or coding loops; the Qwen comparison is from a different day and protocol.

generation:

95.0 tokens/s

prompt processing (prefill):

1180.0 tokens/s

quant:

oQ4e

User reports Qwen3.8-27B at about 1180 t/s prompt processing and 95 t/s decode on an M5 Ultra Studio with 96GB of RAM. The model runs with MTP speculation and an oQ4e quant; SSD cache is disabled and the hot cache is set to 4GB with a 10% hit rate. System RAM sits just over 50% during heavy work, and the user asks whether a larger hot cache would be reasonable.

generation:

4.5 tokens/s

prompt processing (prefill):

25.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Qwen3-Coder-30B-A3B at 25 t/s prompt processing and 4.5 t/s generation on a GTX 1060 6GB, with an 11K-token initial context including MCP tools. Setup is llama.cpp (llama serve) with Q4_K_M weights, q8_0 KV cache, 131072 context, -cmoe CPU offload and 10 threads on a Xeon E5-2666 v3 with 16GB RAM; the run uses 4.5GB VRAM and up to 13GB system RAM. User finds the model struggles with short prompts and asks for alternatives good at agentic coding and tool calling.

generation:

87.4 tokens/s

prompt processing (prefill):

872.0 tokens/s

quant:

UD-Q4_K_M (GGUF)

kv:

q8_0

User reports Qwen3.8-27B at 87.4 t/s decode and 872 t/s prefill at 16K context on one RX 7900 XTX, using a fork of llama.cpp tuned for RDNA3 Vulkan. Setup is the fork's llama.cpp with UD-Q4_K_M GGUF, q8_0 KV cache, flash attention and a DFlash2 draft model for speculative decoding; the card is 24 GB at stock 291 W on Mesa RADV. At 128K context the fork gives 69.1 t/s decode and 657 t/s prefill, against 26.3 t/s and 442 t/s upstream. The fork reaches up to 220K context where upstream fits about 192K, and DFlash2 no longer hits vk::ErrorDeviceLost past ~60K tokens. Only the 7900 XTX was tested.

generation:

49.9 tokens/s

quant:

IQ2_XXS (GGUF)

User reports DeepSeek V4.1 Flash at 49.9 tok/s decode on one RTX PRO 4500 32 GB, median of three runs. Setup is the author's FLE engine fork with a 2-bit IQ2_XXS GGUF (232 GiB on disk), experts in 120 GB of system RAM with the busiest on the GPU, and DeepSeek's DSpark draft model proposing five tokens. Without the draft model decode is 37.2 tok/s. Against the original weights the 2-bit quant has KL divergence 0.248 and top-1 agreement 83.5 %. Speed is bound by DDR5-4800 memory across four sticks. With thinking on at 16k context, a budget of 8000 tokens lets both test tasks finish, while at 64k without a budget both thought 60,000 tokens and never answered. The author also notes Qwen3.8-Flash-Next 125B runs from 16 GB at about 15 tok/s and 91 tok/s on the 32 GB card.

generation:

8.3 tokens/s

quant:

3-bit (GGUF)

User benchmarks Qwen3.8-Flash-Next at a 3-bit quant on an AMD EPYC 7532 with 128 GB DDR4-3200 in eight channels, CPU only, reaching 8.33 tok/s at 32 threads. Setup is llama.cpp with the roughly 180B-parameter mixture (about 6B active per token) held entirely in system RAM, beside the machine's normal services. The same model ran 6.51 tok/s at 8 threads and 7.74 tok/s at 16 threads. The post also reports other models on three more machines, including gemma4:26b at 12.66 to 24.68 tok/s and llama3.3:70b at 2.70 tok/s on the EPYC.

generation:

20-43 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

q8_0

User reports running Qwen3.8-27B (UD-IQ3_XXS GGUF) through TensorSharp on an M5 Pro MacBook Pro with 48 GB unified memory, decoding at approximately 20-43 tokens/second depending on draft acceptance. Setup is TensorSharp with DFlash2 speculative decoding (7 draft tokens per step), q8_0 KV cache and a 65,536-token context, driving the Pi coding agent over an OpenAI-compatible endpoint. First token took about 13-14 seconds from a cold ~4K prompt and 0.45-3.5 seconds on later rounds; the SDK run served 41,962 of 48,166 prompt tokens (87%) from the prefix cache. End-to-end bug-fix runs passed 5/5 tests in 88 seconds (CLI, 10 model requests) and 83.4 seconds (SDK, 8 requests). The CUDA/NInfer preset was not validated with Pi.

generation:

20-43 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

q8_0

User reports running Qwen3.8-27B (UD-IQ3_XXS) through TensorSharp as the backend for the Pi Coding Agent on an M5 Pro MacBook Pro with 48 GB unified memory, with decode throughput ranging from approximately 20 to 43 tokens/second. Setup is TensorSharp with a Q8_0 KV cache, a 65,536-token context limit, and DFlash2 speculative decoding using a Qwen3.8-27B-DFlash2-Q4_K_M draft model at seven draft tokens per step. The SDK run used 48,166 prompt tokens with 41,962 served from prefix cache (about 87%) and 2,117 generated tokens; initial prefill was about 3.9K-4.1K tokens with TTFT around 13-14 seconds, later requests 0.45-3.5 seconds. The CLI and SDK E2E runs each passed 5/5 tests in 88 s and 83.4 s respectively. The user notes these are single-run observations, not a standardized benchmark.

generation:

20-43 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

q8_0

User reports running Qwen3.8-27B (UD-IQ3_XXS GGUF) with a DFlash2 Q4_K_M drafter through TensorSharp on an M5 Pro with 48 GB unified memory, decoding at roughly 20-43 tokens/s. Setup is TensorSharp's OpenAI-compatible server on the ggml_metal backend with a q8_0 KV cache, 65,536-token context and 7 draft tokens per step, driving the Pi Coding Agent. In the SDK end-to-end run, 41,962 of 48,166 prompt tokens came from the prefix cache and 2,117 tokens were generated; first-request TTFT was about 13-14 s and later rounds 0.45-3.5 s. When draft tokens were rejected more often, decoding fell back to around 21 tokens/s. The user notes this is one coding task on one machine, not a general benchmark.

prompt processing (prefill):

950.0 tokens/s

quant:

UD-IQ3_XXS (GGUF)

User reports libosmos v0.6.0 reaching up to 50 tok/s decode with MTP and about 950 tok/s prefill on Qwen3.8-Flash-Next at UD-IQ3_XXS. Optimizations were made in a low power profile to focus on energy per token, with better kernels, fewer launches and less idle time between work. The decode figure is given as a ceiling (up to 50 tok/s), and no GPU is named. DeepSeek numbers are unchanged and more SSD bound; a DeepSeek recipe at about 2.08 bpw is in progress.

generation:

14.6 tokens/s

prompt processing (prefill):

11.4 tokens/s

User benchmarks XeMaya, a new Intel Arc inference engine forked from XeStrata, on DeepSeek V4.1 Flash, GLM 5.3 Flash and Qwen3.8 Flash-Next. The regular DeepSeek V4.1 Flash model runs at 11.4 t/s prefill and 14.56 t/s generation on a B70 32GB plus B65 32GB. Setup is XeMaya over SYCL and Level Zero on an Ubuntu server with a B70 32GB, B65 32GB, B60 24GB, Ryzen 9950X and 128GB RAM. GLM 5.3 Flash reaches 152-161 t/s prefill and 17.4 t/s generation, around 20 t/s with MTP; DeepSeek V4.1 Flash Coder reaches 224.8 / 21.7 t/s after optimization. Using all three GPUs is slower than two for the regular DeepSeek model (11.2 / 11.58 t/s) because the B60 sits on a slower chipset link. The user notes the comparisons are not perfectly controlled, with different model files, prompts and GPU configurations.

generation:

62-68 tokens/s

prompt processing (prefill):

1900-2100 tokens/s

quant:

FP4

User reports Qwen3.8 Flash-Next 125B FP4 decoding at 62-68 tok/s on a Strix Halo 128GB mini PC (Ryzen AI Max+ PRO 495) under real agent traffic. Setup is the Halogen engine with a ~54GB FP4 checkpoint plus a ~48GB n-gram table and MTP drafter, 262k context and 4 concurrent slots; cold 28k prefill ran at ~1.9-2.1k tok/s and prompt cache hit rate was 98-99.6%. MiniMax-M3.1-Flash (428B, 93GB IQ1_S/IQ2_XXS quant) loaded but decoded at about 1 tok/s because 31% of experts streamed from NVMe; GLM-5.3-Flash ran at ~21 tok/s and Qwen3-30B-A3B at 70-93 tok/s. The model scored 73.3/100 on the user's Godot agent benchmark.

generation:

145-185 tokens/s

prompt processing (prefill):

3957.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8-Flash-Next on 2x RTX 3080 10GB with experts in system RAM, reaching 3,957 t/s prefill at a 200K cold prompt and decode of ~145-185 t/s with UD-IQ4_XS. Setup is the Speedy fork of Strata with UD-IQ4_XS, a 200K-token cold prompt, two pipeline windows and speculative decoding; the box is a Threadripper PRO 5975WX with 128 GB DDR4-3200. Decode varies with reply length and context, sitting at the top of the range for short replies and the bottom after a 200K prompt; a 200K cold prompt took 51 s. The 1x RTX 3080 table gives ~113-128 t/s decode and 1,792 t/s prefill at 200K for the same quant.

generation:

32.0 tokens/s

User reports Qwen3.8 27B at around 32 t/s on 2x RTX 3060 12GB with tensor split 1,1 after moving to an x470 SLI motherboard running both cards at x8. Previously, with one card at x16 and the other at x1 on a B550 board, the user got 11-16 t/s in layer split mode. The tensor-split run eventually gets stuck in loops, including repeated tool calls and endless question marks in thinking, which did not happen in layer split mode.