Local model performance on your hardware

generation:

118.0 tokens/s

quant:

Q2_K_XL (GGUF)

User reports Qwen3.6-35B-A3B at about 118 tok/s decode on a short prompt with their RustingStrata engine on an RTX 3060 12GB, versus 68 tok/s for llama.cpp. Setup is RustingStrata with Q2_K_XL GGUF weights (11.7 GiB) on a 3060 12GB, 7600X and 32 GB RAM, with experts that do not fit in VRAM running on the CPU. Decode is 113 tok/s after a 4K prompt and 92 tok/s at 32K; prefill is 2216 tok/s at 4K against about 2236 for llama.cpp, and 1800 tok/s at 32K. With MTP drafts decode reaches 160 tok/s on an easy prompt, with about +16% expected on normal text. Qwen3-Next-80B IQ1_S decodes at about 92 tok/s versus around 44 for llama.cpp.

generation:

27.0 tokens/s

prompt processing (prefill):

616.0 tokens/s

kv:

INT8

User reports Qwen3.6/3.8-27B decoding at 27.0 tok/s without MTP on a single modded 22 GB RTX 2080 Ti under NInfer v0.3.0 on Windows 11, with prefill of 616 tok/s on a 512-token prompt. Setup uses a 64k context with an INT8 KV cache; the weights take 15.4 GiB and the engine runs entirely in VRAM with no system RAM offloading. MTP speculative decoding is on by default. With MTP, decode reaches 47.5 tok/s on a real code prompt (26.3 without) and 28.8 tok/s on a real prose prompt (26.0 without); prefill on a 64k-token prompt is 458 tok/s. The engine supports only Qwen3.6/3.8-27B, and dropping RTX 5090, FP8, FP4 and 35B-A3B code paths cut the binaries from 1.56 GB to 0.28 GB.

generation:

122.5 tokens/s

prompt processing (prefill):

76.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User compares a Mac Studio M5 Max (40-core GPU, 64GB) against their current dual RTX 4060 Ti 16GB setup and asks what prefill and decode speeds Mac owners get. On the 2x 4060 Ti they report Qwen3.6-35B-A3B at UD-Q4_K_XL with 262k context reaching 76.7 t/s prefill and 122.5 t/s decode. A second row gives Qwen3.8-27B dense at UD-Q4_K_XL with 200k context at 55.5 t/s prefill and 34.3 t/s decode. No engine is named.

generation:

32.6 tokens/s

prompt processing (prefill):

3581.0 tokens/s

quant:

bf16 (BF16)

kv:

bf16

User reports Qwen3.6-35B-A3B at 3,581 t/s prefill and 32.6 t/s decode at 2k context on a Strix Halo 128GB APU, comparing stock vLLM against the same vLLM with the libstrix HIP kernel plugin. Setup is vLLM with bf16 weights and bf16 KV cache, single stream on one APU; the plugin installs via pip and only activates on ROCm gfx1151. At 64k the same run goes from 260 to 2,617 t/s prefill and 12.8 to 26.9 t/s decode; at 128k with int8 KV it reaches 1,751 t/s prefill and 26 t/s decode, and a 256k prompt prefills in 241 s versus 7,568 s stock. KLD against stock is 0.001-0.006 with 97-99% top-1 agreement, and the user notes decode gains are absent where the replaced kernels are not the bottleneck.

prompt processing (prefill):

3556.7 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks a llama.cpp CUDA PR that fuses MoE gate/up matmuls and GLU in MMQ, reporting prompt-processing throughput on DGX Spark (GB10) and RTX 5090. On DGX Spark with Qwen3.6-35B-A3B Q4_K_M, prompt processing rises from 3235.8 to 3556.7 t/s (+9.9%); the Qwen3-30B-A3B Q4_K_M row goes from 2911.4 to 3410.8 t/s (+17.2%). On RTX 5090 the same Qwen3.6-35B-A3B Q4_K_M row goes from 15000.3 to 15852.0 t/s (+5.7%), with UD-Q4_K_M at 15030.2 to 15947.8 t/s (+6.1%). No generation speeds are given.

generation:

11.0 tokens/s

prompt processing (prefill):

66.0 tokens/s

User reports Qwen3.6-35B reading at 66 tok/s and writing at about 11 tok/s on a 2013 Mac Pro with two AMD FirePro D700 cards and 64 GB RAM. Setup is llama.cpp with Vulkan/MoltenVK on original macOS 12, measured with llama-bench. User warns that an earlier Metal path produced wrong results and looked about 9x faster than the correct run, so a tok/s figure without a correctness check is meaningless.

generation:

15.0 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.6-35B-A3B at 15.0 tokens/s on an M5 MacBook Air with 16 GB RAM, keeping routed experts on SSD and loading only those selected by the MoE router. Setup uses MLX with a 4-bit checkpoint, around 1.7 GB of MLX memory, top-8 routing unchanged and no experts dropped; output is byte-identical to stock mlx_lm. The 15.0 t/s is a controlled benchmark, 72.6% faster than direct SSD streaming; with other apps open it is around 11.8 t/s, and time to first token for a short prompt is around 2.3 seconds.

generation:

40.0 tokens/s

kv:

8-bit

User reports Qwen 3.6 35B-A3B at about 40 tok/s on an M5 MacBook Pro with 32GB unified memory, against about 17 tok/s for a dense 27B at 4-bit. Setup uses oMLX with an SSD-backed cache server, 8-bit KV cache and context set to roughly 32K. The user says the 16GB dense model plus KV cache filled the 32GB machine and caused a shutdown; the MoE has been stable since.

generation:

62.0 tokens/s

prompt processing (prefill):

464.0 tokens/s

quant:

4-bit group-64 (MLX)

kv:

Q8

User benchmarks Qwen3.6-35B-A3B on an M1 Max 64GB, reporting 464 tok/s prefill and 62 tok/s decode at a 128K prompt with the MTPLX M1 fork. Setup is the MTPLX M1 fork v2.12.2-m1.11 with mlx-community 4-bit group-64 weights plus an MTP head, 8-bit paged affine KV cache, and speculative decoding on. Splash-M1 1.3.2-m1.1 reached 302 tok/s prefill and 41 tok/s decode at 128K, and oMLX 385 tok/s prefill and 33 tok/s decode. The fork was faster from 8K up on agent logs, 1.46x prefill and 1.5x decode at 128K, while Splash-M1 used less memory (23.5 GB vs 28.1 GB at 128K). The same post also compares Qwen3.8-27B, where the fork reached 132 tok/s prefill and 25.0 tok/s decode at 128K. Only the 2K, 32K and 128K agent-log cells were run three times; the rest are single runs.

generation:

125.5 tokens/s

prompt processing (prefill):

1874.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 125.47 t/s generation on an RX 7900 XTX 24GB. Setup is llama.cpp (Vulkan backend, v0.7.0) via LM Studio with Q4_K_M GGUF and no speculative decoding. Three consecutive runs gave 125.47, 125.07 and 126.47 t/s generation; prompt processing was 1874.71 t/s over 427 tokens, 1288.99 t/s over 311 tokens and 770.07 t/s over 157 tokens.

generation:

122.7 tokens/s

quant:

4bit (MLX)

User benchmarks Qwen3.6-35B-A3B at 122.7 tok/s on an M3 Max 36GB with Kapitsa 0.2.12, tying oMLX at 122.9 tok/s. Setup is Kapitsa with a 4-bit MLX build and MTP head plus DFlash2 drafter, sustained 256-token prose, median of 3 runs. Other engines on the same model: Rapid-MLX 110.3, Splash 97.0, MTPLX 79.9 tok/s. On dense Qwen3.8-27B-4bit Kapitsa leads at 41.3 tok/s, ahead of oMLX 34.1, Splash 33.0, MTPLX 31.6 and Rapid 26.2. Flint 40B-A2.5B reaches 134.2 tok/s on Kapitsa and 103.0 on Rapid. Draft acceptance was 75-78% dense and 81-85% MoE. No formal quality eval was run.

generation:

41.7 tokens/s

prompt processing (prefill):

1075.8 tokens/s

quant:

IQ4_XS (GGUF)

kv:

F16

User reports Qwen3.6-27B at 41.70 t/s decode and 1075.8 t/s prompt processing on an RX 7900 XTX. Setup is llama.cpp (ROCm/HIP backend, LTO and WMMA flash attention on) with IQ4_XS weights and F16 KV cache at 16K context, batch 1024 and ubatch 256. The model is a 14.1 GiB MTP build; the user calls this the best overall configuration and notes LTO gives a 5-15% improvement.

generation:

180.0 tokens/s

prompt processing (prefill):

9000-10000 tokens/s

quant:

AWQ 4-bit (AWQ)

User reports Qwen3.6-35B-A3B at about 180 tok/s single-stream decode on two RTX 4090s at roughly 50k context, dropping to about 160 tok/s at roughly 110k. Setup is sglang with a QuantTrio AWQ 4-bit quant, TP=2 over PCIe with no NVLink. Prefill is 9-10k tok/s, and 8 concurrent requests reach 985 tok/s total. The user runs it as a GitLab code-review agent and says repetition is the main problem with local models, with the model looping on itself ending about 7% of runs (56 of 847).

generation:

23.0 tokens/s

prompt processing (prefill):

29.6 tokens/s

quant:

UD-Q4_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 23.0 tok/s generation and 29.6 tok/s prompt reading on a single prompt, on a 16 GB Mac mini M4. Setup is stock llama.cpp b11715 with the Kraits plugin, Unsloth UD-Q4_K_M (22.1 GB) weights, 16K context, temperature 0 and thinking off. A 4-agent crew writing a TypeScript LRU cache ran at 16.0-23.8 tok/s writing and 37.8-47.3 tok/s reading at ~16K context, scoring 9/9; the ~8K run scored 8/8 and the ~1K run 6/8. The single prompt scored 8/8 in 43.7 s.

generation:

154.0 tokens/s

quant:

4bit (MLX)

User reports Qwen3.6 35B-A3B at 154 t/s on a 2024 MacBook Pro M4 Pro with 48GB, using the Kapitsa engine. Setup is Kapitsa, an open-source Metal engine for Apple Silicon, with 4-bit MLX weights and DFlash2 speculative decoding; part of the 27B's prefill runs on the Neural Engine. The same table gives Qwen3.8 27B at 63 t/s and Qwen3.5 4B at 179 t/s on the M4 Pro, and on an M5 Max 128GB Qwen3.6 35B reaches 281 t/s. The author says not to trust benchmarks and notes Kapitsa is not fastest on the M5 Max for Qwen3.8 27B (102 vs mlx-serve's 130) or Flash-Next.

generation:

46.1 tokens/s

prompt processing (prefill):

549.3 tokens/s

quant:

IQ3_M (GGUF)

kv:

q8_0

User reports 46.06 t/s generation and 549.27 t/s prompt processing. Source evidence (verbatim): Source: https://lemmy.world/post/48144487 # Qwen 3.6 27B running at 46 tok/s on an RX 9070 XT (llama.cpp + MTP Speculative Decoding is basically magic) Just got my hands on a new AMD Radeon RX 9070 XT (16GB) and wanted to share some inference numbers. I've been messing around with llama.cpp via their official ROCm Docker image, testing out Qwen 3.6 27B (Omnimerge-v4) in IQ3_M. Honestly, the performance you can squeeze out of a 27B model on a 16GB consumer card right now is blowing my mind. Here’s the breakdown: The Setup - GPU: AMD Radeon RX 9070 XT (RDNA4 / gfx1201) - 16 GB VRAM - CPU: AMD Ryzen 9 9950X3D - OS/Backend: Linux via Docker using ghcr.io/ggml-org/llama.cpp:server-rocm. (Props to the devs, it natively supports RDNA4 gfx1201 out of the box!) - Model: Qwen3.6-27B-Omnimerge-v4-IQ3_M.gguf (~13 GB) - Context: 16k Tweaks: - Set -np 1 since I'm just running it as a single-user chatbot in Open WebUI. - Slapped on 8-bit KV cache (--cache-type-k q8_0 --cache-type-v q8_0) to save about 50% VRAM. - Enabled MTP Speculative Decoding (--spec-type draft-mtp). The Numbers (512-token test) - Prompt Processing (TTFT): 549.27 tok/s (1220 tokens evaluated in ~2.2s). Latency: 1.82 ms/token. - Text Gen: 46.06 tok/s (512 tokens generated in ~11.1s). Latency: 21.71 ms/token. - MTP Stats: Draft acceptance rate was super high at 62.7% (333 drafts accepted / 531 generated). Aggregate speed (including prompt eval) hit roughly 48.97 tok/s. Memory Footprint - VRAM: Sitting at 14.46 GB out of 16 GB. This leaves about 1.5 GB of breathing room, which has been totally stable with zero OOM crashes so far. - System RAM: ~4.3 GB (mostly the host-side prompt cache helping speed up subsequent turns). RDNA4 is ready: The latest ROCm images and llama.cpp HIP libs support the 9070 XT natively. Didn't even need to mess with HSA_OVERRIDE_GFX_VERSION. KV Cache quantization is required: Pushing the KV cache to q8_0 is the only reason a 16k context window fits on a 16GB card alongside a 27B model. If anyone with a 16GB card is looking for the sweet spot, this has to be one of the best price-to-performance setups available right now. Let me know if you want my docker-compose.yml or run scripts... Edit: To see how much MTP actually helps, I ran the exact same 512-token prompt test with speculative decoding toggled off: Standard Inference (No Speculation): - Speed: 25.88 tokens/sec (Latency: 38.64 ms/token) - Prompt Eval: 239.46 tokens/sec MTP Speculative Decoding (MTP): - Speed: 46.06 tokens/sec (Latency: 21.71 ms/token) - Prompt Eval: 549.27 tokens/sec - Draft Acceptance Rate: 62.7% For more info, here are my logs ::: spoiler Logs ``` ROCm GPU and System Info: device_info: - ROCm0 : AMD Radeon RX 9070 XT (16304 MiB, 16200 MiB free) - ROCm1 : AMD Ryzen 9 9950X3D 16-Core Processor (15589 MiB, 25012 MiB free) - CPU : AMD Ryzen 9 9950X3D 16-Core Processor (31178 MiB, 31178 MiB free) I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 Context Initialization (16k single-slot & 8-bit KV Cache): I srv load_model: [spec] estimated memory usage of MTP context is 178.02 MiB W llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized I srv load_model: initializing slots, n_slots = 1 I slot load_model: id 0 | task -1 | new slot, n_ctx = 16384 I srv load_model: prompt cache is enabled, size limit: 8192 MiB I srv llama_server: model loaded I srv llama_server: server is listening on http://0.0.0.0:8080/ Benchmark Request Timing Statistics (Generating 512 tokens): I slot launch_slot_: id 0 | task 3454 | processing task, is_child = 0 I slot print_timing: id 0 | task 3454 | n_decoded = 103, tg = 37.93 t/s I slot print_timing: id 0 | task 3454 | n_decoded = 247, tg = 43.16 t/s I slot print_timing: id 0 | task 3454 | n_decoded = 392, tg = 44.87 t/s I slot print_timing: id 0 | task 3454 | prompt eval time = 195.88 ms / 42 tokens ( 4.66 ms per token, 214.41 tokens per second) I slot print_timing: id 0 | task 3454 | eval time = 11116.24 ms / 512 tokens ( 21.71 ms per token, 46.06 tokens per second) I slot print_timing: id 0 | task 3454 | total time = 11312.12 ms / 554 tokens I slot print_timing: id 0 | task 3454 | graphs reused = 3560 I slot print_timing: id 0 | task 3454 | draft acceptance = 0.62712 ( 333 accepted / 531 generated) I statistics draft-mtp: #calls(b,g,a) = 6 3607 3607, #gen drafts = 3607, #acc drafts = 2824, #gen tokens = 10820, #acc tokens = 6605, dur(b,g,a) = 0.008, 31925.089, 2.740 ms I slot release: id 0 | task 3454 | stop processing: n_tokens = 553, truncated = 0 ``` ___ :::

generation:

23.0 tokens/s

kv:

Q4_0

User reports Qwen3.6 35B-A3B generating 23 t/s at its maximum context window on an RTX 4060 8GB with 32GB of system RAM. The engine is not named; the KV cache is quantized to Q4_0. The user notes the first token took a long time to generate. The 35B model does not fit in 8GB of VRAM, so weights ran partly from system memory.

generation:

70.7 tokens/s

prompt processing (prefill):

1326.0 tokens/s

quant:

UD-Q3_K_M (GGUF)

kv:

FP16

User reports Qwen3.6-35B-A3B at 70.7 tok/s generation on a 31,296-token prompt on an RX 9060 XT 16GB with a Ryzen 5 5600X and 32GB RAM on Windows 11. Setup is the PolyStrata-HIP fork of the strata-poly GGUF engine with Unsloth's UD-Q3_K_M quant, FP16 KV cache and MTP speculation enabled; experts not resident in VRAM are computed on the CPU from system RAM. Prefill on that run was 1,326 tok/s. Shorter warm prompts reached 93.4 tok/s at 2,068 tokens and 85.8 tok/s at 8,235 tokens, while a 45,355-token prompt gave 61.9 tok/s. With Q8_0 KV and 262,144 tokens of allocated capacity, a 400-token response ran at 33.8 tok/s capped at 2,000 resident experts versus 47.6 tok/s uncapped with 5,097 resident. Streaming KV through pinned RAM at roughly 261K tokens dropped generation to 3.64 tok/s.

generation:

154.0 tokens/s

quant:

oQ5e-mtp

kv:

bf16

User benchmarks Qwen3.6-35B-A3B at 154 t/s decode on a 1k prompt with oMLX's native engine on an M5 Max 128GB, falling to 55 t/s at 262k context. Setup is oMLX 0.7.1rc1 with the oQ5e-mtp quant, Lightning MTP on, TurboQuant KV off, and a 128-token greedy generation at each context step. Splash 1.3.2 with int8 KV decoded 147 t/s at 1k and 78 t/s at 262k, and used 30.2 GiB against native's 33.5 GiB prefill peak and 37.4 GiB decode. TurboQuant 5- and 6-bit decoded at 27-29 t/s at 262k, about half of off. The user also ran Qwen3.8-27B, where Splash int8 was faster at almost every context (69 vs 48 t/s at 1k) and native TurboQuant cost 7-25% decode speed with MTP on. No setting showed a detectable accuracy loss of more than about 2 points.

generation:

30.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8_0

User reports Qwen3.6 35B-A3B at close to 30 t/s on an RTX 5060 8GB with a Ryzen 5 3600 and 32GB RAM. Setup is LM Studio with Q4_K_XL weights and a Q8_0 KV cache, reaching 98k context by moving some layers out of VRAM; total VRAM use stays below 7.5GB and RAM around 28-30GB. The user is asking for a better agentic coding model for this hardware and doubts Qwen3.8 27B would fit, and has not yet tested the 2-bit Ternary Bonsai 2.