Qwen3.6

reported speed:

7.4 tokens/s generation · 121.3 tokens/s prompt processing

quant:

8bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6 27B at 7.4 t/s generation and 121.3 t/s prompt processing on an M3 Max 96GB. Setup is oMLX with the MLX 8-bit model at pp1024/tg128, using 28.34 GB peak memory. A pp4096/tg128 run reached 8.8 t/s generation and 133.8 t/s prompt processing. Continuous batching at 4x reached 19.9 t/s aggregate generation. User is new to LLMs and asks whether the slow speed is due to the dense model or a setup problem.

reported speed:

170.7 tokens/s generation

quant:

ninfer quant

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares ninfer-3090 against llama.cpp on an RTX 3090 for Qwen3.6-35B-A3B and Qwen3.8-27B. The primary record is Qwen3.6-35B-A3B with ninfer-3090. llama.cpp with the UD-IQ4_XS GGUF reached 2439.7 t/s prefill, 148.6 t/s generation and 3410 ms TTFT. For Qwen3.8-27B, ninfer reached 187911.6 t/s prefill, 34.9 t/s generation and 27 ms TTFT, while llama.cpp reached 1012.7 t/s prefill, 38.5 t/s generation and 8160 ms TTFT. The benchmark used 7 prompts (4 long at roughly 7.7k-12.9k tokens, 1 medium at roughly 2.3k, 2 short at roughly 400-600), 3 repeats each, max_tokens=1024, seed=42 and cache busting via nonce. The ninfer-3090 system is a Ryzen 5 2600 with 16 GB RAM; the llama.cpp system is a Threadripper PRO 3945WX with 128 GB RAM. The user notes the hardware is not apples-to-apples but that inference is GPU-contained.

reported speed:

37.2 tokens/s generation

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks multiple models on 16 GB of VRAM. Qwen3.6-35B-A3B-APEX-I-Quality reaches 96.7% HumanEval pass@1, 100% agentic success and 37.2 tok/s. Ornith-1.0-35B IQ4_NL runs at 38 tok/s. Other models tested are Qwen3.8-27B variants, gemma-4-26B-A4B, KAT-Coder-V2.5-Dev and Qwen3.5-9B. Ornith 1.0 35B A3B wins overall.

reported speed:

56.0 tokens/s generation

quant:

Q2_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports llama.cpp PR #26563, which caches hot MoE experts on GPU, running Qwen3.6-35B-A3B on 8 GB of VRAM at 33.25 to 56.0 t/s with Q2_M and 17.34 to 35.93 t/s with Q5_K_P. Setup uses autofit with --expert-hot-s -1. The gains are 1.68x and 2.07x respectively. The user reports negative results for Qwen3.5-122B-A10B and Laguna-S-2.1. The change is CUDA only and applies to single-token decoding only.

reported speed:

32.5 tokens/s generation · 250.0 tokens/s prompt processing

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B with MTP enabled on llama.cpp server, reaching 175,000-200,000 tokens of context with a Q8 KV cache. Prompt processing runs 200-300 t/s and generation 25-40 t/s depending on draft hit rates. VRAM is constrained at 10 GB with a single KV cache slot. The user runs a custom fork of the pi coding agent for sub-agents.

reported speed:

29.0 tokens/s generation

quant:

IQ4_NL

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 29 t/s on Qwen3.6-35B-A3B IQ4_NL with the llama.cpp Vulkan backend on a Strix Point HX370 with an 890M iGPU and 64 GB RAM. MTP is enabled with acceptance rates of 70-90%, giving effective tokens/sec of 50-70. User expresses interest in Qwen3.8 but notes current models may not perform as well.

reported speed:

46.0 tokens/s generation · 1050.0 tokens/s prompt processing

quant:

Q4_K_M

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 27B Q4_K_M on Ollama at 100k context, running on an RTX 5060 Ti and 5070 Ti, with ~1050 t/s prompt processing and ~46 t/s generation on a ~35k token prompt. Setup is Open WebUI with Open Terminal integration on Unraid, with inference on a separate machine. The use case is legal document research and creation.

reported speed:

3.0 tokens/s generation

quant:

Q2_XXS

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a CPU-only test of MiniCPM5 2B. The model is described as much more impressive than MiniCPM5 2B, which was also tested but not extracted as primary.

reported speed:

60.0 tokens/s generation · 1000.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports prefill around 1000 PP/s and decode 50-70 t/s with Qwen 3.6 35B-A3B on an RTX 5080 eGPU. User asks about CPU instruction set differences for offloading.

reported speed:

135.0 tokens/s generation

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.6 35B-A3B at 130-140 t/s on an M5 Max at 4-bit. Setup uses MLX and llama.cpp engines in a local AI app for macOS.

reported speed:

24.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.6 27B at ~24 t/s on dual RTX 3060 GPUs. The user also discusses diffusion-based techniques, MOE, quants, and token authority as a wishlist for future local models.

reported speed:

50.0 tokens/s generation · 950.0 tokens/s prompt processing

quant:

Q4/Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 25.6 t/s on an RTX 3090 at 131,072 context. Setup is llama.cpp with the Huihui abliterated Q4_K_S GGUF and a Q4 KV cache, the whole 24 GB card in use. A second run at 196,608 context averaged 25.8 t/s. The user also mentions a 5090 with Qwen3.6 27B at ~140 tok/s and DeepSeek V4 Flash on dual RTX 6000 at ~80-100 tok/s, but the primary benchmarked setup is the 3090.

reported speed:

17.2 tokens/s generation · 706.0 tokens/s prompt processing

quant:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks a 128 GB M5 Max MacBook Pro across multiple models, selecting Qwen 3.6 27B MLX Q8 as the primary one. Other models tested include Gemma 4, MiniMax M2.7 and DeepSeek V4 Flash. The user is enthusiastic about local AI.

reported speed:

79.4 tokens/s generation · 2153.0 tokens/s prompt processing

quant:

Q8 XL

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks several models on an M5 Max 128GB and selects Qwen3.6 35B A3B as the primary model for its MoE detail. Other models tested include Gemma 4, Qwen 3.6 27B, MiniMax M2.7, Mistral Medium 3.5, Step 3.7 Flash and DeepSeek V4 Flash. Results show strong performance for MoE models.

reported speed:

40.0 tokens/s generation

quant:

Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B MoE at about 40 t/s on an RTX 5060 Ti 16GB with mmproj enabled, and just over 50 t/s without mmproj and with --n-cpu-moe 24. Setup is llama.cpp server in Docker with --cache-type-k/v q8_0, --cache-ram 0, --no-mmap, and --n-cpu-moe 19, or 24 without vision. The user experienced an OOM shutdown before adding --cache-ram 0 and is seeking advice on the setup.

quant:

BF16

rating:

4/5

User reports Qwen 3.6 27B abliterated BF16 running locally via vLLM with llama-swap and MTP speculative decoding, scoring 8/10 on a Terminal-Bench 2.0 pilot. The same setup beats DeepSeek-V4 IQ2 at 7/10 and comes close to an FP8 API at 9/10. It is the only configuration to pass the cancel-async-tasks hard task, and it missed build-cython-ext and sqlite-db-truncate on timeout.

quant:

NVFP4

User reports a C++20 port of the vLLM serving stack on a DGX Spark with Qwen3.6-27B NVFP4, reaching 86.05 to 1095.01 output tokens/sec across concurrency 1 to 32. The port runs slightly ahead of vLLM in the same tests. The user also reports DeepSeek-V4-Flash in 2-bit GGUF at 18.69 tok/s on the Spark, and compares against llama.cpp and MLX-LM.

reported speed:

61.4 tokens/s generation · 7800.0 tokens/s prompt processing

quant:

INT4 (W4A16)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.6 35B-A3B MoE on a single R9700 with vLLM, using INT4 weights from Avesed. The user also tests a 27B dense model with MTP spec=4. Prefill and decode speeds are measured at various context depths, and the user is happy with the results.

reported speed:

21.1 tokens/s generation · 287.3 tokens/s prompt processing

quant:

Q8_0

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.6 35B-A3B Q8_0 on a Ryzen 7 260 with a 780M iGPU and 64 GB DDR5, reaching pp8192 287.33 t/s and tg128 21.06 t/s. Setup is the Vulkan backend. Gemma 4 31B Q8_0 on the same hardware gives pp8192 51.59 t/s and tg128 2.46 t/s. With MTP, Gemma 4 31B reaches tg ~5.76 t/s, and Qwen3.6 35B-A3B with MTP and partial offloading reaches tg ~34.85 t/s. The user mentions an RTX 5060 8GB as a bonus for MoE partial offloading.

reported speed:

63.0 tokens/s generation · 1700.0 tokens/s prompt processing

quant:

Q6_K

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 63 t/s on 4x CMP 170HX 8GB cards flashed to 64GB each, 256GB total. Setup uses MTP with little-MoE default. Generation reaches 110 t/s with MTP optimistic.

reported speed:

70.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.6 35B-A3B at about 70 t/s on an RTX 3060. The figure is given as past experience rather than a benchmark run.

reported speed:

72.5 tokens/s generation · 1472.4 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.6-35B-A3B on a Mac with OMLX, comparing MTP enabled against disabled. MTP shows minimal speedup for the 35B MoE model but roughly 2x for the 27B dense model. Results include pp and tg t/s at various context lengths and batch sizes.

reported speed:

72.5 tokens/s generation · 1472.4 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.6-35B-A3B on OMLX with and without Lightning MTP. MTP shows no significant speedup for this MoE model. The same user benchmarks Qwen3.8-27B, a dense model, where MTP gives roughly 2x speedup. The hardware is a Mac, with the exact chip not specified.

reported speed:

59.0 tokens/s generation · 155.8 tokens/s prompt processing

quant:

Q4_K_M (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares Qwen3.8 27B dense at Q2 and Q3 against Qwen3.6 35B-A3B MoE on 12 GB of VRAM. The MoE was fastest and passed a sanity test, while dense Q3 ran at 7.5-9.1 t/s. The user prefers the MoE for local use.

reported speed:

18.4 tokens/s generation · 311.4 tokens/s prompt processing

quant:

Q8 (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks ROCm against Vulkan on a Radeon 780m iGPU. ROCm gives a 50% pp speedup for dense models. The user also reports Qwen3.8 27B results.

quant:

Q8

User benchmarks Qwen3.6-27B at 32 concurrent clients on the Paddock engine, with TTFT of 697 ms against 2.5 s for vLLM and 6.9 s for llama.cpp. User also reports Qwen3.8-27B at Q8 on an RTX PRO 6000 with speculation on and off: 46.8 to 202 tok/s single stream, 320 to 822 at eight concurrent chats, and 1005 to 1285 at 32. The engine is free but not open source.

reported speed:

39.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6 35B at 39 t/s on an 8 GB RTX 4060 laptop using the FreeToken engine. User also reports DeepSeek-V4-Flash 284B at 22-25 t/s on an RTX 5090 and GLM-5.2 753B at 15 t/s on an RTX PRO 6000.

reported speed:

37.5 tokens/s generation

quant:

Q6 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6 35B Q6 at 35-40 t/s decode on an RTX 4000 in a VM. User is considering swapping to a Strix Halo (Bosgame M5) for more concurrent models or a larger MoE. User also mentions Qwen3.8 27B on an RTX 3090 desktop and plans for larger models such as 3.5 122B or 3.8 Next Flash.

reported speed:

237.9 tokens/s prompt processing

quant:

UD-Q8_K_XL (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6 35B A3B prompt processing at 237.9 t/s at depth 0 on Strix Halo 128GB.

reported speed:

42.0 tokens/s generation · 1020.0 tokens/s prompt processing

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports MTP improves generation speed by 42% but reduces prompt processing by 18%. MTP requires at least 2GB free VRAM. Without MTP the run reaches 1020/42 t/s; with MTP it reaches 830/60 t/s.

reported speed:

2.7 tokens/s generation

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.6-27B Q4_K_M on an RX 5700 XT 8GB at 2.70 tok/s. The user also tests Qwen3.5, Qwen3.6-35B-A3B and Gemma-4-31b-it. The user ranks Qwen3.6-27B Q4_K_M second overall for the coding task.

reported speed:

84.5 tokens/s generation · 2000.4 tokens/s prompt processing

quant:

Q5_K_M (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 110-130 t/s generation and 1700 t/s prefill with MTP on. Setup is llama.cpp with the UD-Q5_K_M GGUF, 99 layers offloaded, 262,144 context, q8_0 KV cache, flash attention on, batch and ubatch size 2048, 8 threads, and parallel 1. The user notes ROCm gave much faster prefill but slower generation, and that MTP slows prefill compared to their usual setup.

reported speed:

80.0 tokens/s generation · 3000.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 128 concurrent requests reaching 1000 t/s generation on an unspecified setup. A single user at batch 1 yields 80 t/s generation and 3000 t/s prompt processing. No MTP is used.

reported speed:

82.0 tokens/s generation

quant:

Q8 (gguf)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.5-122B-A10B at 50 t/s and Gemma-4-26B-A4B at 113 t/s on a system with 16 GPUs. The system runs multiple models across the 16 GPUs. User notes hallucinations and pipeline issues.

reported speed:

70.5 tokens/s generation · 977.4 tokens/s prompt processing

quant:

Q4_K_M (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports creating a poker game without issues. User mentions trying Intel's vLLM fork previously.

reported speed:

45.2 tokens/s generation · 2022.5 tokens/s prompt processing

quant:

IQ4_XS (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.6 27B IQ4_XS on an RTX Pro 4500 Blackwell 32GB with llama.cpp b9007, reaching 2022.54 t/s prompt processing and 45.19 t/s generation. The same card also runs Qwen3.6 35B-A3B MXFP4 at 5507.10 t/s prompt processing and 159.81 t/s generation, along with Gemma4 26B-A4B MXFP4, Ernie 4.5 21B-A3B MXFP4, Nemotron Cascade 2 30B-A3B MXFP4, Tesselate OmniCoder 9B Q8, Qwen3.5 4B Q4_K, Qwen3.5 9B UD Q4_K_XL and GLM 4.7 Flash MXFP4. Compared with an RTX 5090, the 5090 is 60-70% faster at 2-3x power. User is happy with the card for 24/7 use.

quant:

Q8_0 (gguf)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

on

User reports 40-67 t/s decode on 2x Radeon AI PRO R9700 32GB at 131,072 context. Setup uses an F16 KV cache. Prefill throughput is 410-1500 t/s. MTP draft acceptance ranges from 0.33-0.61.

reported speed:

23.5 tokens/s generation · 432.3 tokens/s prompt processing

quant:

IQ4_XS (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 70 t/s total across 3 concurrent users, 23.3 t/s each, with 32K context per user on two P102-100 cards of 10 GB each. Setup is Qwen3.6-35B-A3B at IQ4_XS quantization, with prompt processing at 432 t/s. The two cards cost $100 total.

reported speed:

210.0 tokens/s generation · 4000.0 tokens/s prompt processing

kv:

Q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a forked Ninfer on a CMP 170HX roughly doubles performance versus llama.cpp on Qwen3.6-35B-A3B. The setup is used for a Home Assistant voice assistant.

reported speed:

97.7 tokens/s generation · 1330.0 tokens/s prompt processing

quant:

Q6_K (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B Q6_K on an RTX 3090 in llama.cpp, with prompt processing improving from 564.5 to 1330.0 tok/s (2.36x) while TG stays at about 97.7 t/s. Setup offloads 8 MoE expert layers to CPU with larger batch sizes, at 64K context and a Q8 KV cache, on a Threadripper PRO 3955WX with about 100 GB DDR4. Measurements are two repetitions with about 1.6% drift.

reported speed:

52.2 tokens/s generation · 608.0 tokens/s prompt processing

quant:

Q8

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 608 t/s cold prefill and 52.2 t/s decode on 4x RTX 5060 Ti 16GB at 256K context. Setup is a Vast AI instance with the Q8 quant, an FP16 KV cache, and MTP enabled. User considers this excellent for $2K hardware.

reported speed:

50.0 tokens/s generation

quant:

Q8_XL

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.6 35B at 50 t/s on Strix Halo. Setup is the Q8_XL quant. User emphasizes energy efficiency and versatility of Strix Halo compared to Nvidia cards.

reported speed:

100.0 tokens/s generation

quant:

Q8_0 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports tensor split-mode raising throughput from 70+ t/s to 100+ t/s, with a peak of 130 t/s. Power draw is 750W+.

quant:

Q4_K_M (gguf)

User reports a vision benchmark champion model at 70 s/img on an Apple M2 Max 96GB with llama.cpp b9690. The model gives the best quality and stability with thinking disabled, and completed 90/90 successful runs.

reported speed:

19.0 tokens/s generation

quant:

IQ4_XS (gguf)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 19 t/s peak and 14 t/s during long generation at 65k context with the KV cache offloaded to RAM via -nkvo, which allows the whole model on GPU with an f16 KV cache. Speed remained similar at 128k context with 63 layers on GPU. Quantizing the KV cache to RAM did not improve performance.

quant:

Q6_K (gguf)

User reports running Qwen3.6-27B as the reasoning layer in a multi-agent orchestrator for 2 weeks, replacing Claude. Plan generation is good, but tool-call reliability is poor with a 12% format error rate, long-context drift appears past ~14k tokens, and cascade-failure handling is weak. The user concludes the model is viable as a reasoning layer but not as an execution layer.

quant:

4bit

User runs Qwen3.6-35B-A3B-4bit on an M3 Max 128GB for production sub-agent delegations. User also mentions GLM-5.1 for orchestration. User is considering building a 5090 rig.

reported speed:

104.0 tokens/s generation · 1399.0 tokens/s prompt processing

quant:

Q8

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User recommends getting enough GPUs to avoid VRAM hacks, and runs 2x RTX 3090s.

reported speed:

70.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-27B at ~70 t/s on dual RTX 3090s, generating interactive HTML content inline with chat.

reported speed:

43.3 tokens/s generation · 456.1 tokens/s prompt processing

quant:

Q4_K_S (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 43.26 t/s generation and 456 t/s prefill at 12k context on dual RTX 3060 cards. Setup is tensor parallel with MTP enabled and 64k context. Without MTP at 96k context, generation is 31 t/s. User praises the value and stability of CUDA.