Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
mathagentic
User releases Math-Physics-Qwen3.6-10.5B, a 10.5B specialist distilled from Qwen3.6-35B-A3B by expert pruning, reporting ~128-138 tok/s single-stream decode at Q4_K_M and ~87-100 tok/s at Q8_0 on an RTX 5060 Ti.
Setup is llama.cpp with Q4_K_M weights, q8_0 KV cache and 8192 context; the Q4 file is ~6.5 GB and the Q8 ~11.2 GB, fitting within 16GB VRAM.
Aggregated throughput exceeds 300 tok/s across 8 concurrent slots (-np 8). Screening scores on Q8_0: MATH-500 levels 4-5 80.0%, GPQA-Diamond physics 80.0%, MMLU-Pro physics 63.3%, UGPhysics 36.7%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-27B at 52.1 tok/s on an M2 Ultra 137GB with n-gram self-speculative decoding using one draft token.
The run used an experimental 2-bit mixed quant recipe (quant2-all); the model is hybrid with 48 GDN and 16 SA layers. The 1-draft configuration gave a 44% draft hit rate and a 17% speedup.
User notes higher draft counts are counterproductive due to quadratic multi-token forward overhead on bandwidth-bound models, and that the gain is architecture-level rather than tied to weight precision.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.6-27B at 41.70 t/s decode and 1075.8 t/s prompt processing on an RX 7900 XTX.
Setup is llama.cpp (ROCm/HIP backend, LTO and WMMA flash attention on) with IQ4_XS weights and F16 KV cache at 16K context, batch 1024 and ubatch 256.
The model is a 14.1 GiB MTP build; the user calls this the best overall configuration and notes LTO gives a 5-15% improvement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.6-35B-A3B at about 180 tok/s single-stream decode on two RTX 4090s at roughly 50k context, dropping to about 160 tok/s at roughly 110k.
Setup is sglang with a QuantTrio AWQ 4-bit quant, TP=2 over PCIe with no NVLink. Prefill is 9-10k tok/s, and 8 concurrent requests reach 985 tok/s total.
The user runs it as a GitLab code-review agent and says repetition is the main problem with local models, with the model looping on itself ending about 7% of runs (56 of 847).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.6-35B-A3B at 23.0 tok/s generation and 29.6 tok/s prompt reading on a single prompt, on a 16 GB Mac mini M4.
Setup is stock llama.cpp b11715 with the Kraits plugin, Unsloth UD-Q4_K_M (22.1 GB) weights, 16K context, temperature 0 and thinking off.
A 4-agent crew writing a TypeScript LRU cache ran at 16.0-23.8 tok/s writing and 37.8-47.3 tok/s reading at ~16K context, scoring 9/9; the ~8K run scored 8/8 and the ~1K run 6/8. The single prompt scored 8/8 in 43.7 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writingsummarization
User reports Qwen3.6-35B-A3B at about 21 tok/s decode on a single Tesla K80 (both dies), in a 2-core, 8 GB, no-AVX host.
Setup is the Pantheon engine on the Foundry C library with Q3_K_M weights held on the card rather than in system RAM; the CUDA backend targets Kepler sm_37.
Prompt reading is about 110-120 tok/s up to roughly 8k tokens, about 75 at 14k and about 55 at 30k. Prefix caching is built in, and an opencode second turn with a 14k-token system prompt dropped from 170 s to 9 s. The user notes prefill is the K80's weak spot. Nex-N2-mini also ran at about 21 tok/s decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.6-35B-A3B Q4_K_M decoding at 25-30 tok/s on a 16 GB Mac with the adaptive streaming planner enabled, up from 7 tok/s without it.
Setup is llama.cpp with a streaming MoE planner; a 2000-token prompt drops from 21 s on CPU to 3.4 s on GPU.
The same 21 GB model runs on a 12 GB Android phone at 10 tok/s decode and 100-300+ tok/s prefill on the NPU, and on a 16 GB Windows laptop at 5 tok/s. Qwen3.8-Flash-Next 125B reaches 6.5-9 tok/s on the 16 GB Mac and 2.2-3.5 tok/s on the phone; DeepSeek V4 Flash 284B runs at 1 tok/s on the phone.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.6 35B-A3B at 154 t/s on a 2024 MacBook Pro M4 Pro with 48GB, using the Kapitsa engine.
Setup is Kapitsa, an open-source Metal engine for Apple Silicon, with 4-bit MLX weights and DFlash2 speculative decoding; part of the 27B's prefill runs on the Neural Engine.
The same table gives Qwen3.8 27B at 63 t/s and Qwen3.5 4B at 179 t/s on the M4 Pro, and on an M5 Max 128GB Qwen3.6 35B reaches 281 t/s. The author says not to trust benchmarks and notes Kapitsa is not fastest on the M5 Max for Qwen3.8 27B (102 vs mlx-serve's 130) or Flash-Next.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
rp
User reports 46.06 t/s generation and 549.27 t/s prompt processing.
Source evidence (verbatim):
Source: https://lemmy.world/post/48144487
# Qwen 3.6 27B running at 46 tok/s on an RX 9070 XT (llama.cpp + MTP Speculative Decoding is basically magic)
Just got my hands on a new AMD Radeon RX 9070 XT (16GB) and wanted to share some inference numbers. I've been messing around with llama.cpp via their official ROCm Docker image, testing out Qwen 3.6 27B (Omnimerge-v4) in IQ3_M.
Honestly, the performance you can squeeze out of a 27B model on a 16GB consumer card right now is blowing my mind.
Here’s the breakdown:
The Setup
- GPU: AMD Radeon RX 9070 XT (RDNA4 / gfx1201) - 16 GB VRAM
- CPU: AMD Ryzen 9 9950X3D
- OS/Backend: Linux via Docker using ghcr.io/ggml-org/llama.cpp:server-rocm. (Props to the devs, it natively supports RDNA4 gfx1201 out of the box!)
- Model: Qwen3.6-27B-Omnimerge-v4-IQ3_M.gguf (~13 GB)
- Context: 16k
Tweaks:
- Set -np 1 since I'm just running it as a single-user chatbot in Open WebUI.
- Slapped on 8-bit KV cache (--cache-type-k q8_0 --cache-type-v q8_0) to save about 50% VRAM.
- Enabled MTP Speculative Decoding (--spec-type draft-mtp).
The Numbers (512-token test)
- Prompt Processing (TTFT): 549.27 tok/s (1220 tokens evaluated in ~2.2s). Latency: 1.82 ms/token.
- Text Gen: 46.06 tok/s (512 tokens generated in ~11.1s). Latency: 21.71 ms/token.
- MTP Stats: Draft acceptance rate was super high at 62.7% (333 drafts accepted / 531 generated). Aggregate speed (including prompt eval) hit roughly 48.97 tok/s.
Memory Footprint
- VRAM: Sitting at 14.46 GB out of 16 GB. This leaves about 1.5 GB of breathing room, which has been totally stable with zero OOM crashes so far.
- System RAM: ~4.3 GB (mostly the host-side prompt cache helping speed up subsequent turns).
RDNA4 is ready: The latest ROCm images and llama.cpp HIP libs support the 9070 XT natively. Didn't even need to mess with HSA_OVERRIDE_GFX_VERSION.
KV Cache quantization is required: Pushing the KV cache to q8_0 is the only reason a 16k context window fits on a 16GB card alongside a 27B model.
If anyone with a 16GB card is looking for the sweet spot, this has to be one of the best price-to-performance setups available right now. Let me know if you want my docker-compose.yml or run scripts...
Edit:
To see how much MTP actually helps, I ran the exact same 512-token prompt test with speculative decoding toggled off:
Standard Inference (No Speculation):
- Speed: 25.88 tokens/sec (Latency: 38.64 ms/token)
- Prompt Eval: 239.46 tokens/sec
MTP Speculative Decoding (MTP):
- Speed: 46.06 tokens/sec (Latency: 21.71 ms/token)
- Prompt Eval: 549.27 tokens/sec
- Draft Acceptance Rate: 62.7%
For more info, here are my logs
::: spoiler Logs
```
ROCm GPU and System Info:
device_info:
- ROCm0 : AMD Radeon RX 9070 XT (16304 MiB, 16200 MiB free)
- ROCm1 : AMD Ryzen 9 9950X3D 16-Core Processor (15589 MiB, 25012 MiB free)
- CPU : AMD Ryzen 9 9950X3D 16-Core Processor (31178 MiB, 31178 MiB free)
I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 |
AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1
Context Initialization (16k single-slot & 8-bit KV Cache):
I srv load_model: [spec] estimated memory usage of MTP context is 178.02 MiB
W llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
I srv load_model: initializing slots, n_slots = 1
I slot load_model: id 0 | task -1 | new slot, n_ctx = 16384
I srv load_model: prompt cache is enabled, size limit: 8192 MiB
I srv llama_server: model loaded
I srv llama_server: server is listening on http://0.0.0.0:8080/
Benchmark Request Timing Statistics (Generating 512 tokens):
I slot launch_slot_: id 0 | task 3454 | processing task, is_child = 0
I slot print_timing: id 0 | task 3454 | n_decoded = 103, tg = 37.93 t/s
I slot print_timing: id 0 | task 3454 | n_decoded = 247, tg = 43.16 t/s
I slot print_timing: id 0 | task 3454 | n_decoded = 392, tg = 44.87 t/s
I slot print_timing: id 0 | task 3454 | prompt eval time = 195.88 ms / 42 tokens ( 4.66 ms per token, 214.41 tokens per second)
I slot print_timing: id 0 | task 3454 | eval time = 11116.24 ms / 512 tokens ( 21.71 ms per token, 46.06 tokens per second)
I slot print_timing: id 0 | task 3454 | total time = 11312.12 ms / 554 tokens
I slot print_timing: id 0 | task 3454 | graphs reused = 3560
I slot print_timing: id 0 | task 3454 | draft acceptance = 0.62712 ( 333 accepted / 531 generated)
I statistics draft-mtp: #calls(b,g,a) = 6 3607 3607, #gen drafts = 3607, #acc drafts = 2824, #gen tokens = 10820, #acc tokens = 6605, dur(b,g,a) = 0.008, 31925.089, 2.740 ms
I slot release: id 0 | task 3454 | stop processing: n_tokens = 553, truncated = 0
```
___
:::
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B generating 23 t/s at its maximum context window on an RTX 4060 8GB with 32GB of system RAM.
The engine is not named; the KV cache is quantized to Q4_0. The user notes the first token took a long time to generate.
The 35B model does not fit in 8GB of VRAM, so weights ran partly from system memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User asks for small-model recommendations for agentic harnesses on a GTX 1060 6GB with 48GB DDR4 system RAM.
They report running Gemma 4 26B and Qwen3.6 35B in Ollama with a 32k sliding context window, getting about 10 t/s thanks to the MoE architecture.
The speed is given only as an approximate single figure, and the user is unsure about quantization, KV cache and whether Ollama hides features compared with llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 70.7 tok/s generation on a 31,296-token prompt on an RX 9060 XT 16GB with a Ryzen 5 5600X and 32GB RAM on Windows 11.
Setup is the PolyStrata-HIP fork of the strata-poly GGUF engine with Unsloth's UD-Q3_K_M quant, FP16 KV cache and MTP speculation enabled; experts not resident in VRAM are computed on the CPU from system RAM.
Prefill on that run was 1,326 tok/s. Shorter warm prompts reached 93.4 tok/s at 2,068 tokens and 85.8 tok/s at 8,235 tokens, while a 45,355-token prompt gave 61.9 tok/s. With Q8_0 KV and 262,144 tokens of allocated capacity, a 400-token response ran at 33.8 tok/s capped at 2,000 resident experts versus 47.6 tok/s uncapped with 5,097 resident. Streaming KV through pinned RAM at roughly 261K tokens dropped generation to 3.64 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 77.99 t/s with their own MoE inference engine Synapse, up from 45.50 t/s in control mode.
Setup is Synapse with Q8_0 weights; no hardware is named.
The post compares control against accelerated inference across four models: GPT-OSS-120B MXFP4 25.69 to 39.46 t/s, Qwen3.8-Flash-Next IQ3_S 25.39 to 43.88 t/s, GLM-4.5-Air 106B IQ4_XS 12.67 to 17.28 t/s, and Qwen3.6-35B-A3B Q8_0 45.50 to 77.99 t/s. The user credits Strata as inspiration and asks for feedback and testing on additional hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B at 154 t/s decode on a 1k prompt with oMLX's native engine on an M5 Max 128GB, falling to 55 t/s at 262k context.
Setup is oMLX 0.7.1rc1 with the oQ5e-mtp quant, Lightning MTP on, TurboQuant KV off, and a 128-token greedy generation at each context step. Splash 1.3.2 with int8 KV decoded 147 t/s at 1k and 78 t/s at 262k, and used 30.2 GiB against native's 33.5 GiB prefill peak and 37.4 GiB decode.
TurboQuant 5- and 6-bit decoded at 27-29 t/s at 262k, about half of off. The user also ran Qwen3.8-27B, where Splash int8 was faster at almost every context (69 vs 48 t/s at 1k) and native TurboQuant cost 7-25% decode speed with MTP on. No setting showed a detectable accuracy loss of more than about 2 points.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.6 35B-A3B at close to 30 t/s on an RTX 5060 8GB with a Ryzen 5 3600 and 32GB RAM.
Setup is LM Studio with Q4_K_XL weights and a Q8_0 KV cache, reaching 98k context by moving some layers out of VRAM; total VRAM use stays below 7.5GB and RAM around 28-30GB.
The user is asking for a better agentic coding model for this hardware and doubts Qwen3.8 27B would fit, and has not yet tested the 2-bit Ternary Bonsai 2.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen 3.6 35B-A3B at around 82 tok/s with a 16k context on an RTX 5070 Ti paired with an RTX 3070.
Setup is Ollama/llama.cpp splitting the model across the two mismatched cards, with the 3070 on an x1 PCIe riser.
About 23% of layers spill into system RAM and the split ratio between cards is not yet tuned; the 3070 has about 4GB sitting unused.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 83.75 t/s on an Apple M2 Max 64GB, the middle of three sequential runs (82.28, 83.75, 83.09 t/s).
Setup is LM Studio with the MLX runtime (mlx-llm-mac-arm64-apple-metal-advsimd 1.13.1), 4-bit MLX weights, 262144-token context, parallel 4, whole model in unified memory.
One request at a time with no speculative decoding; 705 prompt tokens and 255 completion tokens, all reasoning tokens. Run 1 was cold with 6.973 s to first token, warm runs 1.156 s and 0.897 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 19.53 t/s generation and 132.58 t/s prompt processing on an RTX 4060 Ti 16GB with an i7 7th-gen CPU and 48GB DDR4.
Setup is llama.cpp (llama-server) with Q5_K_XL GGUF, q8_0 KV cache, 131072 context, flash attention on, and --cpu-moe keeping experts in system RAM; only 4.5GB of VRAM is occupied.
User compares this to a dense ~26B model at Q4_K_M with -ngl 25, which crawls at around 2 t/s, and asks why the MoE run is nearly twice as fast despite streaming expert layers over PCIe.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen 3.6 35B-A3B at 500 tok/s decode on a machine with an RTX Pro 6000 and 2x RTX 3090 (96GB + 24GB + 24GB VRAM) and 128GB system RAM.
Setup is NInfer; quant and context length are not stated.
The 500 tok/s figure is for fast, simple tasks; the user's main agentic driver is Qwen Flash Next at Q6, with DSv4F+vision at Q8 as a tossup. Qwen 3.8 27B was the previous driver but struggles on long autonomous runs with 250+ tool uses per turn.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 11.7 tok/s warm cache on a Radxa ROCK 5C (Rockchip RK3588S) with 8 GB RAM, CPU only on 4x Cortex-A76.
Setup uses a custom ~2-bit quantization and a custom inference engine, with the model stored on a microSD; the 11.0 GB on-disk model exceeds RAM so part is streamed from storage.
Typical chat on new topics runs 5~8 tok/s and cold start about 2 tok/s. Quality versus the original BF16 model: MMLU 76.0% vs 84.2%, ARC-Challenge 94.1% vs 95.0%, with MMLU noted as a 10-per-subject subset carrying about ±3.5 points of noise.