Local model performance on your hardware

generation:

122.5 tokens/s

prompt processing (prefill):

76.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User compares a Mac Studio M5 Max (40-core GPU, 64GB) against their current dual RTX 4060 Ti 16GB setup and asks what prefill and decode speeds Mac owners get. On the 2x 4060 Ti they report Qwen3.6-35B-A3B at UD-Q4_K_XL with 262k context reaching 76.7 t/s prefill and 122.5 t/s decode. A second row gives Qwen3.8-27B dense at UD-Q4_K_XL with 200k context at 55.5 t/s prefill and 34.3 t/s decode. No engine is named.

generation:

19.5 tokens/s

prompt processing (prefill):

132.6 tokens/s

quant:

Q5_K_XL (GGUF)

kv:

q8_0

User reports Qwen3.6 35B-A3B at 19.53 t/s generation and 132.58 t/s prompt processing on an RTX 4060 Ti 16GB with an i7 7th-gen CPU and 48GB DDR4. Setup is llama.cpp (llama-server) with Q5_K_XL GGUF, q8_0 KV cache, 131072 context, flash attention on, and --cpu-moe keeping experts in system RAM; only 4.5GB of VRAM is occupied. User compares this to a dense ~26B model at Q4_K_M with -ngl 25, which crawls at around 2 t/s, and asks why the MoE run is nearly twice as fast despite streaming expert layers over PCIe.

generation:

11.0 tokens/s

prompt processing (prefill):

744.0 tokens/s

quant:

IQ3_S (GGUF)

User reports Poolside Laguna S 2.1 IQ3_S at 11 tok/s decode and 744 tok/s prefill at 128K context on an RTX 4060 Ti 16GB with 128GB DDR4 and a Threadripper 1950x. Setup uses the user's own floatie engine, an attempt to replicate Strata's partial-offload approach, with OMP. User compares against Qwen3.8-Flash-Next-UD-Q4_K_XL at 18.2 tok/s and Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S at 35.8 tok/s on the same task; the Laguna run finished in 84 min with 4 bugs found in review versus 30 min/1 bug and 20 min/4 bugs for the Qwen runs. User says they will not pursue this further.

generation:

44.3 tokens/s

prompt processing (prefill):

870.0 tokens/s

User reports Qwen3.8 27B at 44.3 t/s generation and 870 t/s prompt processing on an RTX 4060 Ti 16GB, using a custom llama.cpp fork (llamADALL) built for 40-series cards. Setup uses MTP with 3 draft tokens, chat template and thinking on, power limited to 133 W, over a PCIe 1x to 16x mining riser. Numbers are the average of 2 runs on a chat growing from 8K to 124K tokens. The fork roughly doubles base llama.cpp speeds; at 124K context it reaches 33.0 t/s generation and 496 t/s prompt versus 16.6 and 193 for base llama.cpp. User notes removing the power limit improves results by 5-8%.

generation:

15.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3-30B-A3B at 15 t/s on an RTX 4060 Ti 16GB using TensorTune's smart tensor offloading. Setup is KoboldCpp with Q4_K_M quant, all layers on GPU with tensor offloading. The baseline with traditional layer offloading (30 layers) was 10 t/s, a 50% improvement.

generation:

18.7 tokens/s

prompt processing (prefill):

210.0 tokens/s

quant:

Q5_K_M (GGUF)

User reports Qwen3.8 Flash at 18.7 t/s generation on a Ryzen 7700 with 64 GB DDR5 and an RTX 4060 Ti 8 GB. Setup is a custom llama.cpp fork (myllama.cpp) running the 35B MoE model with 6B active parameters, CUDA and Windows specific, with no MTP support. An initial IQ4_XS quant gave 100 t/s prefill and 20 t/s generation, degrading to 15 t/s at 100K context. A switch to Q5_K_M dropped to 15 t/s prefill and 10-13 t/s generation from MMAP page faulting. A four-layer system followed: an unbuffered raw file reader with a RAM buffer for 2 slabs reached 150 t/s prefill, a RAM expert cache with decay factor (55-56 GB, covering 85-95% of requests) gave 14-15 t/s generation, memcpy from cache to buffer during prefill reached 215 t/s prefill, a 2 GB VRAM cache for top experts gave 18.0 t/s generation, and direct IO for embeddings gave 18.7 t/s.

generation:

20.0 tokens/s

prompt processing (prefill):

800.0 tokens/s

quant:

IQ3_K_XXS

kv:

Q8

User reports Qwen 3.8 27B IQ3_K_XXS by Unsloth running fully on a 4060Ti 16GB at ~100k context, averaging 800 t/s prefill and 20 t/s decode. Setup uses a Q8 KV cache with mmproj and MTP dropped. Decode falls to 17 t/s after 64k context. The user runs agentic coding with parallel tool calls and successfully merged a feature branch, and plans to upgrade to an R9700.

generation:

7.4 tokens/s

prompt processing (prefill):

15.2 tokens/s

quant:

UD-Q2_K_XL (GGUF)

User reports a 98 GB model running on 16 GB VRAM with CPU offload, improving from 2 to 7 t/s with recent llama.cpp updates.

generation:

11.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

Q8

User reports Qwen3.8 27B IQ3_S at about 11 t/s for code generation and 6-7 t/s on reasoning. Setup is speculative decoding with a Q4_K_M draft model. User asks for optimization advice.