RTX 4060 Ti 16GB

reported speed:

18.7 tokens/s generation · 210.0 tokens/s prompt processing

quant:

Q5_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash at 18.7 t/s generation on a Ryzen 7700 with 64 GB DDR5 and an RTX 4060 Ti 8 GB. Setup is a custom llama.cpp fork (myllama.cpp) running the 35B MoE model with 6B active parameters, CUDA and Windows specific, with no MTP support. An initial IQ4_XS quant gave 100 t/s prefill and 20 t/s generation, degrading to 15 t/s at 100K context. A switch to Q5_K_M dropped to 15 t/s prefill and 10-13 t/s generation from MMAP page faulting. A four-layer system followed: an unbuffered raw file reader with a RAM buffer for 2 slabs reached 150 t/s prefill, a RAM expert cache with decay factor (55-56 GB, covering 85-95% of requests) gave 14-15 t/s generation, memcpy from cache to buffer during prefill reached 215 t/s prefill, a 2 GB VRAM cache for top experts gave 18.0 t/s generation, and direct IO for embeddings gave 18.7 t/s.

reported speed:

20.0 tokens/s generation · 800.0 tokens/s prompt processing

quant:

IQ3_K_XXS

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.8 27B IQ3_K_XXS by Unsloth running fully on a 4060Ti 16GB at ~100k context, averaging 800 t/s prefill and 20 t/s decode. Setup uses a Q8 KV cache with mmproj and MTP dropped. Decode falls to 17 t/s after 64k context. The user runs agentic coding with parallel tool calls and successfully merged a feature branch, and plans to upgrade to an R9700.

reported speed:

7.4 tokens/s generation · 15.2 tokens/s prompt processing

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a 98 GB model running on 16 GB VRAM with CPU offload, improving from 2 to 7 t/s with recent llama.cpp updates.

User reports Qwen 3.8 27B overthinking, run on a 4060 Ti with 16 GB VRAM. The model can be run with 16 GB VRAM plus system RAM. Gemma 4 26B A4B and GPT-OSS 20B are named as alternatives for daily tasks. The user cites SWE-Rebench insight that Qwen Next requires huge token counts, and hard limits reasoning to 8192 in llama.cpp and vLLM.

reported speed:

11.0 tokens/s generation

quant:

IQ3_S (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B IQ3_S at about 11 t/s for code generation and 6-7 t/s on reasoning. Setup is speculative decoding with a Q4_K_M draft model. User asks for optimization advice.

reported speed:

32.5 tokens/s generation

quant:

Q3_xxs (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 30-35 t/s from a local model, dropping to 21-22 t/s at long context. The user is impressed with its coding performance, one-shotting tasks that Qwen3.6 35B failed. The user notes occasional misunderstandings in conversation and on basic tasks, possibly due to a low quant.