Local model performance on your hardware

generation:

6-7 tokens/s

quant:

4-bit

User reports DeepSeek V4.1 Flash at 6-7 tok/s on a single RTX 3090 with 128GBx3 DDR4-2933 LRDIMM and a 4-bit quant. Setup uses ik_llama.cpp with experts streamed from SSD and on-demand expert streaming to GPU; there is a 100-200 token warmup while experts populate. User notes the machine is bottlenecked at 40GB/sec RAM throughput with three sticks and expects 20-30 tok/s with a fourth RAM chip. GLM 5.3 Flash support is also included but was slow.

generation:

72.0 tokens/s

quant:

IQ4_KS (GGUF)

kv:

q4_0

User benchmarks Qwen3.6 27B on a single RTX 3090 at an enforced 370 W power cap, comparing ik_llama.cpp with IQ4_KS against mainline llama.cpp with Q4_K_M. ik_llama.cpp reaches ~72 t/s code decode and ~60 t/s narrative decode versus ~58 and ~50 t/s for mainline, an 18-20% gain, with quality tied at 100-103/150 on an 8-pack and ~0.5 GB lower VRAM. Setup uses q4_0 KV cache, MTP n=2, -np 1, and a 200K context configuration that fills to ~183K tokens. A two-stage ngram plus MTP speculator on the same engine lifts code decode to ~98 t/s at the same power cap, with narrative flat at ~59 t/s. The user notes the earlier reported tie was a measurement artifact from a stale mainline container answering on the shared port, and that both engines are strongly power-sensitive between 230 W and 370 W.

generation:

72.9 tokens/s

prompt processing (prefill):

1261.0 tokens/s

quant:

IQ4_KS (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

on

User reports Qwen3.6-27B-MTP-IQ4_KS.gguf at 72.9 t/s decode and 1261 tok/s prefill on an RTX 3090 24GB at 156k context. Setup is ik_llama.cpp with a q8_0/q8_0 KV cache, MTP, and vision on CPU. The user also tested llama.cpp and BeeLlama.