Local model performance on your hardware

generation:

125.0 tokens/s

quant:

UD-Q2_K_XL (GGUF)

mtp (multi-token prediction):

on

User reports 125 tokens per second running Tiel-Coder-35B-A3B-MTP at UD-Q2_K_XL on an RTX 4070 Ti SUPER 16GB, served through LM Studio to the Pi coding agent. Setup uses MTP speculative decoding, MoE architecture and a 65535-token context window; the user also runs the abliterated Cyber-Tiel-Coder variant of the same model. The user notes a dense Qwen 3.8 27B gives only 10-20 tokens per second on this card, and works around the small context by splitting tasks into plan.md phases and a subtask4llm skill.

generation:

18.0 tokens/s

User reports Qwen3.8 Flash-Next at 18 t/s running across two RTX 3090s, an RTX 4070 Ti Super and 130GB of DDR4 system RAM. Setup details are not stated; the model is partly offloaded to system RAM and NVMe, with an 18-core CPU and 1TB NVMe drive. Running only from the 4070 Ti Super plus NVMe gives 8-9 t/s, and NVMe alone gives 2 t/s. The user asks how to get more speed without buying new hardware.

generation:

36.5 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

Q8

User reports Qwen 3.8 27B at 36.5 t/s on an RTX 4070 Ti Super 16 GB with 64 GB system RAM. Setup is Ollama on Windows with UD-Q3_K_XL and an 8-bit KV cache at 32K context. The user says the 8-bit cache alone added about 26% generation speed and Q3+Q8 was about 78% faster than IQ4+Q8; other rows were Q4_K_M at 11.3 t/s, IQ4_XS at 16.3 t/s and IQ4_XS with 8-bit cache at 20.5 t/s. These are short tests on two warmed deterministic prompts, not an accuracy benchmark, and harder-task quality is untested.

generation:

86-91 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

flash attention:

on

User reports Ornith-1.5-35B-A3B Q4 at 86–91 t/s generation and about 1,030 t/s prompt processing on an RTX 4070 Ti SUPER 16 GB, at a roughly 3–6k-token prompt. Setup is llama.cpp b11370 (CUDA) in router mode with Q4_K_M weights, q8_0 KV cache, 65536 context, flash-attn on, and n-cpu-moe 15, using about 15.1–15.6 GB of the 16 GB card. The figures are given as ranges, so no single throughput is recorded. The run is compared against a previous RTX 3060 Ti 8 GB setup; the host is an i7-14700K with 32 GB RAM on Windows 11.

generation:

67.5 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Qwen3-Next-80B-A3B-Instruct Q4_K_M on an RTX 4070 Ti Super 16GB, reporting 67.54 t/s median decode on real text and 80.673 t/s on a repeated prompt. Setup is llama.cpp b11617 with -ncmoe 48, --moe-cache-mib 10752, 4096 context and a single-CCD CPU affinity mask; the 46.64GB model streams experts from system memory over PCIe. All figures are warm-cache, 435 prompt tokens and 256 generated tokens. A 512 MiB expert cache was 23.8% slower than no cache, and decode fell 70.5% at -ncmoe 44. Random token IDs decoded at 93.61 t/s, about 38% faster than real text.

generation:

36.1 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8 Flash-Next at 36.1 t/s on an RTX 4070 Ti Super, up from 30.3 t/s after raising Strata's --adapt-every from 1 to 6. Setup is Strata with UD-Q4_K_XL GGUF on a single card, tested on four Slovak-language chat prompts; the fourth prompt describes 20K tokens of source code. Per-prompt gains ranged from +12% to +55% with UD-Q4_K_XL; a UD-IQ4_XS run reached 63.1 to 71.6 t/s across the same prompts.

generation:

45-53 tokens/s

quant:

Q2

User reports Qwen3.8 Flash-Next at roughly 45-53 output tokens per second on an RTX 4070 Ti SUPER 16GB. Setup is Strata with Q2 quantization and a 128K context configuration, running on a 16GB GPU with 64GB DDR4 DRAM. The complete workflow used multiple computers; the LLM ran on the 16GB GPU while modelling, image generation and other services were handled elsewhere. The user notes Windows was using 15 to 20 GB of the system RAM.

generation:

197.0 tokens/s

prompt processing (prefill):

3300.0 tokens/s

quant:

Q2 (GGUF)

User reports Qwen3.8 Next at 197 t/s generation and 3,300 t/s prefill on an RTX 4070 Ti Super 16GB. Setup is Strata on CachyOS with Q2 weights, 10k and 20k token prompts, 15.6GB VRAM plus 44GB system RAM used. An IQ3XXS run reached 120 t/s generation and 3,000 t/s prefill with 50GB system RAM used; total system power was 350W.

generation:

72.0 tokens/s

prompt processing (prefill):

1700.0 tokens/s

quant:

IQ3_S (GGUF)

User reports Qwen3.8 Flash-Next at 72 t/s generation and ~1,700 t/s prompt reading on an RTX 4070 Ti Super 16GB. Setup is Strata 0.1.39 with IQ3_S (~84GB) at 256K context, on a Ryzen 9 7950X3D with 128GB DDR5. Out of the box it did 8 t/s; after START-HERE.bat --calibrate it reached 72 t/s. A 17K prompt reads in 8 seconds. User's 27-task coding suite scored 27/27 x3 and HumanEval 159/164. Same model family under llama.cpp with offloading ran at 19 t/s. Keeping the 29GB n-gram table in RAM with --ple-io mmap made results worse (25-26/27 and slower).

generation:

10.0 tokens/s

quant:

Q5 (GGUF)

User reports Qwen3.8 27B at around 10 t/s on an RTX 4070 Ti Super in a Threadripper 3975WX system with 256 GB RAM. Setup is llama.cpp server with the Unsloth Q5 GGUF, run with only --ctx-size and --jinja flags. User asks whether 10 t/s is the best achievable at this quant without a better GPU.

generation:

54.1 tokens/s

prompt processing (prefill):

1522.4 tokens/s

quant:

IQ2_XXS (GGUF)

kv:

Q4

User reports a full 256k context offloaded to VRAM with no CPU spillover, using n-gram speculative decoding. Generation speed degrades with context depth: 43.69 t/s at 32k, 36.59 t/s at 64k, 27.55 t/s at 128k, 22.15 t/s at 192k, and 18.48 t/s at 256k. Prompt processing also degrades similarly.

generation:

5.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports a 4070 Super run that was slow because only 38 of 66 layers were on GPU, with the rest running from system RAM. The same model achieved 81.5 tok/s on a 5090. The user introduces a tool called Picchio for diagnosing layer placement and speeds.

generation:

7.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User reports Qwen 3.8 27B Q4_K_M running on an RTX 4070 Ti Super 12GB and a MacBook M5 Air over RPC with llama.cpp, completing a coding game clone task in 117 minutes. Setup uses llama.cpp over RPC between the two machines. LM Studio is also mentioned. The 117 minutes is the total time for the task, not a throughput figure.

generation:

47-50 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q4

User reports a custom hybrid quantization that fits MTP and long context into 16 GB of VRAM, with VRAM usage of ~15.93 GB. Setup uses an asymmetric KV cache mix of kvarn5 (K) and kvarn4 (V), with tail precision of 1024 tokens and speculative decoding with 2 draft tokens.