Local model performance on your hardware

generation:

15.7 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 15.7 t/s on an RTX 4070 12GB and V100 16GB with 32GB of system RAM. Setup is llama.cpp with IQ3_XXS weights, 98k context, flash attention, jinja, and MTP with 2 draft tokens, using layer split rather than tensor parallelism. The user asks whether the model should require 45.8 GB in memory as shown for an AD-3.84bpw-IQ4_XS-M64 build, and notes their own RAM usage stayed around 6 GB.

generation:

20.6 tokens/s

prompt processing (prefill):

300-350 tokens/s

quant:

4.27 bpw (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next (125B-A6B MoE) at 20.65 t/s generation on an RTX 4070 12GB with 64GB DDR5-5600 and Gen4 NVMe on Linux. Setup is llama.cpp with AtomicChat's 4.27 bpw GGUF, n-gram SSD offloading, --fit-target 512, and an MTP variant (PR #28243 + compact shared-Q4_K_M head) with -ncmoe 45, yielding 77-96% acceptance. Prompt processing is 300-350 t/s. Baseline was 6 t/s; master branch reached 19.35 t/s. User notes MTP gives limited gains at low VRAM because layers must be freed for the MTP head.

generation:

11.7 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 12.55 t/s for a 256 token response and 11.65 t/s for a 512 token response on an RPC split across an RTX 4070 Ti and an M5 MacBook Air with 16 GB unified memory. MTP off gives 9.72 t/s. The use case is agentic coding.