Local model performance on your hardware

generation:

178.0 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8-Flash-Next at 178 tok/s decode and 4,828 tok/s prefill on a 64k prompt on two RTX 5070 Ti 16 GB cards. Setup is the user's own capy-flash engine (a fork of NInfer) with NVFP4 weights, tensor parallel across both cards and a shared VRAM expert cache whose misses run on the CPU, on a Ryzen 9 9900X with 96 GB DDR5. Figures are one request at a time, decode measured with LocalMaxxing reasoning-v1. The engine also runs Qwen3.8-27B.

generation:

127.0 tokens/s

User reports the Qwen3-VL 8B text encoder generating at about 127 tokens/s on an RTX 5070 Ti in Kokon 0.10, the same rate llama.cpp gives. Setup is the app's own fork of stable-diffusion.cpp with text generation added to the already-loaded text encoder, so no second model is downloaded or held in memory. The optional Prompt+ feature adds about 2 s per image on the RTX 5070 Ti and about 13 s on an M2 Pro, and works on 8 GB cards. The image side is upstream code and produces bit-identical images.

generation:

178.0 tokens/s

quant:

NVFP4

User reports Qwen3.8-Flash-Next at 178 t/s on 2x RTX 5070 Ti with 96 GB DDR5, using their own capy-flash engine with NVFP4 weights. Setup keeps experts in VRAM with CPU fallback for misses; the same box gives Strata IQ3_S 195 t/s and Strata NVFP4 120 t/s. Prefill on capy-flash is 2.4k / 4.6k / 4.8k t/s at 4k / 16k / 64k. A replayed 53-turn coding session growing to 81k tokens took 177 s on capy-flash, 197 s on Strata IQ3_S and 268 s on Strata NVFP4. The post also lists many other engines' figures for Qwen3.8-27B and Flash-Next, and EU prices where 96 GB of DDR5 costs about as much as a GPU.

generation:

195.0 tokens/s

prompt processing (prefill):

3800.0 tokens/s

quant:

IQ3_S (GGUF)

User benchmarks Qwen3.8-Flash-Next at 195 t/s decode on two RTX 5070 Ti cards with 96 GB of system RAM, using Strata with IQ3_S weights and experts offloaded to RAM. Prefill on the same box is 2.2k t/s at 3.5k tokens, 3.8k at 15k and 5.2k at 55k. On the same hardware capy-flash with NVFP4 and tensor parallel reaches 171-183 t/s decode, and Strata NVFP4 reaches 120 t/s. The user reports Strata IQ3_S beats NVFP4 engines on short decode at about twice the KL divergence (0.144 for IQ3_S vs 0.070 for NVFP4).

generation:

137.7 tokens/s

prompt processing (prefill):

5951.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

K q8_0, V q4_0

User reports 137.7 t/s generation and 5951 t/s prompt processing. Source evidence (verbatim): Source: https://algollabs.com/blog/gemma4-bfcl # Gemma 4 26B on consumer-grade GPU: Benchmarks, Build Pain, and a Week of Daily Use Hardware: NVIDIA RTX 5070 Ti 16 GB GDDR7, AMD Ryzen 9 9950X3D, 64 GB DDR5, Fedora 43, kernel 6.18. Engine: llama.cpp build 9024 built from source with GGML_CUDA_FA_ALL_QUANTS=ON. Model: Gemma 4 26B A4B Instruct (26B total, 3.8B active), unsloth/gemma-4-26B-A4B-it-GGUF UD-IQ4_XS, 12.65 GiB, full GPU offload (31 of 31 layers), flash attention on, asymmetric KV cache (K q8_0, V q4_0). llama-bench, no speculative decoding or MTP, single stream: 5,951 tokens/s prompt processing (pp2048) and 137.7 tokens/s token generation (tg64).

generation:

82.0 tokens/s

User reports Qwen 3.6 35B-A3B at around 82 tok/s with a 16k context on an RTX 5070 Ti paired with an RTX 3070. Setup is Ollama/llama.cpp splitting the model across the two mismatched cards, with the 3070 on an x1 PCIe riser. About 23% of layers spill into system RAM and the split ratio between cards is not yet tuned; the 3070 has about 4GB sitting unused.

generation:

147.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

int8

User reports Qwen3.8 Flash-Next IQ2_XS at 147 tok/s decode on an RTX 5070 Ti 16 GB with 64 GB DDR5-4800, after running Strata's --calibrate. Setup is Strata v0.1.43 with IQ2_XS GGUF, int8 KV cache, 131k context, --kv-resident 32768, MTP --spec 4, reasoning_budget_tokens 8192, conversation cache 4 GiB, vision encoder on CPU. Calibration raised median decode from 118 to 143-147 tok/s and cut the full 28-task x2 run from 20.6 to 15.4-16.0 min, but the benchmark score dropped from 50 to 44/56 twice, all in soft exact-match cases. Greedy decoding scored 50/56 versus 37 with Qwen's recommended sampling (temp 1.0/top_p 0.95/top_k 20), 43 at temp 0.6 and 40 at temp 1.0 + min_p 0.05. reasoning_effort high scored 42/56 and was twice as slow. Run-to-run noise is about +/-3/56. Real agent workload (Hermes, 54k context) gave about 100-104 tok/s decode with 78-83% expert-cache hit rate before calibration. RAM runs at 4800 MT/s with EXPO off, which the user estimates costs a few to ~20% versus DDR5-6000.

generation:

100.0 tokens/s

prompt processing (prefill):

3400.0 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8-Flash-Next 125B-A6B at ~100 tok/s generation and ~3.4k tok/s prefill on a 15k prompt on 2x RTX 5070 Ti 16GB. Setup is the capy-flash engine with NVFP4 weights, experts offloaded to RAM, tensor parallel across both GPUs, on a Ryzen 9 9900X with 96 GB DDR5. User compares against Strata, which gets 78 tok/s generation and 2.7k tok/s prefill with the same NVFP4 weights; Strata's 3-bit IQ3_S build is still faster at generation. User also notes a tensor-parallel finding: writing partial sums in FP16 instead of BF16 restores single-GPU numerical parity at no real speed cost.

generation:

107.0 tokens/s

prompt processing (prefill):

1831.7 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0

User reports Qwen3.8-27B (Swift-1.5 fine-tune, IQ3_S GGUF) at 106.96 t/s generation and 1831.68 t/s prompt processing on an RTX 5070 Ti 16GB. Setup is llama.cpp with Q8_0 KV cache, 60000 context, flash attention on, and MTP plus n-gram speculative decoding (draft-mtp,ngram-map-k4v, draft-n-max 3). User is happy with the speed but wants a larger context window and asks for advice on the launch arguments.

generation:

9.3 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 31B at 9.3 t/s on an NVIDIA RTX 5070 Ti. Setup is Ollama with Q4_K_M at 4096 tokens of context, with partial offload to host memory since the weights do not fit entirely in 16 GB. The source also reports 199 t/s prefill and 15.7 GB peak GPU memory; the run is a GPU-plus-host measurement, not all-GPU speed.

generation:

102.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 26B-A4B at 102.9 t/s on an NVIDIA RTX 5070 Ti. Setup is Ollama with Q4_K_M at num_ctx=4096, with partial offload since the weights do not fit entirely in 16 GB and spill into host memory. The source also reports prefill at 1485 tokens/s and peak GPU memory of 15.7 GB; the figure is a GPU-plus-host measurement, not all-GPU speed.

generation:

78.6 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 12B at 78.6 t/s on an NVIDIA RTX 5070 Ti. Setup is Ollama with Q4_K_M and num_ctx=4096, on one 16 GB card with reported peak GPU memory of 8.9 GB. The source also reports prefill at 1771 tokens/s; the article does not quantify GPU layer split or host offload, and longer-context behavior was not measured.

generation:

148.2 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 E4B at 148.2 t/s on an NVIDIA RTX 5070 Ti. Setup is Ollama with Q4_K_M and a configured context of 4096, on one 16 GB card. The source also reports prefill at 2549 tokens/s and peak GPU memory of 5.1 GB; the figure is source-reported decode, not independently recomputed.

generation:

226.3 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 E2B at 226.3 t/s on an NVIDIA RTX 5070 Ti. Setup is Ollama with Q4_K_M and num_ctx=4096, on one 16 GB card with reported peak GPU memory of 3.6 GB. The article also reports prefill at 3461 t/s; prompt and output token counts, run count and aggregation for this row are not published, and longer-context behavior was not measured.

generation:

124.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

User benchmarks five quants of Qwen3.8-27B on 2x RTX 5070 Ti 16GB, finding standard evals (GSM8K, MMLU-Pro, IFEval) cannot distinguish them within ±2 points. The UD-Q4_K_XL quant reaches 124 t/s decode with MTP and 245,760 tokens of context, versus 93 t/s and 32,768 tokens for Q8_0. Setup is llama.cpp with tensor split, MTP draft and q8_0 KV cache; NVFP4 runs use vLLM TP=2. Q4_K_XL picks a different token than Q8_0 once every 28 tokens by top-1 agreement. On the user's own agentic coding bench (112 runs per model, OpenCode, hidden tests), 27B NVFP4 solves 77% and Flash-Next NVFP4 98%; on the 6 hardest tasks it is 1/12 versus 12/12. SWE-rebench agrees at 69% versus 82% (paired p ≈ 0.007). Flash-Next Q3 on Strata matches NVFP4 on 11 of 12 discriminating tasks but always misses one causality bug.

generation:

81.9 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

User reports Qwen3.8-Flash-Next IQ3_S at 81.9 tok/s in calibration and 85.4 tok/s in chat on an RTX 5070 Ti 16GB with 64GB RAM. Setup is Strata with IQ3_S GGUF, int8 KV cache, 256K context, KV streaming with 32768 resident cells, and MTP speculation. The model is 125B-A6B with 46.84 GiB of experts loaded and an 84GB download. Two concurrent 256K slots exceeded a 58 GiB container limit during startup with and without batch MTP, and earlier unrestricted runs caused global OOMs and a host freeze. The user settled on one active request at 256K with KV streaming.

generation:

66.8 tokens/s

quant:

NVFP4 (GGUF)

flash attention:

on

User reports Gemma 4 31B IT NVFP4 at 66.80 tok/s aggregate effective batch throughput on an RTX 5070 Ti 16GB. Setup is llama.cpp with NVFP4 GGUF weights, a Gemma 4 31B Assistant/MTP draft model, --spec-type draft-mtp, --spec-draft-n-max 5, flash attention on, and all layers offloaded to GPU. The benchmark used mtp-bench across 13 mixed tasks with 2,016 predicted tokens in 30.18 s and a 60.2% draft acceptance rate. Aggregate throughput rose from 53.10 tok/s at n_max 2 to 66.80 tok/s at n_max 5, while acceptance fell from 78.2% to 60.2%. Per-task speeds ranged from 35.8 tok/s on creative writing to 104.6 tok/s on repeated patterns. The author notes these are aggregate figures and that no non-speculative baseline was measured.

generation:

82.7 tokens/s

prompt processing (prefill):

706.0 tokens/s

quant:

IQ3_XXS (GGUF)

User reports Qwen3.8-27B at 82.73 t/s median (79.4-89.6) on an RTX 5070 Ti 16 GB at 32K context. Setup is the den_llama.cpp fork (golden build2) with IQ3_XXS GGUF weights, MTP speculative decoding at n-max 2, accept rate 0.6703, prefill 706 t/s, np=1. At 65K context the median is 64.16 t/s (51.11-72.48) with q8_0 K / q4_0 V cache. A separate FA-fix build measured 79.48 t/s median at 65K (+23.9% vs control) but was marked experiment, not golden. Build3 adds NVFP4 KV at 88.6 t/s decode (KLD 0.0151) and a Q4_G64 KV codec (KLD 0.0163). A gemma-4-26B-A4B MoE lane runs 94-117 t/s with partial CPU+GPU expert offload (-ncmoe 4), and 129 t/s with q8_0/q4_0 KV and -ncmoe 2.

generation:

6.9 tokens/s

prompt processing (prefill):

2181.2 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

User reports Gemma 4 26B-A4B at 6.86 t/s generation and 2181.18 t/s prompt processing on an RTX 5070 Ti with ik_llama.cpp. Setup is ik_llama.cpp with Q4_K_XL GGUF and q8_0 KV cache, 30 MoE layers offloaded to CPU, 8192-token prompt and 128-token generation. Upstream llama.cpp on the same hardware and model reaches 30.10 t/s generation and 577.00 t/s prompt processing, making ik_llama.cpp about 4.4x slower for generation despite being 3.8x faster for prompt processing. Disabling CUDA graphs, using q4_0 KV cache, or keeping the last 4 MoE layers on GPU did not resolve the generation slowdown.

generation:

111.7 tokens/s

prompt processing (prefill):

4326.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3.6-35B-A3B at 111.72 tok/s decode and 4,326.76 tok/s prompt eval on an RTX 5070 Ti 16GB. Setup is llama.cpp with Q4_K_M GGUF, F16 KV cache, 65,536 context, cache 132 and ub=8192, single request. Peak VRAM was 14,369 MiB. At NP4 the same model reached 247.79 tok/s aggregate decode (62.00-67.78 per request) and 1,142.11-2,243.00 tok/s prefill per request. Gemma-4-26B-A4B-it Q4_0 runs are also listed, reaching 104.58 tok/s at NP1 and 175.56 tok/s aggregate at NP16.