Local model performance on your hardware

generation:

13.0 tokens/s

quant:

Q4_K_M (GGUF)

User compares Swift-1.5-Qwen3.8-27B at around 13 tokens/sec against Strata on an RTX 4070 12 GB plus RTX 3060 12 GB with 32GB DDR4, running the same instruction runsheet. Qwen3.8-27B runs Q4_K_M; Strata runs IQ3_XXS with experts streamed from an experts.bin file on an M.2 SSD, context varied from 96k to 160k, KV cache tried as Int8, q8v4 and q4_0, kv-resident 20480 to 32768. Strata reached around 17 tokens/sec, 31-36 t/s for plain text generation and 20-25 t/s for logic/coding, with 5700 to 6800 experts in VRAM and system RAM maxed. Lowering kv-resident raised experts in VRAM but ran 3-4 t/s slower. User asks where the bottleneck is and which low-RAM flags to use.

generation:

29.6 tokens/s

User reports Mixtral 8x7B, Qwen-MoE and DeepSeek-MoE MoE models running through Ollama on an RTX 4070 12GB at 29.6 tok/s, up from 20.2 tok/s with a custom PyTorch orchestration layer (NPOL Engine) that prefetches experts from pinned host RAM. Setup offloads expert weights to system RAM over PCIe Gen4 x16, with a rank-32 lookahead probe staging predicted experts on a non-blocking CUDA stream and a thermal guard shedding tokens to resident experts. The user reports a 67.0% VRAM cache hit rate and 35.8% PCIe stall time saved, with 99.55% output cosine similarity preserved. Other cards: RTX 4080 Laptop 12GB 19.6 to 28.9 tok/s, RTX 3060 12GB 19.2 to 28.5 tok/s with temperature dropping from 82.5C to 58.0C, RTX 4090 24GB 27.6 to 37.2 tok/s.

generation:

73-75 tokens/s

quant:

UD-Q3_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 73-75 tok/s generation and 1,390 tok/s prompt processing on a laptop with an RTX 4070 8 GB, i7-14700HX and 32 GB RAM. Setup is PolyStrata with UD-Q3_K_M GGUF, a 30,000-token prompt, CPU offload of experts and speculative decoding. The same model on an RTX 5090 32 GB writes 332-334 tok/s and reads 5,520-5,580. Gemma 4 26B-A4B, Ornith 1.5 and GLM 5.3 Flash were also measured on both machines; GLM 5.3 Flash does not fit in 32 GB of RAM.

generation:

28.7 tokens/s

quant:

Q3_K_P (GGUF)

kv:

q8_0

User reports Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive at 28.74 t/s generation on an RTX 4070 with a Ryzen 7 7700X and 32GB DDR5, roughly half the speed they get from llama.cpp. Setup is ik_llama.cpp with a Q3_K_P GGUF, q8_0 KV cache, 131072 context, and 12 threads; the same command under llama.cpp gave 63.82 t/s generation and 70.52 t/s prompt eval. The user is asking what could be done about the shortfall.

generation:

69.0 tokens/s

quant:

IQ2_XS (GGUF)

User reports Qwen3.8-Flash-Next at 69 t/s at 98k tokens on an RTX 4070 12GB and Tesla V100 SXM2 16GB. Setup uses IQ2_XS quant with 32GB DDR4 RAM and a 3500MB/s SSD, with CPU offload. Speeds scale from 37 t/s fresh to 69 t/s at 98k tokens. User also gets 45-50 t/s on a 27B Q4_K_M model and asks whether IQ3_XXS would fit.

generation:

15.7 tokens/s

quant:

IQ3_XXS (GGUF)

User reports Qwen3.8 Flash-Next at 15.7 t/s on an RTX 4070 12GB and V100 16GB with 32GB of system RAM. Setup is llama.cpp with IQ3_XXS weights, 98k context, flash attention, jinja, and MTP with 2 draft tokens, using layer split rather than tensor parallelism. The user asks whether the model should require 45.8 GB in memory as shown for an AD-3.84bpw-IQ4_XS-M64 build, and notes their own RAM usage stayed around 6 GB.

generation:

20-40 tokens/s

User asks for advice on orchestrating a local Qwen 3.8 27B GGUF model with cloud subscriptions (Claude, Codex, Antigravity) to avoid burning paid tokens. Setup is Unsloth Desktop on a remote rig with dual GPUs (RTX 4070 + RTX 5060 Ti), tunneled through Cloudflare, with a 120k context window. User reports output of around 20–40 tokens/s.

generation:

20.6 tokens/s

prompt processing (prefill):

300-350 tokens/s

quant:

4.27 bpw (GGUF)

mtp (multi-token prediction):

on

User reports Qwen3.8-Flash-Next (125B-A6B MoE) at 20.65 t/s generation on an RTX 4070 12GB with 64GB DDR5-5600 and Gen4 NVMe on Linux. Setup is llama.cpp with AtomicChat's 4.27 bpw GGUF, n-gram SSD offloading, --fit-target 512, and an MTP variant (PR #28243 + compact shared-Q4_K_M head) with -ncmoe 45, yielding 77-96% acceptance. Prompt processing is 300-350 t/s. Baseline was 6 t/s; master branch reached 19.35 t/s. User notes MTP gives limited gains at low VRAM because layers must be freed for the MTP head.

generation:

7.2 tokens/s

User reports Qwen3.8-Flash-Next at 7.19 t/s on an RTX 4070 Ti 12GB with 128GB DDR4 RAM. Setup is the Nebula C/CUDA engine with native MTP speculative decoding, GPU expert caching (27 of 512 experts per layer resident in VRAM), and CPU MoE execution on an Intel i9-9940X using 14 threads, with a 24,576-token context capacity. A limited-tolerance acceptance mode reaches 7.54 t/s. Time to first token is 25.11 seconds for a 512-token input and 113.94 seconds for 2,048 tokens. Prefill is noted as slow for longer prompts.

generation:

11.7 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8

mtp (multi-token prediction):

on

User reports 12.55 t/s for a 256 token response and 11.65 t/s for a 512 token response on an RPC split across an RTX 4070 Ti and an M5 MacBook Air with 16 GB unified memory. MTP off gives 9.72 t/s. The use case is agentic coding.

quant:

IQ4_XS (GGUF)

kv:

Q4

User reports running Qwen3.8-27B on 16 GB of VRAM with the IQ4_XS quant. Setup is MTP disabled, mmproj offloaded to CPU, and a Q4 KV cache, at roughly 100k context. User notes Windows VRAM overhead as a constraint.

generation:

55.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports about 55 t/s on an RTX 4070 12GB. The figure is estimated from compute-market tiers rather than a measured run.