Local model performance on your hardware

generation:

12.9 tokens/s

prompt processing (prefill):

25.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source and attribution: u/LemesoftNostalgic, "Running some unsloth models on a dual-RTX 3060 workstation — exact setup + tok/s numbers", https://www.reddit.com/user/LemesoftNostalgic/comments/1wxc5tw/running_some_unsloth_models_on_a_dualrtx_3060/ . Primary source re-opened through the web reader on 2026-10-13; the returned primary-page snapshot was labelled crawled three days earlier. No direct live-HTTP timestamp is claimed. The page has only a relative post age; exact publication and measurement dates are unknown. This is an author-reported native benchmark, not an independently repeated run. Model: Qwen3.8-Flash-Next, GGUF, table quant IQ4_XS; source model-file size 87.24 GiB, source quant density 4.25 bpw. The source table calls this IQ4_XS at 4.25 bpw. The separate serving command names Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf. A matching current shard was opened at https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/main/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf . This is a filename match, not proof of the tested version. The upstream card separately describes 125B core parameters with 6B active, plus 51B n-gram embeddings and 4B MTP (180B HF displayed total); the benchmark post does not give an exact GGUF parameter count. Upstream/base-model reference only: https://huggingface.co/Qwen/Qwen3.8-Flash-Next . Tested GGUF revision and checksum are unknown. Hardware: two NVIDIA RTX 3060 GPUs totaling 24 GB nominal VRAM (2 x 12 GB mapping), each reported at PCIe Gen3 x16; HP Z440 workstation, Intel Xeon E5-2690 v3 (12 cores / 24 threads), 96 GB ECC DDR4 in four channels, 256 GB NVMe plus 1 TB SATA, 700 W PSU. Xubuntu 24.04 LTS, NVIDIA driver 580.173.02 and CUDA 13.0 are source-reported system details. Neither GPU power limit nor actual benchmark resident RAM/VRAM is reported. Native benchmark results: PP512 25 t/s and TG128 12.9 t/s. The author explicitly defines pp512 as llama-bench prompt processing and tg128 as llama-bench token generation. These are separate tests with 512 newly processed prompt tokens and 128 generated tokens respectively, not a single 512-input/128-output application request. Repetitions, statistic, variation, raw timings, exact benchmark invocation, backend, fork/build/commit, thread counts, batch/microbatch, KV types, flash-attention state, existing cache depth and configured maximum context are unreported. The separate Serving section mentions beellama.cpp, llama-swap, the pi coding agent, 180224-token server context and a 131072 pi maxTokens setting. Its server commands, quantized-cache options, tensor splits and speculative/MTP flags must not be assigned to the PP512/TG128 measurements. In particular, 180224 is not a verified benchmark context length. The separate fib column is llama-cli Fibonacci with MTP decode; its 11.7 t/s is excluded from the native PP/TG headline. Memory and comparability: the GiB value is a model-file size, not measured resident usage. The 87.24 GiB file exceeds the GPUs' combined nominal 24 GB VRAM. Exact CPU/system-memory placement and GPU offload during the benchmark are unreported. Do not borrow the serving command's -ncmoe 40, -ngl 49 or -ts 6,1, and do not claim full GPU residency.

generation:

30.8 tokens/s

prompt processing (prefill):

755.0 tokens/s

quant:

Q4_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source and attribution: u/LemesoftNostalgic, "Running some unsloth models on a dual-RTX 3060 workstation — exact setup + tok/s numbers", https://www.reddit.com/user/LemesoftNostalgic/comments/1wxc5tw/running_some_unsloth_models_on_a_dualrtx_3060/ . Primary source re-opened through the web reader on 2026-10-13; the returned primary-page snapshot was labelled crawled three days earlier. No direct live-HTTP timestamp is claimed. The page has only a relative post age; exact publication and measurement dates are unknown. This is an author-reported native benchmark, not an independently repeated run. Model: Qwen3.8 27B, GGUF, table quant Q4_K_S; source model-file size 14.29 GiB. The benchmark table calls the quant Q4_K_S; the separate serving command names Qwen3.8-27B-UD-Q4_K_S.gguf. A current public file with that exact filename was opened at https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_S.gguf . This is a verified filename match and plausible source, not proof of the tested GGUF or revision. Preserve Q4_K_S as the measured table label; do not silently replace it with a verified UD benchmark claim. Upstream/base-model reference only: https://huggingface.co/Qwen/Qwen3.8-27B . Tested GGUF revision and checksum are unknown. Hardware: two NVIDIA RTX 3060 GPUs totaling 24 GB nominal VRAM (2 x 12 GB mapping), each reported at PCIe Gen3 x16; HP Z440 workstation, Intel Xeon E5-2690 v3 (12 cores / 24 threads), 96 GB ECC DDR4 in four channels, 256 GB NVMe plus 1 TB SATA, 700 W PSU. Xubuntu 24.04 LTS, NVIDIA driver 580.173.02 and CUDA 13.0 are source-reported system details. Neither GPU power limit nor actual benchmark resident RAM/VRAM is reported. Native benchmark results: PP512 755 t/s and TG128 30.8 t/s. The author explicitly defines pp512 as llama-bench prompt processing and tg128 as llama-bench token generation. These are separate tests with 512 newly processed prompt tokens and 128 generated tokens respectively, not a single 512-input/128-output application request. Repetitions, statistic, variation, raw timings, exact benchmark invocation, backend, fork/build/commit, thread counts, batch/microbatch, KV types, flash-attention state, existing cache depth and configured maximum context are unreported. The separate Serving section mentions beellama.cpp, llama-swap, the pi coding agent, 180224-token server context and a 131072 pi maxTokens setting. Its server commands, quantized-cache options, tensor splits and speculative/MTP flags must not be assigned to the PP512/TG128 measurements. In particular, 180224 is not a verified benchmark context length. The separate fib column is llama-cli Fibonacci with MTP decode; its 45 t/s is excluded from the native PP/TG headline. Memory and comparability: the GiB value is a model-file size, not measured resident usage. The model-file size being smaller than combined VRAM does not establish full GPU residency or the benchmark split mode; both placement and resident usage remain unreported.

generation:

97.5 tokens/s

prompt processing (prefill):

1870.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source and attribution: u/LemesoftNostalgic, "Running some unsloth models on a dual-RTX 3060 workstation — exact setup + tok/s numbers", https://www.reddit.com/user/LemesoftNostalgic/comments/1wxc5tw/running_some_unsloth_models_on_a_dualrtx_3060/ . Primary source re-opened through the web reader on 2026-10-13; the returned primary-page snapshot was labelled crawled three days earlier. No direct live-HTTP timestamp is claimed. The page has only a relative post age; exact publication and measurement dates are unknown. This is an author-reported native benchmark, not an independently repeated run. Model: Ornith 1.5 35B-A3B, GGUF, table quant IQ4_XS; source model-file size 16.94 GiB, source quant density 4.25 bpw. The author explicitly says this is their own Ornith quant, similar to Unsloth UD-IQ4_XS. Do not label an Unsloth or ornith-ai GGUF as the measured artifact. The local serving filename is Ornith-1.5-35B-IQ4_XS.gguf; no public download, revision or checksum is established. Upstream/base-model reference only: https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B . Tested GGUF revision and checksum are unknown. Hardware: two NVIDIA RTX 3060 GPUs totaling 24 GB nominal VRAM (2 x 12 GB mapping), each reported at PCIe Gen3 x16; HP Z440 workstation, Intel Xeon E5-2690 v3 (12 cores / 24 threads), 96 GB ECC DDR4 in four channels, 256 GB NVMe plus 1 TB SATA, 700 W PSU. Xubuntu 24.04 LTS, NVIDIA driver 580.173.02 and CUDA 13.0 are source-reported system details. Neither GPU power limit nor actual benchmark resident RAM/VRAM is reported. Native benchmark results: PP512 1870 t/s and TG128 97.5 t/s. The author explicitly defines pp512 as llama-bench prompt processing and tg128 as llama-bench token generation. These are separate tests with 512 newly processed prompt tokens and 128 generated tokens respectively, not a single 512-input/128-output application request. Repetitions, statistic, variation, raw timings, exact benchmark invocation, backend, fork/build/commit, thread counts, batch/microbatch, KV types, flash-attention state, existing cache depth and configured maximum context are unreported. The separate Serving section mentions beellama.cpp, llama-swap, the pi coding agent, 180224-token server context and a 131072 pi maxTokens setting. Its server commands, quantized-cache options, tensor splits and speculative/MTP flags must not be assigned to the PP512/TG128 measurements. In particular, 180224 is not a verified benchmark context length. The separate fib column is llama-cli Fibonacci with MTP decode; its 115.2 t/s is excluded from the native PP/TG headline. Memory and comparability: the GiB value is a model-file size, not measured resident usage. The model-file size being smaller than combined VRAM does not establish full GPU residency or the benchmark split mode; both placement and resident usage remain unreported.

generation:

15.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 15 t/s on 2x RTX 3060 12GB with 64GB RAM, compared to Strata's 35 t/s on thinking and 50 t/s on code. Setup is vanilla llama.cpp with IQ3_XXS and no MTP drafter, as the only available MTP Q4_K_M drafter is 3.5x larger and slows decode. User asks for a working llama recipe, noting tensor split fails with a cudaMalloc error on device 0.

generation:

20.1-21.1 tokens/s

quant:

IQ2_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 125B-A6B at 20.14-21.13 tok/s on an RTX 3060 12GB with 16GB DDR4 RAM, using a custom llama.cpp build with --moe-direct-io prefetching. Setup is llama.cpp with IQ2_XS quant and a 68GB model file, run under a 6G cgroup memory limit; the model is streamed from NVMe SSD with no expert pruning and bit-exact output. Stock llama.cpp on the same machine gives 1.41-2.12 tok/s with 1,140-1,565 major page faults per token; warm cache runs reach 24+ tok/s, cold starts begin at 3.5-4.5 tok/s, and prompt processing remains slow.

generation:

42.0 tokens/s

prompt processing (prefill):

684.9 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Glimmer 30B at 41.97 t/s generation and 684.86 t/s prompt processing on an RTX 3060 12GB plus RTX 3070 8GB. Setup is llama.cpp with IQ4_XS weights, Q8_0 KV cache, 65,536-token context, Flash Attention on, and a DFlash draft model (Q4_0) on CUDA1 with a 45/55 layer split. The benchmark prompt was 994 tokens, not a filled 65K context; generation ranged 41.71–42.45 t/s over 128 output tokens, with 4.5 s average request time and 9.7 s model load.

generation:

56.0 tokens/s

prompt processing (prefill):

497.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 56.0 tok/s decode and 497 tok/s prefill on 2× RTX 3060 12GB. Setup is llama.cpp CUDA with UD-Q4_K_XL GGUF, q4_0 KV cache, 131072 context, tensor split across the two cards, and MTP speculative decoding. The 106,294-token context gate took 213.8 s. A second recipe runs Qwen3.6-35B-A3B on the same machine at 72.6 tok/s decode and 512 tok/s prefill with q8_0 KV cache and layer split, keeping some expert tensors in system RAM (8.22 GiB host RAM drop).

generation:

47.3 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 47.3 tok/s decode on 2x RTX 3060 12GB. Setup is llama.cpp (server-cuda12-b11146) with UD-Q4_K_XL GGUF, tensor split, auto-fit and the GGUF's built-in MTP head, at 131,072 tokens of context. TTFT 537 ms. The same PR also validates 20 single-card RTX 3060 12GB recipes and adds a second two-card recipe, Qwen3.6-35B-A3B at 74.2 tok/s decode and 147 ms TTFT at 131,072 tokens. Long-context recall was tested at 104,941 tokens. Several candidates failed (missing quant tag pulling default Q4_K_M, empty model path, OOM) and 16 were skipped for 512-2,048-token contexts.

generation:

30.0 tokens/s

quant:

iq2_xxs (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Strata at iq2_xxs quant running at about 30 t/s on 2x RTX 3060 12GB, comparable to vanilla llama.cpp with Qwen3.8 27B. Peak speeds reach up to 50 t/s when working with code. Setup is llama.cpp with iq2_xxs quant, 48GB system RAM, Ryzen 5 3600, and Pi Harness. User notes the larger model offers better quality even at this quant, but it loops badly and fails to complete meaningful tests, becoming indecisive and going in circles without producing output.

generation:

10.0 tokens/s

prompt processing (prefill):

150.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 10 t/s decode and 150 t/s prefill on an RTX 3060 12GB with 128 GB RAM. Setup is llama.cpp with IQ4_XS GGUF and q8_0 KV cache at 200k context, with 48 MoE layers offloaded to CPU. The post is a general inference engineering guide; the figures come from the user's sample launch command for this model.

generation:

9.7 tokens/s

prompt processing (prefill):

225.0 tokens/s

quant:

Q4_K_S (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~9.7 tok/s generation and ~225 tok/s prompt processing on a single RTX 3060 12GB. Setup is llama.cpp with Q4_K_S weights and Q4_0 KV cache at 96K context, using MTP speculative decoding with 46 FFN layers offloaded to system RAM. MTP acceptance was ~88% with 2.76 mean accepted tokens; peak generation reached ~10.6 tok/s and VRAM usage was ~11.04 GB.

generation:

29.6-29.7 tokens/s

prompt processing (prefill):

82.1-320.2 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 29.65-29.69 generated tokens/s on one RTX 3060 12 GB, with a 33.21 tokens/s high-acceptance peak. Setup is llama.cpp with IQ3_XXS GGUF, Q4_0 KV cache, 65,536-token context, MTP depth 6, and Deferred-Commit Factor Replay. A separate prefill sweep improved an 8,595-token prompt from 82.13 to 320.24 prompt tokens/s. Q4 at 32K measured 16.06 tok/s on a Rust task and 14.30 tok/s on a C++ task; Q3 at 96K and 128K measured 21.3-22.0 and 19.7-20.9 tok/s respectively.

quant:

Q4_K_M (GGUF)

User reports running Qwen3.8-27B on a 2x3060 machine with llama.cpp, achieving generation speeds ranging from 43.37 to 49.99 t/s and prompt processing speeds from 45.01 to 198.10 t/s. Setup uses Q4_K_M quantization, 50000 context length, tensor split, flash attention, and speculative decoding with ngram-mod and draft-mtp. User notes the main limitation is short context, which is too short for long agentic sessions.

generation:

25.0 tokens/s

quant:

Q6 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 25 t/s on three RTX 3060 12GB cards. Setup is llama.cpp in layered mode with a Q6 quant. The figure is a current baseline; the user asks whether moving to four or eight RTX 3060s under vLLM would improve or hurt token generation.

generation:

29.0 tokens/s

quant:

IQ3-XXS (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at ~29 t/s on an RTX 3060 12GB at around full context, with 34–40 t/s at lower context. Setup is llama.cpp on CachyOS/Arch Linux with MTP/speculative decoding, comparing the ISTA-DASLab GSQ-RCO-IQ3-XXS quant (~10.4GB, ~2.5 BPW) against the ByteShape IQ3-XXS quant (~500MB smaller, 2.88 BPW). The ByteShape quant performed worse in practice: the GSQ version generated a 3D voxel diorama in one shot under 55k tokens, while ByteShape took roughly three shots and still hadn't completed after over 98K tokens.

generation:

29.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B GSQ-RCO-IQ3-XXS at 29 t/s on an RTX 3060 12GB at full context, and 40 t/s at near-empty context. Setup is llama.cpp with MTP speculative decoding and high thinking enabled, using a 122K context window. The run used roughly 75K tokens and the context was compacted twice; the user rates this as the most impressive local model in a nine-model web-dev comparison.

generation:

11.4 tokens/s

prompt processing (prefill):

337.9 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 10-20 t/s on an RTX 3090 at 65,536 context, with a detailed log showing tg=11.44 t/s. Setup is llama.cpp with the Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf file, a q4_0 KV cache for both K and V, and MTP speculative decoding with draft acceptance 0.4125. Prompt processing runs ~334-338 t/s. The title says IQ3 XXS while the body also mentions IQ3_S - 3.4375 bpw. More than 1 GB of VRAM is left after loading.

generation:

38.4 tokens/s

quant:

IQ3_XXS

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 38.4 t/s and Qwen3.6 35B-A3B (MoE) at 55.9 t/s on an RTX 3060. Setup is llama.cpp with tuning, at 16K context on Ubuntu and 12K context on WSL2. The user notes editing speeds of about 188 t/s and is frustrated at not reaching 50-60 t/s for the 27B model.

generation:

27-29 tokens/s

prompt processing (prefill):

290-340 tokens/s

quant:

IQ2_M

kv:

Q8

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next at 27-29 t/s generation and 290-340 t/s prefill across 4 GPUs using 52 GB of VRAM. Setup is the IQ2_M quant with context offloaded at Q8 and an n-gram cache on SSD. User reports roughly 98% clean Finnish output, but notes safety guardrails cause freezes in grey areas.

generation:

12.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running llama.cpp's ggml-rpc backend across a heterogeneous cluster pooling RAM and VRAM from an Acer laptop CPU, a Windows RTX 3060 on CUDA, and a Mac Mini on Metal. The primary API server runs on the weakest machine, and the setup uses llama.cpp's built-in benchmark script. The project is source-available under the Commons Clause.