Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 40B model (~25GB) running at 16 t/s across a 3-device cluster: a mini PC with an RTX 3060, a 16GB Mac Mini, and an old laptop.
Setup uses the user's own ramdeck-core pooling repo, sharding the model across the 3060 first, then the Mac's unified memory, then CPU spillover routed to the old laptop.
The full load took about 2.5 minutes and latency was roughly 3.6s. User notes that offloading CPU spillover to the slower laptop beat keeping it on the mini PC's own CPU, and asks whether others have seen the same GPU+CPU contention effect.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 27B model at about 16 GB sharded across 4 devices via RAMDeck, reaching 1.92 tok/s at roughly 25 ms latency.
The devices are an old 12 GB Windows laptop as primary with 3.4 GB, a Mini PC with an RTX 3060 with 20 GB, a Mac mini with 3.7 GB, and an Android phone with 1 GB. The model family is not named.
The user notes the run is slower than a prior 13B run and emphasizes feasibility over speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 10-20 t/s on an RTX 3090 at 65,536 context, with a detailed log showing tg=11.44 t/s.
Setup is llama.cpp with the Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf file, a q4_0 KV cache for both K and V, and MTP speculative decoding with draft acceptance 0.4125. Prompt processing runs ~334-338 t/s.
The title says IQ3 XXS while the body also mentions IQ3_S - 3.4375 bpw. More than 1 GB of VRAM is left after loading.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 20 t/s split between an RTX 3060 12GB and an RX 9070 XT, and at 5 t/s on a 780M iGPU with 5400MHz DDR5.
The 20 t/s figure is for the two-GPU split configuration and the 5 t/s figure is for the iGPU.
User prefers the 5 t/s iGPU setup for system usability.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 38.4 t/s and Qwen3.6 35B-A3B (MoE) at 55.9 t/s on an RTX 3060.
Setup is llama.cpp with tuning, at 16K context on Ubuntu and 12K context on WSL2.
The user notes editing speeds of about 188 t/s and is frustrated at not reaching 50-60 t/s for the 27B model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
general-conversation
User reports Qwen3.8 Flash Next at 27-29 t/s generation and 290-340 t/s prefill across 4 GPUs using 52 GB of VRAM.
Setup is the IQ2_M quant with context offloaded at Q8 and an n-gram cache on SSD.
User reports roughly 98% clean Finnish output, but notes safety guardrails cause freezes in grey areas.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User benchmarks a 25-task coding-agent benchmark on an RTX 3060 12GB. KAT-Coder completed 21/25 tasks and won.
Qwen3.8-27B scored 13/25 at 14.73 tps, Nemotron 3.5 Lightning 30B-A3B 8/25 at 29.96 tps, Tiel-Coder 35B-A3B 3/25 strict at 20.57 tps, and Ternary-Bonsai-27B 0/25 at 59.21 tps.
User plans to tweak settings to reach 18 tps.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running llama.cpp's ggml-rpc backend across a heterogeneous cluster pooling RAM and VRAM from an Acer laptop CPU, a Windows RTX 3060 on CUDA, and a Mac Mini on Metal.
The primary API server runs on the weakest machine, and the setup uses llama.cpp's built-in benchmark script.
The project is source-available under the Commons Clause.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.6 27B at ~24 t/s on dual RTX 3060 GPUs.
The user also discusses diffusion-based techniques, MOE, quants, and token authority as a wishlist for future local models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports a dual RTX 3060 setup with RAM offloading, a Ryzen 7500F and 96 GB of 5600 RAM, writing a tetris game.
Prompt eval runs at 3.0 tok/s. Generation is reported as 4.5 tok/s in the summary but 3.5 tok/s in PowerShell, and the user trusts PowerShell.
LM Studio failed to offload to the second GPU, so the user used Unsloth Studio.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.6 35B-A3B at about 70 t/s on an RTX 3060.
The figure is given as past experience rather than a benchmark run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports about 40 t/s with a Q4_K_XL quant on 2x RTX 3060 12GB.
Setup is llama.cpp with flash attention and speculative decoding.
The model wrote CUDA code but could not run it due to VRAM limits.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a dual RTX 3060 12GB setup with tensor split.
MTP draft accept is 4/4 (100%) on probe.
VRAM usage is 23.2/24.0 GiB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports building a game in two prompts with Qwen 3.8 on 2x RTX 3060.
Setup is llama.cpp with the UD-Q4_K_XL quant, 128k context, and a quantized cache at k5_0/v4_1.
The user notes that 32 GB of system RAM caused issues after context compaction, which a 'continue' resolved.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at 10.1 t/s generation and 99.4 t/s prompt processing on 4x RTX 3060 12GB at 368k context.
Setup is llama.cpp build b10181 with a Q8_0 KV cache and a microbatch size of 2048, with the 144 GiB GGUF mostly in 128GB of system RAM.
Runs at 376832 and 360448 context gave similar speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports a llama.cpp RPC cluster with tensor-split 20,23.
Setup uses MTP speculative decoding with 84.37% draft acceptance and a 96.14% prefix cache hit ratio.
Max context was stress-tested at 72,712/77,000 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 503.7 t/s prefill and 27.7 t/s decode on dual RTX 3060 12GB cards.
Setup is a Q4_K_M GGUF with a Q8_0 KV cache at 131k context, BF16 vision projector on CPU, xHigh reasoning, on a Ryzen 5 5500 with 64GB DDR4-3200 and GPUs limited to 130W each.
With speculation the user measured about 503 t/s prefill and about 29 t/s decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a 125B MoE model with 6B active on 2x RTX 3060 12GB in llama.cpp, with prefill improving from 36 to 303 t/s and decode at 12 t/s.
Setup uses -sm layer and -ub 2048.
ik_llama.cpp was also tested at 407 t/s prefill but uses ~108GB RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Flash Next at 12-15 t/s on an RTX 3060 12GB in a Thinkstation P520 with 256GB RAM, at 65k context.
Setup is llama.cpp server with the model offloaded to GPU at --n-gpu-layers 999 and MoE experts on CPU at --n-cpu-moe 48.
The user notes the model is slow and low context, but output quality is much better than the previous Qwen 3.6 35B A3B. Performance degrades significantly when the system is busy.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 7-8 t/s with 2x3060 and RAM offloading, but 4.5-5.5 t/s with 3x3060.
User asks why adding a GPU slows down generation, and notes there is no tensor parallelism in both chats.
User asks about enabling ngram or mlock.