Local model performance on your hardware

prompt processing (prefill):

2080.0 tokens/s

quant:

AWQ-INT4 (AWQ)

kv:

FP8

User benchmarks Qwen3.8 27B (Swift-1.5, AWQ-INT4) on dual RTX 3080 20GB cards in tensor parallel, reporting 2080 t/s prefill at 8192 prompt tokens with gpu-unlock plus P2P versus 967.1 t/s stock, a 115% gain. Setup is vLLM 0.32.0 with FP8 KV cache, MTP speculative decoding with 3 tokens, and both cards limited to 240W in PCIe 3.0 x16 slots. Prefill gains range from 70% to 115% across prompts of 8192 to 261900 tokens. Decode is reported as inter-token latency step rates of about 15 to 16.6 steps/s, which the user says corresponds to roughly 45 tok/s at an MTP acceptance length of 2; decode changes are within noise except at 261k tokens.

generation:

45.0 tokens/s

quant:

Q6 (GGUF)

User reports running Qwen3.8 27B Q6 with 210k context on two modded 20GB RTX 3080s, generating at ~45 t/s with prefill at ~500 t/s. The 40GB of VRAM was filled by the model and context; the build cost $2250 plus $250 for an extra 64GB of RAM, and has run for 6 weeks. Both throughput figures are approximate single readings. The user's goal is a local coding agent with long full-precision contexts and several agents at once.

prompt processing (prefill):

1168.0 tokens/s

quant:

AWQ-INT4 (AWQ)

kv:

FP8

User benchmarks Swift-1.5-Qwen3.8-27B-AWQ-INT4 on two RTX 3080 20GB cards with the new GPU performance unlock, reporting +20.8% prefill at 8K (967 to 1168 tok/s) and no decode change. Setup is vLLM 0.32.0 with tensor parallel 2, FP8 KV cache, MTP enabled with 3 speculative tokens, 240W power limit per card. Prefill gains grow with prompt length, up to +28.6% at 261,900 tokens (562 to 723 tok/s). Decode is reported as steps/s (about 16), which the user says works out to roughly 45 tok/s in real agentic coding with MTP acceptance around 2. Both GPUs thermally throttled to 1750MHz.

generation:

21.0 tokens/s

quant:

Q8 (GGUF)

User reports Qwen3.8-27B at about 21 t/s on two modded RTX 3080 20GB cards, with MTP raising it to 33 t/s. Setup is llama.cpp with Q8 weights and 131.1k context plus vision, run on the two 3080s; the 3080s are power limited to 220W. The user also ran Gemma 4-26B-A4B on a single RTX 4070 Ti 12GB and found coding output poor, and has just reinstalled the 4070 Ti so has limited data with all three cards.

generation:

31.3 tokens/s

prompt processing (prefill):

997.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8_0

User benchmarks Qwen3.8 27B at ~31.3 t/s generation and ~997 t/s prompt processing on a single modded RTX 3080 20GB. Setup is llama.cpp with Q4_K_XL GGUF and Q8_0 KV cache at ~98k context. Native MTP raises generation to ~56.9-60.7 t/s but cuts context to ~68k. Adding an RTX 3070 8GB via layer split reaches ~223k context but drops generation to ~27.8 t/s; tensor parallelism over the chipset x4 slot hits ~59.7 t/s generation with a ~20% prompt-processing penalty, while moving the second card to a CPU-attached Oculink x4 link recovers ~976-1,005 t/s prompt processing at ~60.4 t/s generation and ~204k context.

generation:

105.0 tokens/s

prompt processing (prefill):

5000.0 tokens/s

quant:

IQ3_XXS (GGUF)

User reports Qwen3.8 Flash-Next 125B at 105 t/s generation and 5000 t/s prompt processing on 4x RTX 3080 20GB modded cards. Setup is Strata with IQ3_XXS quant at 150000 context length, running via docker-compose on Unraid. User says they are absolutely impressed and tested with a Hermes agent at 150k context.

generation:

143.0 tokens/s

prompt processing (prefill):

889.0 tokens/s

quant:

W4A16 (safetensors)

kv:

int8_per_token_head

User reports Qwen3.8-27B (Swift-1.5-HyperQwen W4A16) at 143 t/s decode and 889 t/s prefill at 32K context on 4x RTX 3080 20GB (modded) with tensor parallelism 4. Setup is vLLM 0.29.0 with W4A16 weights, int8_per_token_head KV cache, DFlash2 speculative decoding with 7 spec tokens, max_model_len 262144, max_num_seqs 4, chunked prefill and prefix caching enabled. Single-task decode ranges from 143 t/s at 32K to 240 t/s at 2K context; two concurrent tasks drop to 17-115 t/s. DFlash2 draft acceptance is 64.9% with 5.54 tokens/step. User previously got 40-50 t/s without DFlash2 and notes MTP degraded speeds.

generation:

57.7 tokens/s

prompt processing (prefill):

2500.0 tokens/s

quant:

AWQ-INT4 (AWQ)

User reports Qwen3.8 27B at 57.7 t/s decode on two RTX 3080 20GB cards (40GB pool) at 160k context. Setup is vLLM in Docker on Unraid with the AWQ-INT4 quant, tensor-parallel-size 2, max-num-seqs 1, max-num-batched-tokens 16384, and GPUs throttled to 250W. Prefill is roughly 2,500 t/s at 10k context with 4.23s TTFT. Decode drops to about 51 t/s at 42k context and 43 t/s at 110k context; lifetime average across 611 agent requests is 47 t/s. User asks whether speeds can be improved.

generation:

39.0 tokens/s

quant:

Q4

User reports Qwen3.8 27B at 39 t/s on dual RTX 3080 20GB GPUs. Setup uses Q4 quantization with 180k unquantized context on a home server built for just under $2000. The user notes the cards have coil whine during decode and that the motherboard could support up to 4 GPUs at 16 lanes or 7 GPUs at 8 lanes.

generation:

25.0 tokens/s

prompt processing (prefill):

870.0 tokens/s

quant:

4.05 EXL3

User reports CPU-offloaded inference with 128 GB of system RAM, running 3.2x faster prefill and 2x faster decode than llama.cpp. User also tested GLM 5.3 Flash with 3.05 EXL3, which ran 2x slower in decode than llama.cpp. Decode speeds warm up over time.

generation:

17.5 tokens/s

prompt processing (prefill):

273.2 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

kv:

q8_0

User reports 17.53 t/s generation and 273.21 t/s prefill at 128k context on 2x RTX 3080 20GB with 64GB DDR5. Setup is the llama.cpp fork by fairydreaming with an 86.7 GB model, --n-cpu-moe 28 and split-mode layer.

generation:

20.0 tokens/s

quant:

Q4_K_M (GGUF)

User asks which GPU to buy to run Qwen 3.8 27B Q4_K_M at over 20 t/s. Options under consideration are an RTX 3080 20GB, an AMD Mi50 32GB, and a modified 2080 Ti 22GB. No benchmark results yet.

generation:

11.5 tokens/s

prompt processing (prefill):

300.0 tokens/s

quant:

IQ2_M (GGUF)

User reports a local model reaching 100% on an SQL benchmark, running on dual RTX 3080 20GB cards with 96GB RAM and a 9800X3D. Setup is a custom IQ2_M GGUF with grafted tensors from antirez GGUF, running on a modified ds4 engine. Mainline llama.cpp gives 100pp and 8tg.