Local model performance on your hardware

prompt processing (prefill):

2080.0 tokens/s

quant:

AWQ-INT4 (AWQ)

kv:

FP8

User benchmarks Qwen3.8 27B (Swift-1.5, AWQ-INT4) on dual RTX 3080 20GB cards in tensor parallel, reporting 2080 t/s prefill at 8192 prompt tokens with gpu-unlock plus P2P versus 967.1 t/s stock, a 115% gain. Setup is vLLM 0.32.0 with FP8 KV cache, MTP speculative decoding with 3 tokens, and both cards limited to 240W in PCIe 3.0 x16 slots. Prefill gains range from 70% to 115% across prompts of 8192 to 261900 tokens. Decode is reported as inter-token latency step rates of about 15 to 16.6 steps/s, which the user says corresponds to roughly 45 tok/s at an MTP acceptance length of 2; decode changes are within noise except at 261k tokens.

prompt processing (prefill):

1168.0 tokens/s

quant:

AWQ-INT4 (AWQ)

kv:

FP8

User benchmarks Swift-1.5-Qwen3.8-27B-AWQ-INT4 on two RTX 3080 20GB cards with the new GPU performance unlock, reporting +20.8% prefill at 8K (967 to 1168 tok/s) and no decode change. Setup is vLLM 0.32.0 with tensor parallel 2, FP8 KV cache, MTP enabled with 3 speculative tokens, 240W power limit per card. Prefill gains grow with prompt length, up to +28.6% at 261,900 tokens (562 to 723 tok/s). Decode is reported as steps/s (about 16), which the user says works out to roughly 45 tok/s in real agentic coding with MTP acceptance around 2. Both GPUs thermally throttled to 1750MHz.

generation:

143.0 tokens/s

prompt processing (prefill):

889.0 tokens/s

quant:

W4A16 (safetensors)

kv:

int8_per_token_head

User reports Qwen3.8-27B (Swift-1.5-HyperQwen W4A16) at 143 t/s decode and 889 t/s prefill at 32K context on 4x RTX 3080 20GB (modded) with tensor parallelism 4. Setup is vLLM 0.29.0 with W4A16 weights, int8_per_token_head KV cache, DFlash2 speculative decoding with 7 spec tokens, max_model_len 262144, max_num_seqs 4, chunked prefill and prefix caching enabled. Single-task decode ranges from 143 t/s at 32K to 240 t/s at 2K context; two concurrent tasks drop to 17-115 t/s. DFlash2 draft acceptance is 64.9% with 5.54 tokens/step. User previously got 40-50 t/s without DFlash2 and notes MTP degraded speeds.

generation:

57.7 tokens/s

prompt processing (prefill):

2500.0 tokens/s

quant:

AWQ-INT4 (AWQ)

User reports Qwen3.8 27B at 57.7 t/s decode on two RTX 3080 20GB cards (40GB pool) at 160k context. Setup is vLLM in Docker on Unraid with the AWQ-INT4 quant, tensor-parallel-size 2, max-num-seqs 1, max-num-batched-tokens 16384, and GPUs throttled to 250W. Prefill is roughly 2,500 t/s at 10k context with 4.23s TTFT. Decode drops to about 51 t/s at 42k context and 43 t/s at 110k context; lifetime average across 611 agent requests is 47 t/s. User asks whether speeds can be improved.