NVIDIA RTX 3060 Ti 8GB

generation:

18.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Swift-1.5-Qwen3.8-Flash-Next at IQ2_XS on an RTX 3060 Ti 8GB with 32GB system RAM, getting 18 t/s generation at 128k context. Setup is Strata with IQ2_XS quant, int8 KV cache, and expert offloading to system RAM. Prefill is 27 t/s with MTP enabled (334 experts cached in GPU) and 33 t/s with MTP off (1027 experts cached in GPU). User is happy with generation speed given the low VRAM but is struggling with prefill performance and is asking for suggestions.

generation:

27.0 tokens/s

prompt processing (prefill):

475.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next IQ3_XXS at ~27 tok/s decode and ~475 tok/s prefill on an RTX 3060 Ti 8GB. Setup is Strata 0.1.44 on CachyOS with 128K context, running on an i7-4930K with 50 GB DDR3; the model is partially offloaded to system RAM. About 7 s to first token per turn. A second GPU (RTX 2080 Ti) is planned but not yet installed.

generation:

120.0 tokens/s

quant:

W4A16 (safetensors)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 120 t/s on Qwen3.8-27B (Swift fine-tune) with a 150k context window on four RTX 3060 Ti 8GB cards. Setup is vLLM with W4A16 weights, bf16 KV cache, tensor parallel across 4 GPUs, and MTP speculative decoding; cards are power-limited to 110W. Quantizing the KV cache to kv8 extends context to the full 262k but drops speed to around 70 t/s. User also reports running 2 concurrent agents with little slowdown, and previously got about 70 t/s on a Swift-Qwen3.8-27B EXL3 4.00bpw model with MTP and 196k context under ExLlamaV3.