NVIDIA RTX 3060 12GB reports

generation:

43.3 tokens/s

prompt processing (prefill):

456.1 tokens/s

quant:

Q4_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 43.26 t/s generation and 456 t/s prefill at 12k context on dual RTX 3060 cards. Setup is tensor parallel with MTP enabled and 64k context. Without MTP at 96k context, generation is 31 t/s. User praises the value and stability of CUDA.

generation:

70.0 tokens/s

prompt processing (prefill):

780.0 tokens/s

quant:

Q2-XS (GGUF)

kv:

Q8

flash attention:

off

rating:

4/5

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a dense model at 11.9 GB GGUF size, barely offloading to CPU. The user wants to try a higher quant, having chosen this one because of the 11.9 GB GGUF size.

generation:

46.8 tokens/s

prompt processing (prefill):

914.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports llama-bench results of about 914 t/s for pp512 and about 46.8 t/s for tg128. A practical coding profile at 32k context generates at about 43.4 t/s. MTP gave about 47.7 t/s, a 2% improvement.

generation:

60.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports E2B at about 60 t/s on an RTX 3060 12GB. The figure is estimated from compute-market tiers rather than measured.

generation:

45.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 45 t/s on an RTX 3060 12GB. The E4B model fits easily.