Local model performance on your hardware

generation:

80.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 E2B at 80 tok/s on an RTX 3050 6GB laptop with 16GB RAM. No engine, quant or context length is stated. User is satisfied with the speed but says the model sometimes mishandles tools and hallucinates text in screenshots, and asks whether any good fine-tune of Gemma E2B exists for a local Jarvis-like workflow.

generation:

30.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B-A3B at 30 t/s on an RTX 3050 6GB with 24GB system RAM, dropping to around 22 t/s once the 64K context is full. Setup uses the Unsloth Q4_K_M quant with 64K context; the model is partially offloaded to system RAM. User is building a voice agent with Moonshine Small and Faster-Whisper STT/TTS and asks for better voice model combinations and a faster alternative to Qwen3.6 MoE.

generation:

10.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at over 10 t/s on an RTX 3050 6GB with 16GB DDR4 system RAM. Setup uses the Ultra-Uncensored-Heretic IQ4_XS GGUF. User is using it for translation and finds the results decent.