Local model performance on your hardware

generation:

51.9 tokens/s

prompt processing (prefill):

1831.8 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 51.92 t/s generation and 1831.78 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/zanzibar.txt # NVIDIA RTX A2000 12GB: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA RTX A2000 12GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp 82764d8f4 (8770), backend CUDA. pp512 1831.78 ±13.79 tokens/s; tg128 51.92 ±0.39 tokens/s. One device, ngl99; 11906 MiB visible VRAM, not resident use. CPU and system RAM unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/zanzibar.txt .

generation:

5.5 tokens/s

prompt processing (prefill):

118.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 5.5 t/s generation and 118 t/s prompt eval at 131,072 context on an RTX A2000 12GB with a Xeon Silver 4314 and 128 GB DDR4. Setup is llama.cpp master with the unsloth UD-Q4_K_XL GGUF (103.7 GiB), q8_0 KV cache, --n-cpu-moe 99 and -ub 1536, all routed experts in RAM, 11.6 GB VRAM peak. Generation falls from about 15.4 t/s empty to 8.8 t/s at 58k and 5.5 t/s at 118k. A 3000x2000 image request took 46 s. The same box also ran Qwen3.6-35B-A3B at 26.1 t/s and DeepSeek-V4-Flash at 7.5 t/s at 118k deep.

generation:

26.1 tokens/s

prompt processing (prefill):

233.0 tokens/s

quant:

UD-Q5_K_M (GGUF)

kv:

q8_0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 26.1 t/s generation and 233 t/s prompt eval at 131,072 context on an RTX A2000 12GB with a Xeon Silver 4314 and 128 GB DDR4. Setup is llama.cpp master with the unsloth UD-Q5_K_M GGUF, q8_0 KV cache, --n-cpu-moe 32, MTP, and mmap on. Empty-context generation was 35-38 t/s; the 118k-deep figure was measured at --n-cpu-moe 31. A 3000x2000 image request took 32 s, and the user notes that --n-cpu-moe 31 crashed on that image while 32 passed.