Local model performance on your hardware

generation:

93.0 tokens/s

quant:

mix_q3gud_im (GGUF)

kv:

q4_0

User reports MiniCPM-V 4.7 35B-A3B at about 93 tok/s on a Tesla V100 16GB under Windows 11, with vision and 128k context. Setup is stock llama.cpp with OpenBMB's official mix_q3gud_im quant (15.5 GB), q4_0 KV cache, flash attention on, and the vision encoder kept on CPU so all MoE experts fit on the GPU. A 120k-token prompt was answered correctly without a crash, using 16.17 of 16.38 GB VRAM. At 262k context generation drops to 70 tok/s. DSpark n-gram speculative decoding was slower because the drafter's VRAM pushed experts back to RAM. The user also benchmarked 15 models on a 72-task code-graded suite, where this model scored 87.5 overall.

generation:

50.0 tokens/s

quant:

IQ3_XXS (GGUF)

User asks whether NInfer can run on a V100 16GB plus RTX 4070 12GB, reporting 50 t/s on llama.cpp and 35 t/s on coding at long context. Setup is llama.cpp with an IQ3_XXS Flash-Next quant; the machine has 32GB of system RAM. The user also gets a sustained 40 t/s on Strata with the same IQ3_XXS Flash-Next, which they find performs better than a Q4 27B, but the 32GB of RAM causes slow prefill and occasional freezing.

generation:

39.9 tokens/s

prompt processing (prefill):

1376.9 tokens/s

quant:

Q6_K_M (GGUF)

kv:

Q8

User reports Qwen3.8 27B at 39.88 t/s decode and 1,376.87 t/s prompt processing on a mismatched pair of Tesla V100 PCIe cards (16 GB + 32 GB, 48 GB total). Setup is mainline llama.cpp with the Q6_K_M GGUF, tensor split, Flash Attention, Q8 KV cache, batch/ubatch 2,048, 20 threads, and NUMA distribute, with the 32 GB V100 as main GPU. Prompt processing measured 1,324.34 t/s at 4k and 1,221.49 t/s at 16k. A Qwen3.8 Flash Next 177B/6B-active MoE Q4_K_XL GGUF ran at 26.29 t/s decode and 380.67 t/s prompt on the same two cards.