Local model performance on your hardware

generation:

68.1 tokens/s

prompt processing (prefill):

3057.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3-14B at 68.1 t/s generation and 3057 t/s prompt processing on a 2x RTX 2080 Ti node. Setup is llama.cpp with Q4_K_M quant, 32768 context, and a Qwen3-0.6B-Q8_0 draft model for speculative decoding with depth 6. The user also reports Pixtral-12B at 77.0 t/s generation and 2862 t/s prompt processing, and a distributed 4-stream aggregate of 118.68 t/s. They ask if Linux would give better t/s and mention P2P is working on Windows.

prompt processing (prefill):

255.0 tokens/s

quant:

W8A8

User reports custom Turing CUDA kernels for W8A8 INT8 matmul, heterogeneous inference with 4x 11/22GB VRAM and 1TB system RAM, and computation-communication overlap for MoE routing. The work is open-sourced on GitHub.