Local model performance on your hardware

generation:

83.0 tokens/s

quant:

IQ3_S (GGUF)

User reports about 83 tok/s generation on an RTX 3090 24 GB running a Swift-1.5 fine-tune of Qwen3.8 27B at IQ3_S GGUF with MTP (draft 4) and vision on. Setup is Ollama 0.35.1 on Windows 11, one request at a time, 131072-token window fully on the card. The figure is for a local agent driving ten Windows apps; the user notes writing files directly is still about 10x faster and made the agent stop doing that by default.

generation:

60.0 tokens/s

quant:

Q4 (GGUF)

User reports Qwen 3.8 27B MTP Q4 at 60 tokens/s on 2x RTX 3090 with 256GB RAM and an EPYC 7443. Setup is Ollama with shared scheduling across both cards; a single card gave 40 tokens/s. User asks whether this is the sweet spot for the hardware and whether an NVLink bridge would help further.

generation:

112.4 tokens/s

User reports Qwen3.6 35B-A3B at 112.4 tok/s on an RTX 3090 24GB with 64K context. Setup is Ollama 0.32.14 with a Modelfile alias baking in num_ctx 65536; at 64K, 96.6% of layers were on GPU with about 0.71 GiB spilled to CPU. The same runner at 32K reached 139.6 tok/s, so 64K was about 20% slower. The user also compares agent task success rates: warning the model to fix wrong values gave 1/9 successful runs with 4 false completions, deterministic tool-layer correction gave 8/15 with 2 false completions, and the same tool-layer approach at 64K gave 5/6 with 0 false completions.