Qwen3.5 35B (3B active)
2× NVIDIA RTX 2000 Ada · Ollama · 16,000 ctx
- generation:
- 58.9 tokens/s
- prompt processing (prefill):
- 2317.5 tokens/s
- quant:
- Q4_K_M (GGUF)
User benchmarks a broad set of local models on a dual RTX 2000 Ada 16GB setup (32GB total VRAM) with 64GB system RAM, running Ollama in Docker on Debian 13. The headline result is Qwen3.5 35B-A3B Q4_K_M at 58.9 tok/s generation and 2317.5 tok/s prefill on a 16K-token prompt, though the user notes a reproducible CUDA illegal-memory-access crash with Flash Attention enabled on one prompt shape. Setup is Ollama with GGUF quants, 16K context for the headline run, and no KV cache quant stated. The user also reports a 100-task quality harness (250 points) and a separate speed harness with 512, 2K and 16K prompts. Other models tested include Gemma 4 26B-A4B Q4 at 42.5 tok/s, GPT-OSS 20B at 46.9 tok/s, Qwen3 30B-A3B IQ3 at 53.4 tok/s, and LFM2 24B-A2B Q4 at 102 tok/s. Dense models were slower: Mistral Small 3.2 24B Q4 at 12.6 tok/s and Qwen3.8 27B Q6_K_M at 8.8 tok/s. The user kept seven models and deleted the rest.