Unknown family
NVIDIA RTX 4070 · Ollama
- generation:
- 29.6 tokens/s
User reports Mixtral 8x7B, Qwen-MoE and DeepSeek-MoE MoE models running through Ollama on an RTX 4070 12GB at 29.6 tok/s, up from 20.2 tok/s with a custom PyTorch orchestration layer (NPOL Engine) that prefetches experts from pinned host RAM. Setup offloads expert weights to system RAM over PCIe Gen4 x16, with a rank-32 lookahead probe staging predicted experts on a non-blocking CUDA stream and a thermal guard shedding tokens to resident experts. The user reports a 67.0% VRAM cache hit rate and 35.8% PCIe stall time saved, with 99.55% output cosine similarity preserved. Other cards: RTX 4080 Laptop 12GB 19.6 to 28.9 tok/s, RTX 3060 12GB 19.2 to 28.5 tok/s with temperature dropping from 82.5C to 58.0C, RTX 4090 24GB 27.6 to 37.2 tok/s.