Unknown family
on 2× NVIDIA RTX 3090
Use cases
agentic
Summary
User compares two quant builds of the same unnamed model on dual RTX 3090s with NVLink and 128GB RAM. The UD Q5_K_XL build with F16 KV cache runs at around 100 t/s with the Q8 PLE on SSD, while the iQ3_S int8 setup runs at 150 t/s.
All quality-reducing speed flags are off in the Q5 run. The user says the Q5 build thinks less and is much more logical, takes fewer steps in agentic tasks and needs fewer corrections.
No benchmarks were run; the comparison is subjective.