104.0 t/s gen · 1399.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
Q8
KV cache
F16
VRAM reported
24 GB
Summary
User recommends getting enough GPUs to avoid VRAM hacks, and runs 2x RTX 3090s.