llamaperf

Unknown family 40B

on NVIDIA RTX 3060 12GB

Tone: positiveNVIDIA hardware
Sep 18, 2026
Throughput
16.0 t/s gen

Summary

User reports a 40B model (~25GB) running at 16 t/s across a 3-device cluster: a mini PC with an RTX 3060, a 16GB Mac Mini, and an old laptop. Setup uses the user's own ramdeck-core pooling repo, sharding the model across the 3060 first, then the Mac's unified memory, then CPU spillover routed to the old laptop. The full load took about 2.5 minutes and latency was roughly 3.6s. User notes that offloading CPU spillover to the slower laptop beat keeping it on the mini PC's own CPU, and asks whether others have seen the same GPU+CPU contention effect.