DeepSeek V2 Lite-Chat
on Unknown GPU · Project Zero
Summary
User reports Project Zero v0.2.0 running DeepSeek-V2-Lite-Chat Q4_K_S at 11.7-12.6 tok/s on a 4-vCPU AVX-512 VNNI Xeon at four threads, up from 6.3-7.3 tok/s.
The engine is CPU-only, written in C with no ML framework dependency; the same host recorded llama.cpp at 9.4-10.5 tok/s, and resident memory fell from 12.0 GB to 10.6 GB.
With four concurrent clients, aggregate throughput rose from 45 to 101 tok/s for SmolLM2-135M Q8_0 and from 12.7 to 29.8 tok/s for SmolLM2-1.7B Q4_K_M. Startup time dropped to roughly 1-8 seconds.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.