Local model performance on your hardware

generation:

58.1 tokens/s

User reports gpt-oss-120b at 58.1 tok/s on two DGX Sparks (128 GB each) with tensor parallel over RPC in their Ollama fork, up from 42.7 tok/s on one node and 33.8 tok/s with a layer split. Setup is Ollama driving llama.cpp RPC with two servers, each holding half of every layer and doing the all-reduce between themselves, over a direct 200 Gb/s RDMA cable. Measured with ollama cluster bench at 32k context, tg128, warm, mean of 3. Qwen3.8 27B with an MTP draft reached 34.7 tok/s (24.4 one node, 21.4 layer split) and DeepSeek-V4-Flash MXFP4 at 156 GB reached 21.4 tok/s (14.5 layer split, does not fit on one node). Prompt processing stays about 10% below one node for models that fit, and load time is a few seconds longer than the layer split. vLLM with TP=2 gave 55.7 tok/s for gpt-oss-120b on the same pair in eager mode, single request. Limits are exactly two machines, one tensor model at a time.

generation:

15.3 tokens/s

quant:

UD-IQ3_XXS (GGUF)

User reports Qwen3.5-397B at 15.3 t/s across two DGX Sparks (128 GB unified each) over llama.cpp RPC, a model that does not fit on one node. Setup is a forked Ollama 0.40.3 with llama.cpp b11351, UD-IQ3_XXS quant (141 GB), 256 output tokens, warm run, two nodes linked by a 200 Gb/s cable. Splitting across nodes costs speed: fast small models lose 30-60% (gemma4:e2b 106.6 to 44.1 t/s), large models lose 15-20%, and load time is 3-5x longer. The default policy keeps a model on one machine and only splits when it does not fit.