Local model performance on your hardware

generation:

10.9 tokens/s

prompt processing (prefill):

19.7 tokens/s

quant:

IQ3_XXS (GGUF)

User compares mainline llama.cpp against ik_llama.cpp on a hybrid 4-GPU rig with a Qwen3.8 Flash-Next 177B Uncensored IQ3_XXS model at 32k context, 35 layers offloaded to GPU and 13 on the CPU. Mainline reaches 19.70 t/s prompt eval and 10.86 t/s generation with 2,490 CUDA graphs reused; ik_llama.cpp drops to 5.48 t/s prompt eval and 8.43 t/s generation with no CUDA graphs and context checkpoints during generation. User asks whether ik_llama.cpp's advantage applies only to CPU or single-GPU setups, and whether its threadpool struggles with pipelined layer splits across heterogeneous GPUs over PCIe.

generation:

23.5 tokens/s

prompt processing (prefill):

432.3 tokens/s

quant:

IQ4_XS (GGUF)

User reports Qwen3.6-35B-A3B at 23.49 tokens per second per user with 3 concurrent users on two NVIDIA P102-100 cards of 10 GB each. Setup is llama.cpp server with the UD-IQ4_XS GGUF, all layers on the two cards, 96K context giving 32K to each user, with prompt processing at 432.28 tokens per second. The user gives the end result as 70tk/s and 1,257 PP across the three users, from $100 worth of cards.