- generation:
- 101.0 tokens/s
- quant:
- 8-bit (MLX)
User reports NVIDIA Nemotron-Labs Dory 1.0 4B at 101 tok/s on an M5 Max 128GB with one loop, dropping to 69 tok/s at 2 loops and 39 tok/s at 4 loops. Setup is an MLX port with the 8-bit 4.2 GB build, medians of repeated runs; peak memory is 7.7 GB at 32K context with one loop. After a 32K prompt the same runs give 82, 54 and 31 tok/s, with prefill at 3,200, 1,980 and 1,070 tok/s. Batch generation reaches about 450 tok/s total across 16 requests at 2 loops, and a 16 GB M1 Pro measured about 28 tok/s at 1K and 19 tok/s at 32K in 4-bit at 2 loops. 8-bit WikiText-2 perplexity falls from 12.18 to 11.77 to 11.70 as loops increase.