Local model performance on your hardware

generation:

91.3 tokens/s

prompt processing (prefill):

5257.7 tokens/s

quant:

Q8 (GGUF)

User benchmarks LFM3-1.7B Q8 in Magnitude against llama.cpp on an RTX 3060 Laptop 6GB, getting 91.26 t/s generation and 5257.71 t/s prompt processing with Magnitude versus 112.29 t/s and 5710.16 t/s with llama.cpp. Setup is Magnitude's own Rust/TypeScript engine with the Q8 GGUF, benchmarked with llama-benchy at pp2048/tg512; llama.cpp was run via llama-server with only -m set. On a Raspberry Pi 5 CPU-only run at pp128/tg128, Magnitude reached 3.59 t/s generation and 5.07 t/s prompt against llama.cpp's 5.25 and 31.42 t/s. The user could not confirm the claimed 2x speedup, notes Magnitude cannot do partial CPU offload, and says it overestimated VRAM use at 5.3GB when about 4.2GB was reserved.

generation:

25-30 tokens/s

quant:

1-bit

User reports 25-30 t/s decode in a chat UI and 32 t/s raw in a WebGPU browser inference engine, with prompt processing of 1489 tokens in about 25 seconds. The model is natively 1-bit, 27B parameters in 3.8 GB, running at 3072 tokens of context on a 6 GB card. The user also mentions smaller tiers (Qwen3.5 0.8B, 2B, 4B) and a vision tower.