LFM3

generation:

1.0 tokens/s

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks how to run LFM3 1.7B at high concurrency locally, using llama.cpp with q8_0 KV cache and --parallel 16 --kv-unified on an RTX 4060 Laptop with 8 GB VRAM and 16 GB system RAM. User reports that beyond about 6 concurrent sequences throughput stops scaling, and that starting a new request drops every other instance to about 1 tok/s until that request finishes prefill. No single-stream throughput figure is given; the 1 tok/s is a per-stream rate under concurrent load.

generation:

32.3 tokens/s

prompt processing (prefill):

1616.1 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports LFM3-VL 2B at 32.3 t/s decode and 1616.1 t/s prefill on the Hexagon NPU of a Snapdragon X Elite X1E-78-100 laptop with 64 GB RAM. Setup is GenieX with Q4_0 GGUF weights, comparing NPU, hybrid NPU+CPU, Adreno OpenCL GPU and ARM CPU backends. The NPU doubles CPU prefill at 12 W versus 35-60 W, but decode is about half the CPU's 66.5 t/s. Gemma 4 E2B, Gemma 4 26B MoE and Gemma 4 31B were also run on the NPU.