Llama 3.1 8B
on Unknown GPU
Summary
User reports LLaMA 3.1 8B at 11.6 tok/s sustained (11.9 peak) on a Motorola Moto G Stylus 5G with Snapdragon 6 Gen 1 and 8GB RAM, running CPU-only.
Setup is a custom pure Rust runtime with 1.58-bit ternary quantization, speculative decoding, and mmap; model size is about 2.05 GB with 2.10 GB active RAM.
Single-token decode without speculation is 4.2 tok/s. Qwen 2.5 1.5B reaches 20.3 tok/s at about 480 MB. WikiText-2 perplexity is 6.45 for the 8B and 9.15 for the 1.5B.