Llama 3.2 3B
on Unknown GPU · mlc-llm
Summary
User benchmarks mlc-llm on an Orange Pi 5 Pro running Ubuntu off an m.2 NVMe SSD, with mlc-llm compiled for OpenCL.
The table reports prefill and decode tok/s for seven models; the record captures Llama-3.2-3B-Instruct-q4f16_1-MLC at 8.8 prefill tok/s and 6.1 decode tok/s.
Other rows: SmolLM-1.7B-Instruct 16.6/10.1, Qwen2.5-3B-Instruct 8.5/5.1, Phi-3.5-mini-instruct 5.5/4.8, SmolLM-360M-Instruct 44.4/25.7, SmolLM-1.7B-Instruct 18/9.8, TinyLlama-1.1B-Chat-v1.0 25.9/13.9.