llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: AMD Strix Halo 128GB
Compare setup details (1 active)

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveAMD hardware
generation:
29-30 tokens/s
prompt processing (prefill):
650.0 tokens/s
quant:
UD-Q4_K_M (GGUF)
visiontool-uselong-context

User reports Inkling-2-Mini decoding at ~29-30 tok/s on Strix Halo with a from-scratch HIP engine written as a Gufo extension, against ~25 tok/s for llama.cpp on the same machine. Setup is Gufo with UD-Q4_K_M GGUF weights in the Mainline_Support layout, single machine on gfx1151; prefill is ~650 tok/s and decode falls to 26.5 tok/s at 20k depth. Continuous batching gives 56 tok/s aggregate across 4 concurrent requests versus 30 single. DSpark speculative decoding is lossless but slower at 17.6 tok/s with 10-30% acceptance. A 31k-token needle test took 71 s versus 255 s for llama.cpp. The author notes the hyperloom implementation is about 50% faster, placing this between llama.cpp and loom.

Nov 21, 2026
Tone: mixedAMD hardware
generation:
22.9-35.2 tokens/s
quant:
Q4
codingagentictool-use

User reports Qwen3.8 27B Q4 with DFlash2 through Gufo on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S, 64 GB graphics allocation) repairing a JavaScript sales-summary bug in 67.057 seconds across eight model passes, passing 6/6 tests. Engine-reported decode rates ranged 22.9–35.2 tok/s, with a per-request mean of about 28.35 tok/s; dividing output by whole-job wall time gives 17.45 tok/s including prompt processing and tool overhead. A harness display bug showing roughly 76 tok/s was traced to buffered tool-call timing and is not engine performance. Separately, Qwen3.8 27B Q6 at 131K context scored 9/9 in about 19m33s at Medium reasoning and 8/9 in about 37m13s at High. A lean coding preset cut a Qwen3.6 35B-A3B repair from 149.366 s to 48.449 s by dropping tool schemas from 91 to 13. A Flint UD-Q4_K_XL run through LM Studio/Vulkan decoded at 25.19–45.02 tok/s and passed 30/30 hidden checks but was stopped after about 14m36s for not executing tests.

Nov 16, 2026
AMD hardware
generation:
39.0 tokens/s
prompt processing (prefill):
1161.0 tokens/s
quant:
Q4

User reports Qwen3.8 Flash-Next at 39.0 tok/s per request on two Strix Halo 128GB boxes with tensor parallelism, against 29.8 tok/s on one box. Setup is a fork of gufo with Q4 weights (111 GB), TP=2 over USB4 RDMA, each box holding 57 GB of the model. Decode is plain autoregressive with no speculation. At 2 concurrent requests the two-box run gives 31.7 tok/s versus 23.3 on one box, and at 4 concurrent 24.6 versus 18.0. Short-prompt prefill is 1161 tok/s on two boxes and 1155 on one.

Nov 14, 2026
Tone: mixedAMD hardware
generation:
34.1 tokens/s

User reports Qwen3.8-Flash-Next at 34.1 tok/s decode on a Ryzen AI Max+ 395 128GB mini PC. Setup is Gufo at 16K context, with the model weights taking around 103.7 GiB. With MTP decode rises to 53.7 tok/s, and adding the NPU gives 34.2 tok/s decode with about 10% faster prefill. The user questions whether the largest model is the best use of the memory.

Nov 11, 2026
AMD hardware
generation:
34.1 tokens/s
prompt processing (prefill):
1888.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)

User compares Gufo against stock llama.cpp on a 128GB Strix Halo (Ryzen AI Max+ 395) running the full UD-Q4_K_XL Qwen3.8-Flash-Next, with FCLK locked at 2000 MHz. Gufo was run with its internal bench and GUFO_PREFILL_TILE=16384; figures are after warm up, three runs. Gufo reached 1888 t/s on a 16K prefill, 35.0 t/s decode at short context and 34.1 t/s decode at 16K context.

Nov 9, 2026

Unknown family

AMD Strix Halo 128GB · Gufo · 16,384 ctx

Tone: mixedAMD hardware
generation:
60.9 tokens/s
prompt processing (prefill):
1888.0 tokens/s

User benchmarks Gufo on a Strix Halo 128GB mini PC (Nimo 395), splitting work between GPU and NPU with FCLK locked at 2000 MHz. At 16k context, GPU-only prefill reaches 1888 t/s and GPU+NPU 2080 t/s without MTP; with MTP the figures are 1812 t/s and 2002 t/s, roughly +10% either way. Short-context decode stays at 60.9 t/s with or without the NPU. Higher 16k decode numbers in some GPU+NPU+MTP runs are not treated as a pure NPU speedup because acceptance rate and output change. The model is not named.

Nov 9, 2026
Tone: positiveAMD hardware
generation:
41.0 tokens/s
prompt processing (prefill):
1240-1335 tokens/s
quant:
UD-Q4_K_XL (GGUF)
kv:
q8_0
codingagenticlong-context

User reports Qwen3.8-Flash-Next decoding at about 41 tok/s at 55-67K context and 36-48 tok/s at 230K on a Strix Halo 128GB laptop (96 GB GPU / 32 GB Windows split), with cold prefill of about 1,240-1,335 tok/s. Setup is Gufo v0.31 with UD-Q4_K_XL weights, q8_0 KV cache, two sessions of 262,144 context, the shared Q8_0 MTP head with 7 draft tokens, and spill-mode disk cache; about 98.4 GB of GPU memory at load. The patches cut per-turn restore at 221-231K from 3.9-4.9 s to 0.67-1.0 s, and a 237K chat spills 3.69 GB to disk in 15.7 s. MTP acceptance is 70-91%. No formal f16-vs-q8 KL measurement was done, and testing was on one Windows machine only.

Nov 9, 2026

Qwen3.8 27B

AMD Strix Halo 128GB · Gufo · 262,144 ctx

Tone: positiveAMD hardware
generation:
24.0 tokens/s
prompt processing (prefill):
450.0 tokens/s
quant:
UD-Q4-K-XL (GGUF)
coding

User reports Qwen3.8-27B at 24 t/s generation and 450 t/s prefill at 262,144 context on a Strix Halo box. Setup is recent Gufo builds with UD-Q4-K-XL weights and DFLASH speculative decoding, using around 40GB of memory. User says speed stays steady as context grows, unlike earlier llama.cpp and MTP attempts that fell to single digits. They ask about automatically switching between QFN for planning and coding and the 27B for memory-heavy builds, noting QFN is faster but uses about 102GB of RAM.

Nov 6, 2026
Tone: positiveAMD hardware
generation:
99.8 tokens/s
prompt processing (prefill):
1648.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)
tool-use

User reports Qwen3.8-Flash-Next at 99.82 t/s MTP decode and 1,648 t/s prefill at 32K context on a Strix Halo machine running Windows 11. Setup is Gufo-win v0.5.0 with UD-Q4_K_XL weights and an MTP Q8_0 draft, greedy decoding, two repetitions, one process at a time. Autoregressive decode without MTP was 32.07 t/s at 32K; the table spans 8K to 256K, where MTP decode falls to 90.03 t/s and prefill to 1,508 t/s. Cached follow-up turns return a first token in 21-23 ms at 32K and 27-30 ms at 100K, against 8.6-9.0 ms and 102-113 ms for Bubo 0.1.1 on the same machine.

Nov 2, 2026
Tone: positiveAMD hardware
generation:
40.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)
kv:
q8_0
agenticlong-context

User reports about 40 tok/s single-user with Qwen3.8 Flash-Next UD-Q4_K_XL on a Strix Halo laptop with 128 GB unified memory under Windows. Setup is Gufo v0.26.0 with three patches, q8_0 KV cache and MTP, serving two sessions of 262,144 context to two LAN users; live KV for both sessions uses 98.5 GB of GPU memory versus about 105 GB at f16. Combined throughput with both users generating is about 55-58 tok/s. One chat reached 261,107 tokens with cache=memory on every turn, first token in about 5-6 s at 250K+ instead of a 4-5 minute re-read, and about 250-280K combined context was held in the RAM cache. Full re-reads still occur when an agent is steered mid-task or the client restarts.

Nov 1, 2026
Tone: positiveAMD hardware
generation:
25.9 tokens/s
prompt processing (prefill):
1277.0 tokens/s

User reports Laguna S 2.1, a ~118B MoE with 256 experts (10 routed plus 1 shared), running end to end on a Strix Halo / Ryzen AI MAX+ 395 with a gufo fork using the NPU alongside the GPU. Decode is 25.9 tok/s plain and prefill reaches up to 1,277 tok/s with the NPU helping, versus ~1,145 tok/s GPU-only. Setup is the gufo engine with NPU+GPU offload; quant and context length are not stated. With the DFlash v1 drafter decode rises to 44-45 tok/s on code and JSON, while prose is more like 28 tok/s, as acceptance depends on content type.

Nov 1, 2026
AMD hardware
generation:
35.0 tokens/s
prompt processing (prefill):
1888.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next at 35.0 t/s short-context decode on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S). Setup is Gufo with UD-Q4_K_XL GGUF, GPU only, no MTP, 128 GB unified memory with about 117 GiB GTT and a 512 MiB BIOS VRAM reservation. At 16K context prefill is 1888 t/s and decode 34.1 t/s. The 103.69 GiB model spans four GGUF shards and fits without issue.

Oct 31, 2026
Tone: positiveAMD hardware
generation:
35.0 tokens/s
prompt processing (prefill):
1888.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next at 35.0 t/s short-context decode and 1888 t/s prefill on a Strix Halo 128GB (Ryzen AI Max+ 395) box. Setup is Gufo with UD-Q4_K_XL GGUF weights (103.69 GiB across four shards), GPU only, no MTP, FCLK locked at 2000 MHz, ROCm 7.2.3. At 16K context decode is 34.1 t/s. Stock llama.cpp ROCm HIP on the same machine gave around 568 t/s prefill and 23-24 t/s decode. The model is the ~125B MoE with ~6B active plus n-gram embedding and MTP modules, fitting in the 128 GB pool with about 117 GiB GTT reported.

Oct 30, 2026
Tone: positiveAMD hardware
generation:
24.5 tokens/s
prompt processing (prefill):
1228.0 tokens/s
quant:
IQ1_M (GGUF)
codingvisionlong-context

User reports DeepSeek V4.1 Flash at 24.5 t/s decode and 1,228 t/s prefill on a single Strix Halo with their Gufo fork. Setup is Gufo with the IQ1_M GGUF resident, HTTP with real chat prompts, greedy decoding and thinking off; DSpark speculation reaches 30.7 t/s on mixed text and 73.6 t/s on repetitive text at depth 0. At 32K depth decode is 24.1 t/s with 1,177 t/s prefill, and at 128K 23.1 t/s with 1,045 t/s prefill. Eight concurrent users get 86.1 t/s total plain and 73.8 t/s with DSpark on mixed text. IQ2_XXS now runs with experts streamed from a Samsung 990 PRO SSD, at 20.8 t/s decode and 1,064 t/s prefill at depth 0.

Oct 30, 2026
Tone: positiveAMD hardware
generation:
23.3 tokens/s
prompt processing (prefill):
1077.0 tokens/s
quant:
IQ1_M (GGUF)
codinglong-contextvision

User reports DeepSeek V4.1 Flash at 23.3 t/s decode and 1,077 t/s prefill on a Strix Halo 128GB (Bosgame M5), fully resident in GPU memory with no SSD streaming. Setup is the user's Gufo fork with IQ1_M GGUF weights, a 112 GiB backbone in GPU memory, IQ4_NL Engram tables (103 GiB) read from NVMe, and a 4.6 GiB Q2_K/Q4_K drafter. The reported figure is plain decode; DSpark speculative decoding with prompt lookup reaches 28.0 t/s on mixed text and 65.3 t/s on repetitive text at depth 0. At 32K depth decode is 22.9 t/s with 993 t/s prefill, and at 128K 21.7 t/s with 700 t/s prefill. Eight concurrent users total 78.5 t/s, and editing a source file through prompt lookup gives 67.6 t/s. The user warns the setup needs the whole machine, leaving about 1.75 GiB of RAM with the drafter loaded.

Oct 26, 2026
Tone: positiveAMD hardware
generation:
50.0 tokens/s
prompt processing (prefill):
1400.0 tokens/s
codingtool-uselong-context

User reports Qwen3.8 Flash-Next on a Strix Halo 128GB (Framework Desktop) at 1400 t/s prefill and about 50 t/s generation, which they call good enough for their usage. Setup is Gufo, serving the model alongside embedding and STT/TTS models; the 50 t/s is described as an average. The user is weighing selling their RTX 5090 to scale out for three concurrent users with 2-3 sessions each at 200k context, and is leaning toward buying a second Strix Halo.

Oct 26, 2026
Tone: positiveAMD hardware
generation:
57.2 tokens/s
prompt processing (prefill):
1637.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)

User benchmarks Qwen3.8-Flash-Next on a Strix Halo 128GB under Gufo v0.3.0 on Windows 11, reporting 57.16 t/s MTP decode and 1,637 t/s prefill at 8,192 tokens of context. Setup is Gufo with UD-Q4_K_XL weights plus an MTP Q8_0 draft, greedy decoding, two repetitions, one process at a time. MTP decode holds 87.71 t/s at 262,144 tokens with 1,464 t/s prefill; without MTP, autoregressive decode is 33.52 t/s at 8K and 27.35 t/s at 256K. MTP acceptance is 69% at 8K and 100% at longer contexts. The user compares v0.3.0 against v0.2.0 and notes the release build showed heavy disk activity.

Oct 26, 2026
Tone: positiveAMD hardware
generation:
42-46 tokens/s
prompt processing (prefill):
966-1186 tokens/s
quant:
UD-Q4_K_XL (GGUF)
codingagenticmath

User compares Qwen3.8-Flash-Next and its Swift 1.5 fine-tune across Halogen, gufo and llama.cpp on a Strix Halo 128GB machine, all with MTP. The gufo run with Unsloth's UD-Q4_K_XL decodes at 42-46 t/s with 966-1,186 t/s prefill at a 32k prompt and 64k context. Setup is gufo with the 111.3 GB UD-Q4_K_XL GGUF and Unsloth's shared Q8_0 MTP head; TTFT was 20-27 s and server start 25 s. Halogen's own .hgn build decodes faster at 51-60 t/s, and llama.cpp trails at 20-23 t/s decode and 298-320 t/s prefill. The user suggests gufo with the Swift requant for coding, which finished the 11 Pi tasks in 24 min against 33 min for Unsloth's file on gufo.

Oct 26, 2026
Tone: positiveAMD hardware
generation:
18.9 tokens/s
prompt processing (prefill):
740.0 tokens/s
coding

User reports GLM-5.3 Flash at 18.9 tok/s plain decode on a Strix Halo 128GB box (Ryzen AI Max+ 395, 8060S gfx1151). Setup is a fork of gufo that splits prefill between the GPU and the XDNA2 NPU and stacks speculative decoding on top. With DFlash2 speculation on code the model reaches 29.7 tok/s. Prefill peaks at 740 tok/s at 8k, 17.5% over GPU only. The same fork also runs Flint, Qwen3.6-35B-A3B, Qwen3.8 Flash-Next and Qwen3.8-27B, with best-case decode of 73, 159, 118 and 62 tok/s respectively.

Oct 25, 2026
Tone: positiveAMD hardware
prompt processing (prefill):
806.0 tokens/s
coding

User reports NPU-assisted prefill on a Strix Halo (Ryzen AI MAX+ 395) box using a fork of Gufo, with the GPU and NPU splitting each projection. Prefill reaches 806 t/s on Qwen3.8-27B, 3,276 t/s on Qwen3.6-35B-A3B and 2,125 t/s on Qwen3.8 Flash-Next. Decode with speculative decoding (MTP, DFlash2, or an auto mode that picks per request) hits up to about 160 tok/s on code and JSON with the 35B-A3B. Quality checked against llama.cpp on CPU showed no measurable perplexity change on the 27B or 35B. The GPU fabric clock must be held at max or the GPU and NPU together can hang the machine; Linux only, gfx1151 only.

Oct 19, 2026
Showing 1–20 of 58
Page 1 of 3

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB89NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3270 total
Qwen3.81569
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other784

Records by engine

2312 total
llama.cpp1136
Strata236
vLLM227
MLX107
Ollama99
other507

Use cases

coding 938agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding938agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active678Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M397UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480