llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA Jetson AGX Thor 128GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: mixedNVIDIA hardware
generation:
12.0 tokens/s
quant:
Q4

User reports Qwen3.8 27B Q4 at around 12 t/s on a Jetson Thor, below the ~20 t/s they expected from its memory bandwidth. No engine, context length or KV cache is stated; the figure is without MTP or DFlash. On an RTX 5070 Ti the same model reaches about 66 t/s, and DFlash on the Thor gives only a roughly 2x speedup, slightly better than MTP.

Nov 18, 2026

Qwen3.8 27B

NVIDIA Jetson AGX Thor 128GB · TensorRT Edge-LLM · 2,048 ctx

NVIDIA hardware
generation:
39.6 tokens/s
prompt processing (prefill):
926.0 tokens/s
quant:
INT4

User reports Qwen3.8-27B at 39.6 tok/s decode on a Jetson AGX Thor, with 926 tok/s prompt processing and 23.6 tok/s end-to-end. Setup is TensorRT Edge-LLM 0.10.1 with INT4 weights, a FlashHead LM head and MTP speculative decoding, at batch 1 with 2,048-token prompts and 128 tokens out, averaged over 20 requests. Peak memory was 24.8 GiB and the weights download is about 20 GiB. Accuracy against the FP16 model: GSM8K 96.4 vs 96.1, IFEval 83.5 vs 81.9, MMLU-Pro 79.1 vs 79.5. The user notes Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M plus MTP, without stating prompt length or accuracy.

Oct 26, 2026
Tone: mixedNVIDIA hardware
generation:
0.7 tokens/s
quant:
FP8 (FP8)

User reports MiMo 2.6 Pro at 0.646 t/s on a Jetson AGX Thor 128GB. Setup is FP8 weights of about 566GB read from disk mid-inference, with experts streamed through 128GB of unified memory. The user notes the model technically loaded and ran, and that every decoder layer, routed expert and SSD read was traced and replayable step by step.

Oct 17, 2026
NVIDIA hardware
generation:
27.7-34.4 tokens/s
quant:
NVFP4 (NVFP4)

User reports a repository of launcher scripts for running local LLMs on the 128 GB Jetson AGX Thor Developer Kit, with vLLM or NVIDIA's Thor-optimized llama.cpp container. The default profile is Qwen3.8-27B NVFP4 with DFlash2 speculative decoding, a 30.3 GB download, listed at a published 27.69-34.44 tok/s on Thor. Other profiles include Qwen3.6-35B-A3B NVFP4 + DFlash at 116.5 average and 139.1 peak tok/s, Nemotron Nano 9B v2 NVFP4 at 30 tok/s, Nemotron 3.5 Lightning NVFP4 + DSpark at 123.01-138.02 tok/s, Gemma 4 26B-A4B NVFP4 + MTP at 50 tok/s c1 and 180 tok/s aggregate c8, and Muse Glimmer 30B K-Quant + DFlash at up to 36 tok/s. The GLM-4.7-Flash and GLM-4.5-Air profiles are experimental with no published Thor speeds. The speeds come from different workloads and are not a controlled comparison.

Oct 9, 2026
NVIDIA hardware
generation:
36.7 tokens/s
prompt processing (prefill):
899.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
int8

User reports Qwen3.8-Flash-Next NVFP4 at 36.7 tok/s one-user decode and 899 tok/s prefill at 32k on a Jetson AGX Thor 128GB. Setup is TensorFold 0.6.0 with int8 KV cache, full 262,144-token window, --parallel 4. Prefill measured at 926 / 899 / 798 tok/s for 8k / 32k / 128k tokens; combined decode at 1 / 2 / 3 / 4 users is 34.8 / 53.7 / 70.3 / 76.7 tok/s; first answer to a 60k-token prompt takes 69.7 s. User compares against vLLM's FP8 build on the same box (38.4 tok/s decode, 2,101 tok/s prefill at 32k) and describes a local Thor prompt path with gated runs r09 and r11 reaching 1,549 tok/s prefill at 32k. 128 GB and 273 GB/s are published Thor specs, not re-measured.

Oct 4, 2026
NVIDIA hardware
generation:
25.0 tokens/s
prompt processing (prefill):
3557.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
8-bit

User benchmarks Qwen3.8-27B at 24.96 t/s generation and 3557 t/s prompt processing on a Jetson AGX Thor 128GB, using the Mjolnir vLLM image with the FA4 GEMV decode kernel. Setup is vLLM 0.30.0 with NVFP4 weights and an 8-bit KV cache at 8K context, single request. The GEMV kernel is default-on and dispatches only on M=1, head_dim=256, GQA shapes. The GEMV leg wins at c=1 but regresses at c=4 (57.74 t/s at 8K context, −15.4% versus stock vLLM), which the user attributes to an open investigation. Stock vLLM reaches 24.43 t/s and 2468 t/s prompt at the same setting.

Oct 4, 2026
Tone: positiveNVIDIA hardware
generation:
15.3-18.5 tokens/s
prompt processing (prefill):
90-170 tokens/s
quant:
Q4_K_XL (GGUF)
visioncodingagenticlong-context

User reports Qwen3.8-Flash-Next running on a Jetson AGX Thor 128GB with vision support, decoding at 15.3-18.5 tok/s on free-form text. Setup is llama.cpp (qwen4exp branch, commit d4a943f plus cherry-pick 24ea62d and canreuse-v2.patch) with UD-Q4_K_XL GGUF, 65536 context, ngram-mod speculative decoding, and the 51B n-gram table offloaded to CPU/NVMe via -ot per_layer_token_embd=CPU -lm mmap, leaving about 80GB resident. Prefill is 90-170 tok/s depending on caching; ngram-mod speculation reached 82.9 tok/s on verbatim code reproduction (91% draft acceptance, mean accepted span 59 tokens) but only fires on long untouched spans. The 120W power mode costs about 10% versus MAXN, and images cost a one-off 1-2s to encode without affecting decode speed.

Sep 27, 2026
NVIDIA hardware
generation:
139.1 tokens/s
quant:
NVFP4 (NVFP4)

User reports Qwen3.6-35B-A3B-NVFP4 at 139.1 tok/s on a single NVIDIA Jetson AGX Thor with 117 GB unified memory. Setup is vLLM built from source for sm_110a with DFlash speculative decoding (12 tokens), marlin MoE backend, flash_attn attention backend, 65536 context length, and 0.78 GPU memory utilization. The post also benchmarks Qwen3.5-4B-NVFP4 at 155.8 tok/s, Qwen3.6-27B-NVFP4 at 50.1 tok/s, and Qwen3.5-122B-A10B-NVFP4 at 52.6 tok/s, all at concurrency 1. The 122B requires cutlass MoE and TRITON_ATTN due to a Marlin crash at 256 experts, and achieves 27-42 tok/s with DFlash versus 10.9 tok/s autoregressive.

Sep 27, 2026
Showing 1–8 of 8
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480