llamaperf

NVIDIA Jetson AGX Thor 128GB reports

8 reports. These are individual setups, not a controlled benchmark. NVIDIA Jetson AGX Thor 128GB speeds and models →

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Tone: mixedNVIDIA hardware
generation:
12.0 tokens/s
quant:
Q4

User reports Qwen3.8 27B Q4 at around 12 t/s on a Jetson Thor, below the ~20 t/s they expected from its memory bandwidth. No engine, context length or KV cache is stated; the figure is without MTP or DFlash. On an RTX 5070 Ti the same model reaches about 66 t/s, and DFlash on the Thor gives only a roughly 2x speedup, slightly better than MTP.

Nov 18, 2026

Qwen3.8 27B

NVIDIA Jetson AGX Thor 128GB · TensorRT Edge-LLM · 2,048 ctx

NVIDIA hardware
generation:
39.6 tokens/s
prompt processing (prefill):
926.0 tokens/s
quant:
INT4

User reports Qwen3.8-27B at 39.6 tok/s decode on a Jetson AGX Thor, with 926 tok/s prompt processing and 23.6 tok/s end-to-end. Setup is TensorRT Edge-LLM 0.10.1 with INT4 weights, a FlashHead LM head and MTP speculative decoding, at batch 1 with 2,048-token prompts and 128 tokens out, averaged over 20 requests. Peak memory was 24.8 GiB and the weights download is about 20 GiB. Accuracy against the FP16 model: GSM8K 96.4 vs 96.1, IFEval 83.5 vs 81.9, MMLU-Pro 79.1 vs 79.5. The user notes Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M plus MTP, without stating prompt length or accuracy.

Oct 26, 2026
Tone: mixedNVIDIA hardware
generation:
0.7 tokens/s
quant:
FP8 (FP8)

User reports MiMo 2.6 Pro at 0.646 t/s on a Jetson AGX Thor 128GB. Setup is FP8 weights of about 566GB read from disk mid-inference, with experts streamed through 128GB of unified memory. The user notes the model technically loaded and ran, and that every decoder layer, routed expert and SSD read was traced and replayable step by step.

Oct 17, 2026
NVIDIA hardware
generation:
27.7-34.4 tokens/s
quant:
NVFP4 (NVFP4)

User reports a repository of launcher scripts for running local LLMs on the 128 GB Jetson AGX Thor Developer Kit, with vLLM or NVIDIA's Thor-optimized llama.cpp container. The default profile is Qwen3.8-27B NVFP4 with DFlash2 speculative decoding, a 30.3 GB download, listed at a published 27.69-34.44 tok/s on Thor. Other profiles include Qwen3.6-35B-A3B NVFP4 + DFlash at 116.5 average and 139.1 peak tok/s, Nemotron Nano 9B v2 NVFP4 at 30 tok/s, Nemotron 3.5 Lightning NVFP4 + DSpark at 123.01-138.02 tok/s, Gemma 4 26B-A4B NVFP4 + MTP at 50 tok/s c1 and 180 tok/s aggregate c8, and Muse Glimmer 30B K-Quant + DFlash at up to 36 tok/s. The GLM-4.7-Flash and GLM-4.5-Air profiles are experimental with no published Thor speeds. The speeds come from different workloads and are not a controlled comparison.

Oct 9, 2026
NVIDIA hardware
generation:
36.7 tokens/s
prompt processing (prefill):
899.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
int8

User reports Qwen3.8-Flash-Next NVFP4 at 36.7 tok/s one-user decode and 899 tok/s prefill at 32k on a Jetson AGX Thor 128GB. Setup is TensorFold 0.6.0 with int8 KV cache, full 262,144-token window, --parallel 4. Prefill measured at 926 / 899 / 798 tok/s for 8k / 32k / 128k tokens; combined decode at 1 / 2 / 3 / 4 users is 34.8 / 53.7 / 70.3 / 76.7 tok/s; first answer to a 60k-token prompt takes 69.7 s. User compares against vLLM's FP8 build on the same box (38.4 tok/s decode, 2,101 tok/s prefill at 32k) and describes a local Thor prompt path with gated runs r09 and r11 reaching 1,549 tok/s prefill at 32k. 128 GB and 273 GB/s are published Thor specs, not re-measured.

Oct 4, 2026
NVIDIA hardware
generation:
25.0 tokens/s
prompt processing (prefill):
3557.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
8-bit

User benchmarks Qwen3.8-27B at 24.96 t/s generation and 3557 t/s prompt processing on a Jetson AGX Thor 128GB, using the Mjolnir vLLM image with the FA4 GEMV decode kernel. Setup is vLLM 0.30.0 with NVFP4 weights and an 8-bit KV cache at 8K context, single request. The GEMV kernel is default-on and dispatches only on M=1, head_dim=256, GQA shapes. The GEMV leg wins at c=1 but regresses at c=4 (57.74 t/s at 8K context, −15.4% versus stock vLLM), which the user attributes to an open investigation. Stock vLLM reaches 24.43 t/s and 2468 t/s prompt at the same setting.

Oct 4, 2026
Tone: positiveNVIDIA hardware
generation:
15.3-18.5 tokens/s
prompt processing (prefill):
90-170 tokens/s
quant:
Q4_K_XL (GGUF)
visioncodingagenticlong-context

User reports Qwen3.8-Flash-Next running on a Jetson AGX Thor 128GB with vision support, decoding at 15.3-18.5 tok/s on free-form text. Setup is llama.cpp (qwen4exp branch, commit d4a943f plus cherry-pick 24ea62d and canreuse-v2.patch) with UD-Q4_K_XL GGUF, 65536 context, ngram-mod speculative decoding, and the 51B n-gram table offloaded to CPU/NVMe via -ot per_layer_token_embd=CPU -lm mmap, leaving about 80GB resident. Prefill is 90-170 tok/s depending on caching; ngram-mod speculation reached 82.9 tok/s on verbatim code reproduction (91% draft acceptance, mean accepted span 59 tokens) but only fires on long untouched spans. The 120W power mode costs about 10% versus MAXN, and images cost a one-off 1-2s to encode without affecting decode speed.

Sep 27, 2026
NVIDIA hardware
generation:
139.1 tokens/s
quant:
NVFP4 (NVFP4)

User reports Qwen3.6-35B-A3B-NVFP4 at 139.1 tok/s on a single NVIDIA Jetson AGX Thor with 117 GB unified memory. Setup is vLLM built from source for sm_110a with DFlash speculative decoding (12 tokens), marlin MoE backend, flash_attn attention backend, 65536 context length, and 0.78 GPU memory utilization. The post also benchmarks Qwen3.5-4B-NVFP4 at 155.8 tok/s, Qwen3.6-27B-NVFP4 at 50.1 tok/s, and Qwen3.5-122B-A10B-NVFP4 at 52.6 tok/s, all at concurrency 1. The 122B requires cutlass MoE and TRITON_ATTN due to a Marlin crash at 256 experts, and achieves 27-42 tok/s with DFlash versus 10.9 tok/s autoregressive.

Sep 27, 2026