← Back to all posts
Tools

Open-Source TrnRT Serves GLM-5.3-Flash on One Trainium Box at 3.57M Tokens/Min

November 19, 2026 · 03:15 UTC · Tools
On this page 8 sections
Open-Source TrnRT Serves GLM-5.3-Flash on One Trainium Box at 3.57M Tokens/Min

TL;DR

TrnRT is an Apache-2.0 inference engine that serves GLM-5.3-Flash, a 320B-parameter open MoE, on one AWS trn2.48xlarge: 16 Trainium2 chips, no GPUs, behind an OpenAI-compatible API. Its makers, RunAnywhere, measured it on SemiAnalysis's AgentX benchmark of coding-agent traffic: 3.57 million tokens a minute from the box, a million-token prompt to its first token in 190.8 seconds, $0.167 per million tokens at the capacity-block price and $0.040 at spot. The repo went public on 30 October, the Show HN landed on 17 November, and the accompanying engineering writeup runs 34 minutes and volunteers every caveat before anyone can object.


At a glance
One box, no GPUs
3.57M
tokens a minute on one box, AgentX batch 12
$0.167
per million tokens at the capacity-block price
190.8 s
million-token prompt to its first token
24.2 ms
batch-8 decode step, down from 32.2 ms
320B/18B
glm-5.3-flash parameters, total vs active per token
1,633
github stars since 30 october, apache-2.0

What shipped

The repository ships a complete serving stack. You get chat and completions endpoints with streaming, tool calls and reasoning output; a session-affine router that pins each agent session to one of two engines so its prefix stays cached; a radix-tree prefix cache over FP8 KV pages with a host-memory tier for evicted pages; and an offline compile pipeline that builds every graph on a plain x86_64 host first, so the serving box loads precompiled graphs and compiles nothing at runtime. Bring Docker, an AWS account and a few hours for the compile cache.

The first supported model is GLM-5.3-Flash: 45 layers, about 18B of its 320B parameters active per token, 288 routed experts with eight picked per token, FP8 weights, and a one-million-token context built from 34 linear-attention layers and 11 sparse-attention layers. It has 6.4M downloads on Hugging Face, so the audience exists. More model families are promised, and the architecture docs spell out what a new one needs.

How it works
From agent call to tokens
Client
OpenAI-compatible call
chat, completions, streaming, tool calls
Router
Session pinned to one engine
two engines per box, prefix stays warm
Engine
Decode as NKI megakernels
tensor-parallel over 32 logical NeuronCores on 8 chips
Cache
Radix tree of FP8 pages
device memory first, host tier behind it
Result
3.57M tokens a minute
$0.021 per agent request at batch 12
Prompts are long and mostly cached: about 95 percent of counted tokens are prefix reads.

The numbers, asterisks attached

AgentX replays the shape of real coding-agent sessions: long prompts that are almost entirely cached by the next turn, subagents spawning mid-flight, a user waiting on every answer. RunAnywhere ran it self, one hour per point, at a one-million-token context. At decode batch 12 with 32 concurrent agents, the box served 28.9 agent requests a minute. At 16 agents, the friendlier number when your users wait on each reply: a median cached prompt starts in 1.24 seconds, decodes at 35.9 tokens a second, and costs $2.24 per concurrent agent per hour.

AgentX, 32 agents
More decode batch, more requests a minute
decode batchdefault, batch 12
batch 4
19.3 $0.031/req
batch 8
25.2 $0.024/req
batch 12
28.9 $0.021/req
Agent requests per minute on one trn2.48xlarge; cost per request at the $35.76/h capacity-block price. Self-run, one hour per point. Source: RunAnywhere benchmarks, Nov 2026.

Now the asterisks. About 95 percent of the tokens in those throughput figures are cached prefix reads, so per-token costs computed from them are not comparable to API prices, which charge cache hits, new input and output separately. Charge the whole box to output tokens alone and the figure is $24.47 per million. Each point is a single self-run on harness version 1.0.7 rather than the current 1.0.13, and there is no GPU baseline on the same traces, so Trainium-versus-H200 claims stay out of this post. The makers say a same-trace GPU comparison is on their list.

Layer megakernels, the actual trick

A batch-8 decode step takes 24.2 ms. The memory floor for that step, streaming weights and attention state at the chip's measured bandwidth, is 6.1 ms. The other 18 ms was going to small vector-engine instructions that each pay about 153 ns of fixed overhead, to kernel boundaries where every intermediate takes a round trip through HBM, and to 125 cross-chip collectives per step that each leave the cores idle for roughly 23 microseconds.

The fix is what the authors call layer megakernels: one NKI kernel, written in AWS's Python-embedded kernel language for Trainium, owning a whole layer half, whether that is the linear-attention block, the sparse-attention layer with its collectives moved inside, or the MoE feed-forward half from routing to combine. Three rules shape every kernel: the working set stays in the core's 28 MiB scratchpad, both physical cores of each pair run the same structure in lockstep, and a blocking collective gets independent work packed around it inside the kernel. XLA's default graph hands data from kernel to kernel like a relay team passing the baton through a parking garage two streets away; a megakernel keeps it in one runner's hand for the whole lap. The rewrite took the batch-8 step from 32.2 ms to 24.2 ms.

Decode step, batch 8
Four times the memory floor
before and aftermegakernels
17 oct build
32.2 ms
1 nov build
24.2 ms
mem floor
6.1 ms
NKI layer megakernels cut the step 25 percent; the floor is what bandwidth alone requires. Source: RunAnywhere blog, Nov 4, 2026.

What the silicon costs

Per terabyte of accelerator memory, which is what a 320B FP8 model with million-token sessions actually buys, AWS prices the trn2.48xlarge at $23.84 per TB-hour on a capacity block. An eight-H200 p5e box is $48.69. An eight-B200 p6 box is $79.38. The whole 16-chip box rents at $35.76 an hour on a capacity block, $103.16 on-demand, and spot was $8.60 on 3 November, if you can stomach AWS repossessing the machine and your prefix cache mid-session. Two cautions come from the makers themselves: rental prices are not serving costs, and nobody has published the same traces on both silicon families yet.

AWS capacity blocks
Cost per TB-hour of accelerator memory
gpu boxestrainium2
trn2.48xl
$23.84
p5e h200
$48.69
p5en h200
$55.99
p6 b200
$79.38
US East (Ohio) capacity-block pricing read on 3 Nov 2026: trn2 is 16 chips and 1.5 TB, the GPU boxes are 8 chips each. Source: AWS via RunAnywhere blog.

The gap it fills

Until this month, serving a frontier open model on Trainium was a write-your-own-engine problem. AWS's vLLM plugin for Neuron registers no GLM model in its latest release, with an open pull request for the larger GLM-5.3. AWS's own NxD Inference library sits in maintenance mode. SGLang announced a Trainium beta on 22 October that lists GLM among its models but has published no performance numbers. The one earlier option, Kiln, shipped on 5 October; RunAnywhere measured Kiln v0.2.0 on the same box and reports 5.0x faster first tokens on 8K prompts, 3.6x faster single-user decode, and a million-token prompt reaching its first token in 190.8 s against Kiln's published 1,651.7 s. Kiln v0.3.0 has since improved that published figure to 315.7 s, which RunAnywhere has not tested.

Read this part before you rent a box

  • Every AgentX point is one self-run hour on one box. These are not official InferenceX results.
  • One approximation remains: 1.2 percent of sparse-attention selections use a merged shortlist instead of the exact top set. It moved next-token log-probabilities by 0.012 to 0.021 nats on average, and 2 of 6 greedy test continuations diverged.
  • At 48 agents the 4 GiB-per-rank host cache tier filled, evicted 93,548 pages, and throughput fell from 3,720 to 2,642 tokens/s per chip. The ceiling today is host memory, not silicon.
  • The public repo passes the makers' equivalence checks against the benchmarked build: the same 157 graphs compile on all 32 ranks, and outputs are byte-identical on 384 of 384 test requests.

Key Takeaways

  • TrnRT (Apache-2.0) serves GLM-5.3-Flash on one trn2.48xlarge behind an OpenAI-compatible API: 3.57M tokens a minute on AgentX at batch 12, $0.167 per million tokens at the capacity-block price.
  • NKI layer megakernels cut the batch-8 decode step 25 percent, from 32.2 ms to 24.2 ms, against a 6.1 ms memory floor. The remaining headroom is instructions, boundaries and collectives.
  • Agent-shaped economics: $2.24 per concurrent agent per hour at 16 agents, with median cached first tokens in 1.24 seconds.
  • Trainium2 costs roughly half an eight-H200 box per TB-hour of accelerator memory, but no same-trace GPU baseline exists yet, so hold the comparisons.
  • The host-memory prefix cache tier, 4 GiB per rank, is the current scaling limit past 32 agents.
  • The benchmark numbers are self-run single hours with about 95 percent cached tokens, and the makers publish every caveat themselves.

Sources: TrnRT on GitHub, RunAnywhere engineering writeup, Show HN discussion, GLM-5.3-Flash model card, AgentX harness, AWS Trn2 instances, AWS Capacity Blocks pricing, Kiln, vllm-neuron, SGLang Trainium beta

AITrnRTAWS TrainiumGLM-5.3-FlashLLM InferenceNKIOpen SourceAgentX
CONSOLE
$