3,400 Tokens a Second. The Whole Rack Has 128GB of Memory.

TL;DR

On August 24 Nvidia put the Groq 3 LPX into full production, the first shipping silicon out of the roughly $20 billion license-and-hire deal it struck with Groq in December 2025. The headline number is real: 3,400 output tokens per second on Google's Gemma 4 31B with a 100,000-token input, measured by Artificial Analysis, which Nvidia calls 4x the nearest alternative platform. The fine print is the interesting part. Each accelerator carries 500 MB of on-die SRAM, a fully loaded rack of 256 of them carries 128 GB, and per The Register that single 31B model at FP8 already needs just under 64 chips. Nebius is the first cloud to deploy, later this year. Nobody published a price.

What actually shipped

The unit of compute is a 2U liquid-cooled tray holding sixteen Groq 3 LPUs, a host CPU, a BlueField-4 DPU or ConnectX-9 NIC, 32 optical chip-to-chip links and 400 Gb/s Ethernet, per StorageReview. Sixteen of those trays make a rack: up to 256 accelerators wired directly to each other rather than through a switch fabric.

It is not a standalone product. Nvidia is selling it as the decode half of the Vera Rubin platform, with Rubin GPUs handling context ingestion and prefill while the LPUs generate tokens. Nebius will run it through Nebius Token Factory alongside Vera CPUs and Rubin GPUs later this year. Nvidia CEO Jensen Huang framed the launch as "inference is the growth engine of AI," and Nebius CTO Danila Shtan put the pitch more usefully: generation is the phase of inference that decides how responsive a system actually feels.

Also listed among the earliest adopters: Groq, which still exists as an independent company, buying racks built from the design it sold.

The number, and the conditions attached to it

3,400 output tokens per second, Gemma 4 31B, 100,000-token input, benchmarked by Artificial Analysis. StorageReview notes the run came off a private pre-release endpoint on August 21. Nvidia's "4x faster than the nearest alternative platform" resolves, per The Register, to Cerebras at 882 tokens per second under the same conditions.

Two things to hold on to. First, this is a long-context claim on a small model. The public Artificial Analysis provider table for Gemma 4 31B runs a 10,000-token default workload, where Cerebras posts roughly 1,578 tokens per second as of publication and Groq does not appear among the listed providers at all. Second, decode at 100K input is exactly the shape of work agents generate, so the context length is not a thumb on the scale. It is the entire pitch.

A full rack holds 128 GB

Here is the architectural fact everything else follows from. The Groq 3 LPU is an SRAM-heavy dataflow chip: 500 MB of on-die memory running at 150 TB/s, per The Register, which points out that the capacity is 576x less than the 288 GB of HBM on a single Rubin GPU. Fill an LPX rack with all 256 accelerators and you get 128 GB of aggregate SRAM. That is still less memory than the one GPU.

The mental model: a GPU is one warehouse with a very wide loading dock. An LPX rack is 256 hotel minibars linked by fiber. Nothing large fits in any single minibar, but if your model does fit spread across them, you can pull from all 256 at once at a speed no warehouse door will ever match.

The cost of that trade is chip count, and it scales badly. The Register's arithmetic: Gemma 4 31B at FP8 needs a little over 31 GB of weights, so just under 64 LPUs. Cerebras fits the same model on one wafer, two at the outside. Go bigger and the gap opens up. DeepSeek V3 at 671B parameters would want 1,342 accelerators, just over five full racks, before it serves a single user.

Why Nvidia bought the problem anyway

Because prefill and decode have never been the same workload. Prefill is compute bound: the whole prompt is in hand, you push it through, FLOPS decide. Decode is memory bandwidth bound: one token at a time, every active parameter read for each one, bandwidth decides. GPUs win prefill. SRAM dataflow chips win decode. Batching hides this in throughput averages and does nothing at all for the single stream a user is watching.

The vendor number attached to the pairing is up to 30x the token throughput per megawatt for Vera Rubin NVL72 against GB300 on 140,000-token agentic coding workloads. That is Nvidia's figure on Nvidia's comparison, so weigh it accordingly. The direction still matters if you pay an inference bill: per-user decode speed is the metric an agent loop actually feels, because a loop that budgets by wall clock gets to think, verify and call tools proportionally harder. We made the same argument when Cerebras launched the CS-4. It did not stop being true when Nvidia bought a horse in the race.

What the launch does not tell you

• No price. Not per rack, not per million tokens, not from Nvidia and not from Nebius.
• No general availability. The chips are in full production; the racks go live "later this year."
• No public leaderboard entry. Until Artificial Analysis can measure a production endpoint, 3,400 is a pre-release figure Nvidia selected and published.
• No large-model numbers. Every published result is on a 31B model small enough to run on a consumer graphics card.
• A moving target. The Register notes the Cerebras CS-4 doubles compute, I/O bandwidth, fabric speed and memory bandwidth over the part that scored 882, so the 4x has a shelf life.

Key Takeaways

• The Groq 3 LPX is in full production as of August 24, the first shipping product from Nvidia's roughly $20 billion December 2025 license-and-hire deal with Groq.
• Headline: 3,400 output tokens per second on Gemma 4 31B at 100,000-token input, measured by Artificial Analysis off a private pre-release endpoint on August 21.
• The "4x faster than the nearest alternative" comparison is against Cerebras at 882 tokens per second on the same test, per The Register.
• Each LPU has 500 MB of on-die SRAM at 150 TB/s. A full 256-chip rack holds 128 GB, less than the 288 GB on one Rubin GPU.
• Chip count is the trade: about 64 LPUs to hold Gemma 4 31B at FP8, and an estimated 1,342 (just over five racks) to hold DeepSeek V3.
• It ships as the decode half of Vera Rubin, with Rubin GPUs doing prefill. Nebius deploys first through Token Factory later this year. No pricing disclosed.

Sources: NVIDIA Newsroom: Groq 3 LPX Now in Full Production, NVIDIA Blog: With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents, The Register: What Nvidia's first Groq 3 LPU benchmarks do and don't tell us about its $20B gamble, StorageReview: NVIDIA Groq 3 LPX Enters Full Production, Artificial Analysis: Gemma 4 31B provider benchmarks, CNBC: Nvidia says Groq racks will be online this year, Nebius Token Factory, CNBC on the December 2025 Groq deal