Modal Auto Endpoints: One Command to Production Inference You Actually Own

TL;DR

On June 23, Modal launched Auto Endpoints, a self-serve path to production LLM inference that you create with a single command. Run modal endpoint create --name agent --model zai-org/GLM-5.2-FP8 and you get an OpenAI API-compatible service, backed by a Modal App whose code, GPU selection, regionalization, and inference-engine flags are all visible and editable. That is the whole point: managed inference APIs hide the serving stack behind a black box, and until now the only way to truly own your inference was to build the serving layer yourself. Modal is betting you can have both, the one-command convenience and the open internals.

The black-box problem builders keep hitting

If you ship anything on top of an LLM, you have lived the two-bad-options dilemma. Option one: rent an endpoint from a managed inference provider. It is one HTTP call away, but the serving stack is opaque. You cannot see the engine, you cannot tune the batching, you cannot pin a GPU, and when latency spikes you are filing a support ticket instead of reading a flame graph.

Option two: roll your own. You get full control and full pain. Now you are provisioning GPUs, wiring up SGLang or vLLM, writing autoscaling logic, and babysitting cold starts at 3am. Modal frames Auto Endpoints as a third door: the convenience of a managed API with the transparency of a stack you own, because under the one-line command sits a real Modal App you can open, read, and modify.

Managed inference providers make it easy to get an API, but the serving stack is a black box. So until now, teams that wanted proper ownership of their inference have had only one option: roll an inference service yourself.

What the one command actually gives you

The example in the launch deploys a frontier open model, GLM-5.2-FP8 from Z.ai, with no sales call and no capacity reservation. What you get back is not a sealed product but an inspectable app:

• OpenAI-compatible API. Point your existing client at the new URL and the chat-completions calls just work, so swapping providers is a base-URL change, not a rewrite.
• Open internals. GPU selection, regionalization, and inference-engine flags are exposed, not hidden. The serving code is yours to read and edit.
• Real autoscaling and usage billing. No months-long GPU reservation. You pay for what you use as you use it, and the launch lists $30 per month of free compute to start.
• Debuggable by default. The dashboard surfaces the metrics you actually need when inference misbehaves: speculative-decoding acceptance length, per-replica engine-side token latency quantiles, plus GPU temperature and utilization.

The speed story: a speculator in the default path

Auto Endpoints does not just wrap a model in a web server. The default serving path uses speculative decoding built on the DFlash block-diffusion drafter from Z Lab, paired with SGLang and FlashAttention-4 kernels. Modal claims the DFlash speculator runs more than 4x faster than an unoptimized baseline and more than 1.5x faster than other speculators on multiple benchmarks. Speculative decoding is the single biggest free lunch in LLM serving right now, and the news here is that you get it without hand-assembling a draft model and a verifier yourself.

Modal Servers: the plumbing underneath

The launch also introduces a new primitive called Modal Servers, aimed at the low-latency routing that real-time inference demands. Modal says Servers keep the elastic scaling and deep compute pool of its Web Functions but remove queueing and are regionalized by default, so HTTP requests carry only 5ms of overhead. For interactive agents and chat, where every millisecond of routing tax shows up in time-to-first-token, that is the difference between a snappy app and a laggy one.

It fits Modal's broader pitch as a serverless GPU platform with a multi-cloud pool (B200, H200, H100, A100) and sub-second cold starts, the same machinery it described in its truly serverless GPUs writeup. The company is reportedly raising at around a $2.5 billion valuation, so the inference-infrastructure bet has real money behind it.

The honest caveats

The speedup figures are Modal's own, measured on their stack, so treat the 4x as a vendor benchmark until you run your own model and your own traffic through it. Speculative decoding gains are workload-dependent: acceptance length swings with how predictable your outputs are, which is exactly why exposing that metric in the dashboard matters. And "you own it" still means owning it on Modal's platform, with Modal's pricing, on Modal's GPUs. It is far more transparent than a sealed API, but it is not the same as racking your own hardware. The right test is the boring one: deploy your actual model, point your actual client at it, and watch the latency quantiles under your actual load.

Key Takeaways

• Modal Auto Endpoints (launched June 23) creates a production, OpenAI-compatible LLM endpoint from a single CLI command.
• The endpoint is backed by a Modal App you can read and edit: GPU choice, regionalization, and engine flags are exposed, not hidden behind a managed black box.
• The default path uses a DFlash speculative decoder with SGLang, claimed at more than 4x over baseline and more than 1.5x over other speculators.
• A new Modal Servers primitive adds regionalized, queue-free routing with about 5ms of HTTP overhead for low-latency inference.
• Billing is usage-based with $30 per month of free compute, and the dashboard exposes acceptance length and per-replica token latency quantiles for debugging.
• The benchmark numbers are vendor-measured, so validate with your own model and traffic before trusting them in production.

Sources: Modal blog: Introducing Auto Endpoints, Modal blog: How we achieved truly serverless GPUs, GLM-5.2-FP8 (Hugging Face), TechCrunch: Modal Labs valuation