This Open-Source Speech Model Learned to Hear by Training 0.16% of Its Weights

TL;DR

Interfaze, a young YC startup, just open-sourced diffusion-gemma-asr-small, a speech-to-text model that transcribes six languages. The headline is not accuracy. It is method and cost: they froze Google's 26B DiffusionGemma and a whisper-small encoder, then trained a roughly 42M-parameter adapter on top, about 0.16% of the model's weights. Because the backbone is a diffusion language model, it does not spell out a transcript left to right. It denoises the whole sequence in parallel over 8 to 16 steps. It hits 6.6% word error rate on LibriSpeech test-clean, which trails autoregressive Whisper but beats the previous diffusion-ASR baseline, on a fraction of the training. If you build voice features, the interesting part is that speech recognition just became a small adapter you can bolt onto a general-purpose model.

What actually shipped

The release is small on purpose. What lands on the Hub is an adapter (around 170 MB), a projector of roughly 19M parameters, and the glue: a model.py, an audio.py, and a runnable inference.py. The two heavy components, the whisper-small acoustic encoder and the DiffusionGemma decoder, stay frozen and load separately. Nothing about the big models was retrained. The adapter's only job is to teach a text-generation model to condition on audio.

That framing is the whole story. Interfaze did not train a speech model. They taught an existing diffusion language model to listen, and the tuition bill was 0.16% of the weights. The other 99.84% came along for the ride, unchanged.

Why diffusion for speech is weird, and why it matters

Almost every speech recognizer you have used, Whisper included, is autoregressive: it predicts one token, appends it, then predicts the next, conditioned on everything so far. It writes the transcript the way you would type it, left to right, each word waiting on the one before it.

A diffusion decoder does not do that. It starts from a fully masked sequence and refines every position at once, over a handful of passes, sharpening a rough guess into a clean transcript. Think of developing a Polaroid: the whole picture surfaces together, getting less blurry each second, instead of being painted one brushstroke at a time. DiffusionGemma generates text this way natively, so the adapter inherits parallel decoding for free. The practical payoff is that transcript length stops dictating the number of sequential steps. Whether it is 8 steps or 16, all the words resolve together.

For the record, DiffusionGemma is a real base model worth knowing: Google's 25.2B-parameter mixture of experts with 3.8B active per token, 8 of 128 experts routed plus a shared one, and Apache-2.0 weights. It denoises blocks of tokens in parallel rather than emitting them one by one. Interfaze essentially discovered that a model trained to hallucinate text out of noise can be pointed at an audio signal and made to hallucinate the correct text instead.

The numbers, told straight

On LibriSpeech test-clean, diffusion-gemma-asr-small lands at 6.6% WER. That beats Whisfusion, the prior masked-diffusion ASR baseline, at 8.3%, and it does so with a smaller encoder. It does not beat autoregressive whisper-small, which sits around 3.4% on the same set. The model card is refreshingly honest about why: the adapter saw only about 219 hours of audio. Whisper trained on the better part of a million hours. The wonder is not that a 219-hour model loses to Whisper. It is that it comes this close.

Multilingually it is a research preview, and the numbers say so plainly. On FLEURS, English comes in at 15.7% WER, Hindi at 15.8% character error rate, and Mandarin at 29.6% CER, with VoxPopuli English at 18.5% WER. These are not production numbers for a customer-facing transcript in Mandarin. They are a proof that one adapter, from 219 hours, can straddle six languages at all, which a year ago would have taken six fine-tunes and a lot more data.

Why a builder should care

The specific model may never touch your stack. The pattern behind it should. A few things change if this approach holds:

• ASR becomes an adapter, not a model. If you already run a diffusion LLM for text, adding speech input could cost you a 42M-parameter head and a few hundred hours of audio, not a from-scratch training run. The marginal cost of a new modality just fell through the floor.
• Parallel decoding is the latency lever. Interfaze reports roughly 0.7 to 1.5 seconds of model time for a 10-second clip at 16 steps. Drop to 8 steps and you trade accuracy for speed on a dial, without the length of the audio inflating your step count. That is a different performance curve than autoregressive ASR.
• The license lets you actually ship it. The adapter is Apache-2.0, DiffusionGemma is open-weight, and the whisper-small encoder is MIT. You can self-host the whole thing, which for a lot of voice products is the difference between a feature and a compliance headache.
• It is a preview, so treat it like one. Six languages, 219 hours, single-clip benchmarks. Do not put it in front of paying users transcribing Mandarin support calls next week. Do prototype with it, and do watch whether Interfaze scales the training data, because that is the only thing standing between these numbers and Whisper's.

The bigger pattern

This is the third or fourth time in recent memory that someone has taken a large frozen model and unlocked a new capability by training a rounding error's worth of new weights on top. The lesson keeps repeating: the expensive part is the general-purpose backbone, and once it exists, whole new abilities are cheap adapters away. Speech recognition, one of the most heavily engineered corners of ML, just got demoted to a fine-tuning exercise. That is either the democratization of voice AI or the commoditization of it, depending on whether you sell shovels or dig.

Key Takeaways

• Interfaze open-sourced diffusion-gemma-asr-small, a six-language speech recognizer built by training a roughly 42M-parameter adapter, 0.16% of the weights, on a frozen 26B DiffusionGemma plus a frozen whisper-small encoder.
• Because the backbone is a diffusion model, it decodes transcripts in parallel over 8 to 16 steps rather than one token at a time, so step count does not scale with audio length.
• It scores 6.6% WER on LibriSpeech test-clean, beating the Whisfusion diffusion baseline (8.3%) but trailing autoregressive whisper-small (about 3.4%), a gap the model card attributes to seeing only about 219 hours of audio.
• Multilingual results are preview-grade: 15.7% WER FLEURS English, 15.8% CER Hindi, 29.6% CER Mandarin. Useful for prototyping, not production Mandarin transcription.
• The real signal for builders: adding a new modality to an existing frozen model can now cost a tiny adapter and a few hundred hours of data, and the whole stack (Apache-2.0 adapter, open-weight backbone, MIT encoder) is self-hostable.

Sources: diffusion-gemma-asr-small model card (Hugging Face), MarkTechPost, DiffusionGemma model card (Google), Whisfusion (arXiv 2508.07048)