A Skyrim Dog Answers in 500ms. The LLM Doesn't Pick Actions.

TL;DR

Varkos is a real-time AI companion that plays Skyrim with you: it hears you through an always-open microphone, fights, loots, fetches, holds multi-step plans across minutes, and answers in a voice. Pantelis Kalogiros runs the game on Windows and the entire audio and reasoning loop locally on an M4 MacBook, and reports a per-turn budget of 40-80 ms for speech to text, about 20 ms for action selection, 300-600 ms for response generation and grounding, and 20-60 ms for audio out. With aggressive prefill, he says the dog can begin speaking in under 500 ms in certain cases, measured from the moment you stop talking. The part worth stealing is not the dog. It is that no LLM is anywhere near the action decision.

What it actually does

Most AI NPC demos are dialogue wrappers: the LLM talks in character and the game does nothing it was not already scripted to do. The two chronic failures are weak world agency and latency, and you can watch demos hide the second one by cutting between the player speaking and the character replying.

Varkos is built the other way around. Kalogiros shows it taking a conditional instruction, registering a future trigger, waiting for an actual arrow to hit, then executing the rest of the plan. It searches real world state for a named object instead of inventing an answer, gets the wrong sword, gets told so, and offers to keep looking. It plays hide-and-seek as a persistent goal with movement, waiting and completion conditions rather than as a line of dialogue.

None of that is pre-scripted, and all of it has to happen fast enough that you never notice the machinery. Which is where the budget comes in.

The budget says one thing loudly

Look at the copper bar. Speech recognition, action selection and speech synthesis together cost less than a tenth of what the language model costs. If you let the LLM also decide what the character does, you do not add a second LLM call, you add a longer one with more tokens and more retries, and the whole illusion collapses.

So Kalogiros took the decision away from it.

ALE: the idea worth stealing

He calls the replacement ALE, an Action Latent Encoder, and describes it as a hybrid of embeddings, small classifiers, explicit rules and traditional machine learning. It detects structure, negation, commands, continuation, pronouns and sequences. "Pick up the sword and bring it to me" becomes two linked action slots. It embeds both the raw text and the extracted structure, compares that against action prototypes, and runs a separate classifier to decide whether the turn was a command, a question, chat, a clarification or something complex.

It is closer to a mail sorter than to a mind. It does not understand your sentence, it reads the address off the front and drops it in the right bin. That is why it costs 20 ms instead of 500.

The genuinely novel bit is that ALE also ingests the game's world state as JSON and tries to match your request against what is actually there. Say "grab", "fetch", or "go get the damn sword you fool" and you land in the same slot. If context is missing it pulls from earlier turns; if it still cannot resolve, it falls through to a clarification and the dog asks what you meant, which is a far better failure mode than a confident wrong action.

The local fine-tuned LLM comes in after the decision is made. It fuses persona, emotional state, recent history and the chosen action into something the dog says, then a grounding pass tries to strip hallucinations. If that fails, the dog falls back to a cached line. If there is time budget left, the response gets reprocessed.

Two honest caveats, both his. ALE needs a per-game version, because it has to know that game's action set and world state schema, so it is not plug and play. And his claim that it performs "surprisingly close to large LLMs" at action selection comes from limited internal evals that he explicitly declines to call a benchmark. Take it as an existence proof, not a result.

The audio stack is the part you can run today

Speech to text is Qwen3-ASR 1.7B, Alibaba's open ASR model, with custom kernels and a harness Kalogiros wrote around it. Qwen3-ASR does not stream out of the box, so he stitches rolling partials to hold 40-80 ms whether you say "hey" or monologue for a minute. Turn-taking uses Silero and a second VAD-like method he calls turnpipe, plus lexical analysis on the transcript to guess whether you actually finished a thought or just paused mid-sentence. He says that ASR harness will be open sourced, but has not been yet.

The synthesis side already is. PocketTTS-Raven is his optimized runtime for Kyutai's Pocket TTS, one C++ file on ONNX Runtime, no Python and no GPU, with a browser demo you can try right now.

He also runs an optimized Qwen3-TTS for lines that need better emotional control, and falls back to PocketTTS when a generation is running long. The trick underneath is unglamorous and effective: pre-generate a separate voice per emotion, angry, sad, neutral, confused, keep them all resident in memory, and pick at speak time instead of conditioning at generate time.

What he admits does not work

Local models fast enough for this loop are not good at holding a thread. Over a long session the conversation drifts and the dog starts to seem confused. He expects hardware and small-model quality to close that gap, and until it does, it is the real ceiling on the experience.

Reaching for a cloud model does not save you either, and the reason is funny in a bleak way. Big models are still too slow for a 500 ms budget. Ultra-fast providers like Cerebras solve the latency outright, but he found the models they serve were poor conversationalists, which is a rough flaw in a companion whose entire job is company.

One subsystem still calls a large cloud model on purpose: slow personality evolution. It runs off the real-time path, turning notable interactions into evidence for gradual changes to the character's traits and emotional homeostasis. His demo dog starts as a demon who finds his canine body humiliating and can end up fetching toys and asking to be called a good boy. Changes are versioned and reversible, which is the kind of detail that tells you someone has actually watched a character evolve itself into something unusable.

What a builder should take from this

Varkos is not a product you can install. Only PocketTTS-Raven is public; the ASR harness is promised "soon" and the rest "eventually", which in side-project time means treat it as unreleased. The write-up is the artifact, and it is a good one, with the videos to back the claims.

The transferable lesson generalizes well past games. If you are building a voice agent, a phone bot, a kiosk, a wearable, anything where a human is waiting for a reply, the LLM is your entire latency budget. Every millisecond you can move out of it, you should. Routing, slot filling, intent, negation, entity resolution against known state: these are solved problems with 20 ms answers, and they were solved before the transformer existed. Let the model be the voice, not the dispatcher.

Key Takeaways

• The LLM is the budget. Speech to text, action selection and speech synthesis together cost 80-160 ms; the language model costs 300-600 ms. Optimize accordingly.
• Classical NLP is back on the hot path. ALE, a hybrid of embeddings, small classifiers and explicit rules, picks the action and decomposes the plan in 2-20 ms, and it reads game world state so answers stay grounded in real objects.
• Grounding beats a bigger model. Feeding world-state JSON into the decision layer is what stops the companion inventing items, and it costs nothing at inference time.
• Fast cloud inference is not a fix. Kalogiros found the sub-100 ms providers served models he judged poor at sustained conversation, so latency and conversational quality were still a trade.
• One component is actually shippable. PocketTTS-Raven runs Kyutai's Pocket TTS at roughly 33x realtime on an M4 Max with about 30 ms to first audio, on CPU, in a single C++ file.
• Believe the videos, not the eval. The action-selection quality claim rests on limited internal testing the author refuses to call a benchmark. The demos are the evidence here.

Sources: Pantelis Kalogiros, "I Built an AI Companion That Actually Plays With You", PocketTTS-Raven on GitHub, Qwen3-ASR, Kyutai Pocket TTS, Hacker News discussion