NPCTalk Open-Sources Voice-Cloned GTA V NPCs That Reply in 1.15 to 2.2 Seconds

TL;DR

NPCTalk is an MIT-licensed repo that makes GTA V characters talk back. You speak into your mic, the NPC you are facing answers in character, in a cloned voice, and can act on what you said. The author's in-game logs put the gap between the end of your sentence and the NPC's first word at 1.15 to 2.2 seconds with push-to-talk. The voice runs on Fish Audio Drama 3 open weights on a rented H100 at about $4 an hour. What you can reuse elsewhere is the build log, which times each step and includes a voice-quality claim the author later took back, and eight agent skills that let a coding agent do the same wiring in another game.

What it is

NPCTalk comes from Universal Modder, a toolkit for letting AI coding agents mod games you own. The repo went up on October 15 and had 195 stars and 16 forks on October 23. It ships in three parts:

• Eight skills in the Agent Skills format: the full make-NPCs-talk loop, hosting the voice server, cutting reference clips, writing dialogue, choosing actions, voice input, measuring voice likeness and recording showcase clips.
• The npctalk CLI (Python): health, say, voice make, voice upload, and check, which refuses to let audio, transcripts, keys or real server URLs into a commit.
• A working GTA V example: a ScriptHookVDotNet C# mod with an open mic, 19 characters, their homes, actions and a dev mode that logs the timing of every step.

It installs as a plugin for Claude Code, Codex, Gemini CLI and VS Code Copilot, and Cursor or OpenCode pick it up from a clone. The README's demo is what you would expect from GTA: Lamar roasts your hoodie, a cop tells you to file a civil claim and flips you off, and Michael defends his robe as if it were a business suit.

The pipeline

A spoken exchange takes five hops. An open mic with voice activity detection hears you stop talking. NVIDIA's Nemotron ASR on fal transcribes what you said. An LLM on Cerebras (qwen-3.8-27b, reasoning off) writes the reply using the game's scene facts. At the same time TypeSafe Jev decides whether you gave an order the NPC should physically carry out. Then the Fish Audio voice server on Modal streams audio back, and the mod pans it to where the NPC stands on screen, fades it with distance, and moves the NPC's mouth while a subtitle shows the line.

Most of that code has no idea it is running inside GTA. The dialogue, the LLM and voice clients, playback, speech-to-text and the action picker all compile into a console app and were tested there before the game loaded them. A game only has to provide four hooks: who you are talking to, what the scene is, a way to play audio at the NPC, and a list of actions the NPC can perform. That is why the skill carries notes for Unity, Unreal, Bethesda games and Minecraft as well.

The two-second budget

The author's target was simple: the answer has to start within about two seconds, or it stops feeling like a conversation.

Two choices made the budget work, according to the log. One was a fast LLM with reasoning switched off, because a reasoning model on its own would have used the whole two seconds. The other was a voice server that streams. The NPC starts talking while the rest of its line is still being generated, so you hear audio after the first 0.35 to 0.55 seconds of synthesis, not after the whole sentence renders.

The server also encodes every reference clip in its volume at startup, so the game sends a voice name. A clip sent by URL cost about 9 seconds the first time. While the H100 wakes, which takes one to three minutes from cold, the mod falls back to MiniMax voices on fal, which needs no GPU.

The GPU bill

The Modal H100 costs about $4 an hour while it is up and scales to zero five minutes after the last request. The mod pings it every 90 seconds of silence while you play, so you pay roughly for your play session. The safety notes put a 12-hour keep-warm at $45 to $50. They also warn that the voice server's /health endpoint needs no token and wakes the GPU, so your server URL is effectively a secret: anyone who has it can run up your bill.

Drama 3 itself is large for a TTS model. The model card describes 35B parameters with 3B active per token, a mixture-of-experts backbone and FP8 experts. NPCTalk serves it with SGLang-Omni behind the same POST /speak API it had used for the smaller S2 Pro. Switching models meant changing one URL in the game.

The retraction in the build log

To check voice quality, the author built a scorer. Voice match is ECAPA speaker-embedding similarity between each generated line and the centroid of 60 real clips of the character that were never used as references. Word error comes from Whisper large-v3. The test set was ten lines spoken by Lamar.

The log is upfront about one result. Lowering Drama 3's sampling temperature looked like a big win on one run (0.715 against 0.637). A second identical run gave 0.645. The log calls the first result luck, notes that it had already been reported, and says the correction was posted where the claim was made, which is more than most voice-cloning READMEs will admit to.

The improvement that held up was best-of-N with checks. The server renders several takes and keeps the one closest to the reference voice. Without checks this went wrong: about one take in ten rambled to the token limit (47.6 seconds for a 15-word line), and one demo clip ended with a 14-second whimper. Adding a length cap based on word count, a small ASR that has to hear the right words, and a pick made by two speaker models together raised Drama 3 to 0.704, better than a single take on 9 of 10 lines. Think of a casting director who records eight takes, bins the ones where the actor forgot the script or never stopped, and keeps the closest of the rest. The log adds two rules for anyone copying the method: run every comparison more than twice, and never let a picker grade itself.

The best number in the whole project, 0.785, came from a LoRA fine-tune of S2 Pro, and getting it took fixing two bugs in fish-speech's trainer: unscaled codebook embeddings, and the older S1 prompt layout used where S2 Pro's chat layout belonged. Before those fixes the fine-tune produced garbage. The default stays on untrained Drama 3 because it handles [crying] and [laughing] tags better and starts speaking faster.

Making characters sound like themselves

The first replies, the log says, sounded like a polite assistant doing an impression. Every prompt now carries a fresh random sample of 30 of the character's real lines, so the model picks up their slang, rhythm and catchphrases without parroting the same lines. Rules keep each reply to the words said out loud, one or two short sentences and a single delivery tag. Jev checks whether you gave an order before the line is written, so the NPC grumbles while doing what you asked instead of refusing. Code then strips refusals and *actions*, caps replies at 45 words, filters slurs by default and blocks racial insults from every character.

The catches

• It is not one install. You need accounts with Modal, fal, Cerebras and optionally TypeSafe, plus Python, uv, ffmpeg and the .NET SDK for the GTA build. The author's own to-do list includes "a hosted backend behind one key, so players don't need four accounts and a GPU deployment." Setup is the hardest part of the game.
• Voice rights. The GTA clones were cut from voice-line compilations for a private, single-player build. That audio and the clones are kept out of the repo, and npctalk check blocks them from commits. The safety notes say outright that game characters are voiced by real people and their voices belong to them. If you ship something built on this, you need recordings you hold the rights to.
• Single-player only. The rules forbid running the mod in GTA Online or getting around anti-cheat, DRM or ownership checks.
• The weights license is not MIT. NPCTalk's code is MIT, but Drama 3 ships under the Fish Audio Community License. It is free for research and non-commercial use, allows commercial use only while your annual revenue is under $500,000, and requires a separate agreement for any hosted service.
• Self-reported numbers. The latency figures come from the author's own in-game logs, and the likeness scores cover one character and ten lines on the author's scorer. Use them to see the method; nobody has reproduced them independently.

Key Takeaways

• NPCTalk (MIT) makes GTA V NPCs answer your voice in character and in a cloned voice, 1.15 to 2.2 seconds after you stop talking with push-to-talk, according to the author's in-game logs.
• The budget fits because of a fast LLM with reasoning off and a streaming Fish Audio Drama 3 server, which produces first audio in 0.35 to 0.55 seconds on a Modal H100 costing about $4 an hour.
• A game only has to supply four hooks (target, scene, audio at the NPC, actions), which is why the skills cover Unity, Unreal, Bethesda and Minecraft as well.
• Best-of-8 with length and ASR checks raised Drama 3 voice match from 0.643 to 0.704; a fine-tuned S2 Pro reached 0.785.
• The build log retracts a temperature "win" that did not survive a second run, and it is worth reading for the method alone.
• Check the Drama 3 license and your voice rights before building a product on this: the code is MIT, the weights and the GTA voices are not.

Sources: NPCTalk on GitHub, NPCTalk: How it was made, make-npcs-talk skill, NPCTalk safety rules, Fish Audio Drama 3 model card, Fish Audio Community License, Universal Modder, SGLang-Omni