christos-antono/NPCTalk



NPCTalk

By Universal Modder, the toolkit that lets AI agents mod almost any game you own · Join the Discord

Make any NPC in any game talk back, with Fish Audio voices.
Skills, a self-hosted voice server and tools that let an AI coding agent wire it up: the player speaks, the NPC answers in character, in a cloned voice, about a second later, and acts on what was said.

Claude Code · Codex · Cursor · Gemini CLI · Copilot · OpenCode · how it was made · GTA V example · Discord · MIT

Built first for GTA V: walk up to anyone in Los Santos and just talk. Lamar roasts your hoodie, a cop tells you to file a civil claim and flips you off, Michael defends his robe like it's a business suit. Every line is written live in character and spoken in that character's cloned voice by Fish Audio Drama 3 (or S2 Pro), self-hosted on a serverless GPU. Most of the code doesn't know it's in GTA, so the same pieces work in other games: the skills say what a game has to provide, and how to build the rest.

Install

Pick your agent. Each gets the same skills (Agent Skills format), the fal MCP server and the npctalk CLI.

| Agent | Install | |---|---| | Claude Code | /plugin marketplace add christos-antono/NPCTalk
/plugin install npctalk@npctalk | | Codex | codex plugin marketplace add christos-antono/NPCTalk
codex plugin add npctalk@npctalk | | Gemini CLI | gemini extensions install https://github.com/christos-antono/NPCTalk | | VS Code / Copilot | Enable chat.plugins.enabled, run Chat: Install Plugin From Source, enter this repo's URL | | Cursor | Clone it (Cursor reads AGENTS.md and .cursor/mcp.json) | | OpenCode | Clone and run opencode inside it (opencode.json adds the skills and the fal MCP server) | | Skills only (any agent) | npx skills add https://github.com/christos-antono/NPCTalk | | Anything else | git clone https://github.com/christos-antono/NPCTalk and start your agent inside it |

Inside a clone, .agents/skills and .claude/skills are copies of skills/, and AGENTS.md holds the instructions (CLAUDE.md and GEMINI.md point to it).

The npctalk CLI. Plugin installs and clones put it on PATH (bin/). Anywhere else: uv tool install git+https://github.com/christos-antono/NPCTalk.

Accounts and keys (environment variables, never files):

| What for | Service | Variable | |---|---|---| | The voice server (GPU) | Modal | modal setup; NPCVOICE_TOKEN = the value you put in the npcvoice-auth secret | | Speech-to-text, MiniMax voices, the MCP server | fal | FAL_KEY | | The NPC's lines (any OpenAI-compatible LLM) | Cerebras | CEREBRAS_API_KEY | | Actions and spoken orders | TypeSafe | TYPESAFE_API_KEY |

You also need Git, Python 3.10+, uv and ffmpeg. The GTA V example needs Windows, the .NET SDK (8+) to build it, and the game.

Try it

Make the villagers in my game answer when I talk to them, in their own voices.

Set up a Fish Audio Drama 3 voice server on Modal and clone a voice from narrator.wav.

In GTA V story mode, let me talk to anyone near me with my mic.

My NPC's cloned voice sounds thin. Is it the reference clip or the model? Measure it.

Record five 8-second clips of NPCs arguing that look like normal gameplay.

The agent starts with the make-npcs-talk skill and runs the same loop every time: 1. recon the game: version, how code runs in it, single-player or not; 2. pick the engines (voice, LLM, speech-to-text, actions); 3. stand up the voice server first, outside the game, and test it with npctalk say; 4. make reference voices from recordings you have the rights to; 5. one NPC answering typed text in the game: the four hooks, nothing else; 6. voice input (push-to-talk, then the open mic); 7. characters: personas, their own lines, scene facts; 8. actions, and doing what they're told; 9. verify in the real game, with every step's timing logged; 10. show it, and measure the voices before claiming anything; 11. write a field note for the next agent.

What's inside

Skills (skills/, Agent Skills format)

| Skill | What it does | |---|---| | make-npcs-talk | The whole loop, the four hooks a game must provide (who's talked to, scene facts, voice at the NPC, actions), engine choices, latency budget, costs, verification, rules; per-engine notes for GTA V, Unity, Unreal, Bethesda, Minecraft and your own game | | fish-voice-server | Host Fish Audio Drama 3 or S2 Pro on Modal (or fal) behind one streaming API; settings that measurably help likeness; best-of-N for pre-rendered lines; keep-warm and cost; the S2 Pro fine-tune; every gotcha | | voice-references | From a recording you have the rights to, to clean reference clips, a 25 s clone clip and mood clips, on the server | | npc-dialogue | Personas, a fresh sample of the character's own lines in every prompt, scene facts, short spoken lines with delivery tags, output cleaning, content rules, speed | | npc-actions | What NPCs physically do while they talk, and following spoken orders, with TypeSafe Jev; per-action thresholds; mapping to animations | | voice-input | Open mic with voice activity detection, push-to-talk, speech-to-text, the NPC's own voice and the room's noise | | voice-likeness-eval | Measure whether a voice change really helped: speaker similarity to held-out real clips, word error, enough runs, independent checks | | npc-showcase-clips | Gameplay-style clips of conversations: fixed gameplay camera, lines rendered first, sync, 5-10 s cuts |

The npctalk CLI (Python). Every command has --help with examples.

| Command | What it does | |---|---| | npctalk health | Is the voice server up? Waits through a cold start | | npctalk say | Speak a line in a voice, like a game does; prints time to first audio, writes a WAV | | npctalk voice make | Recording → voices//clone.wav, ref_0..2, mood clips, with transcripts | | npctalk voice upload | A voice folder → the server's Modal volume | | npctalk check | Blocks what must never ship: audio, transcripts, keys, real server URLs |

The voice server (server/): drama3.py (Fish Audio Drama 3, the default), s2pro.py (S2 Pro, with fine-tuned voices), keepwarm.py, fal/ (S2 Pro on fal over a realtime WebSocket), finetune/ (dataset, LoRA, and the scorer behind every number here).

The GTA V example (examples/gta-v): the full mod (open mic, any NPC, 19 characters, homes, actions, dev mode, a director for recording clips), its tools, and a field note.

Field notes (knowledge/): one per game where NPCs were made to talk. Add yours.

How good are the voices?

Measured on one character (Lamar, ten test lines): voice match is speaker similarity to 60 real clips that were never used as references. Details and the method: skills/voice-likeness-eval.

| Setup | Voice match | Word error | |---|---|---| | ElevenLabs instant clone | 0.612 | 10.5% | | Fish Audio Drama 3, default settings | 0.643 | 7.1% | | Drama 3, tuned sampling, one take | 0.660 | 7.9% | | Drama 3, best of 8 takes, checked | 0.704 | 8.6% | | Fish Audio S2 Pro, its best reference clip | 0.707 | 5.6% | | S2 Pro fine-tuned on the character (needs training) | 0.785 | 7.5% |

Without training, Drama 3 and S2 Pro tie on likeness; Drama 3 acts [crying] and [laughing] better and starts talking faster. One early result here didn't survive a second run: how that was caught.

Rules it follows

• Single-player games, or servers you run. It never injects into online games with anti-cheat (GTA V: never

launch GTA Online with a script hook installed) and never bypasses anti-cheat, DRM or ownership checks.

• Voices you have the rights to. It never commits or publishes voice clips, cloned voices, game audio or a

game's dialogue; npctalk check blocks them.

• No secrets in files. Keys come from environment variables, and voice server URLs stay private (their

/health wakes the GPU).

• It asks before driving your keyboard and mouse in a game, installing into game folders, keeping a GPU up,

or publishing anything.

• It measures before it claims a voice setting is better.

Full reasoning: skills/make-npcs-talk/references/safety.md.

Community

NPCTalk is a Universal Modder project. Universal Modder's skills and tools cover the rest of a mod (engine recon, decompiling, fal-generated art, 3D and audio, in-game testing, showcase videos), and NPCTalk adds NPCs that talk back. Install both and an agent can take a game from "can I mod this?" to characters you can talk to.

Show what you made, ask for help, and follow along in the Universal Modder Discord.

Credits

• Voices: Fish Audio (Drama 3, S2 Pro, fish-speech), served with

SGLang-Omni on Modal and fal.

• Lines: Cerebras. Speech-to-text: NVIDIA Nemotron ASR on fal. Actions:

TypeSafe Jev.

• GTA V: Script Hook V,

ScriptHookVDotNet, NAudio.

• A Universal Modder project; the repo layout and principles come from universal-modder.
• Built in real sessions with an AI coding agent (Claude Code), from the first line of the mod to the measurements.

MIT licensed. GTA V and its characters belong to Rockstar Games; this project isn't affiliated with Rockstar, Fish Audio or any service above.