Build a Real-Time Voice Agent in Python: Mic to Answer in Under a Second
Voice agents used to be assembled from three separate systems: a speech recognizer that turned audio into text, a language model that produced a reply, and a speech synthesizer that read it aloud. Each hop added latency and each hop lost something, particularly the parts of speech that carry meaning without words, like a pause that signals uncertainty or a rising tone that turns a statement into a question.
This tutorial builds a different kind of agent. You will connect a Python client to a realtime speech-to-speech model over a WebSocket, stream microphone audio in small chunks, and play the response back as it is generated. The whole thing is a few hundred lines. The interesting part is not the code volume, it is the decisions about how to handle turns, tools, and failure.
Why Speech-to-Speech Changes the Design
With a pipeline, the transcript is the interface. The model reads words and writes words, so tone, timing, and hesitation are discarded before reasoning begins. With a speech-to-speech model, audio goes in and audio comes out, and the model can react to how something was said rather than only to what was said.
The practical benefit is latency. A pipeline accumulates delay at each stage, and each stage has its own buffering strategy. A single realtime session removes two of the handoffs entirely, which is usually the difference between a conversation that feels responsive and one where people talk over each other because the silence felt too long.
Step 1: Issue Ephemeral Keys From Your Server
Never put a long-lived API key inside a client application. The correct pattern is a small server endpoint that authenticates your user, then asks the provider for a short-lived session credential and hands that to the browser or desktop client. The credential expires quickly and can be scoped to a single session, which limits what a leak costs you.
On the server, a request to the realtime sessions endpoint with your standard key returns a client secret. Your Python service stores nothing sensitive and returns only that secret. If you are building a local tool for yourself, you can skip the indirection, but write the code as if you were going to ship it, because that refactor is the one people postpone forever.
Step 2: Open the Session and Configure It
With a credential in hand, the client opens a WebSocket connection and sends a session configuration message. That message is where most of your product decisions live: the model, the voice, the system instructions, the input audio format, the output audio format, and the turn detection strategy.
import asyncio, json, os, websockets
async def run(ephemeral_key):
uri = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1"
headers = {"Authorization": f"Bearer {ephemeral_key}"}
async with websockets.connect(uri, extra_headers=headers) as ws:
await ws.send(json.dumps({
"type": "session.update",
"session": {
"instructions": "You are a concise support agent. Ask one question at a time.",
"voice": "verse",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"turn_detection": {"type": "semantic_vad", "eagerness": "medium"}
}
}))
await pump(ws)
Two fields deserve attention. Semantic turn detection asks the model to judge whether the user has finished a thought, rather than waiting for a fixed number of silent milliseconds. That is what stops the agent from interrupting someone who paused to think. And PCM16 as a wire format is deliberately dumb: raw audio frames, no container, no decoding library in your hot path.
Step 3: Stream Audio in Both Directions
The main loop has two jobs running concurrently. One reads microphone frames, base64-encodes them, and sends append events. The other reads server events, plays audio deltas as they arrive, and logs transcripts and tool calls. Getting these into the same event loop is the entire trick.
- Send audio in frames of 20 to 40 milliseconds; larger frames add perceived delay.
- Append a commit event when a chunk of speech ends, so the model can start responding.
- Play output audio deltas as they arrive rather than buffering the full response.
- Cancel the current response when new input arrives, so barge-in works naturally.
- Log every event with timestamps; you will debug latency from those logs, not from intuition.
Barge-in is worth building on day one. If the user speaks while the agent is talking, the correct behavior is to stop playback immediately, cancel the in-flight response, and process the new input. Users expect this from humans and become irrationally annoyed when software keeps talking over them.
Step 4: Give the Agent Tools
A voice agent that cannot check a system is a chatbot with a throat. Tools are declared in the session configuration and invoked through events, exactly as in text-based function calling. Keep the set small and the parameters flat, because spoken requests are messier than typed ones.
{
"type": "function",
"name": "check_order_status",
"description": "Look up the status of an order by its numeric identifier.",
"parameters": {
"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"]
}
}
When the model calls a tool, your client executes the real function and returns the result as a tool output event. Validate the arguments in your own code before touching any backend. A model that has just heard a noisy phone call can mishear an order number, and the lookup should fail safely rather than charge the wrong account.
Step 5: Budget Your Latency
Users perceive voice response time differently from text. Anything under roughly 800 milliseconds feels conversational, and beyond about two seconds people start repeating themselves or assuming the call dropped. Spend that budget deliberately instead of discovering it in production.
Measure four things separately: time from speech end to commit, time from commit to first audio delta, your own playback start delay, and time spent in tool execution. Tool calls are usually the biggest variable, so stream a short spoken acknowledgment before a slow lookup. A clipped "let me check that" buys you several seconds of user patience, which is the cheapest latency fix available.
Step 6: Ship It Without Shipping a Hole
Audio streams contain secrets by accident. People read verification codes aloud, spell out account numbers, and dictate internal details they would never type into a form. Log conservatively, encrypt transcripts at rest, and give users a way to delete a conversation, because you are now handling voice data, and voice data is identifiable in ways text is not.
Then apply the same rules agent builders apply everywhere else: scope every tool to the minimum permission it needs, require confirmation for irreversible actions, and rate limit both sessions and tool calls. A voice agent with a payment tool and no confirmation step is a compliance incident waiting for a noisy room.
The hardest part of a voice agent is not speech recognition or audio plumbing. It is deciding, in advance, what the agent must ask a human before it acts.
Where to Go Next
Once the loop works, the improvements that matter most are unglamorous. Add a reconnection path for dropped sockets. Persist conversation state so a refresh does not reset context. Add a text fallback for users in a noisy environment. Instrument the latency budget on every call so a regression shows up as a graph rather than a complaint.
Then extend the same session object with more tools, and the agent stops being a demo. Order status, calendar lookups, a search over your documentation, and a handoff to a human queue are all the same event pattern you already built. A voice agent that can actually do something is only a handful of tool definitions away from the walkthrough you just finished, and that is the point at which it becomes worth deploying.
