Beating the Round-Trip: A Two-Plane Architecture for Real-Time Voice AI

Why a voice agent lives or dies in the first half-second

Picture calling a support line and talking to an AI agent. You ask your question, you stop talking - and then there's a pause. Half a second of silence. It's just long enough that you start to wonder if it heard you, or begin repeating yourself. That awkward little gap is the single hardest thing to get right in voice AI.

Here's the uncomfortable truth: in voice, latency is the product. In a chat window, half a second is invisible. On a phone call, half a second of silence is a held breath - the caller thinks the line dropped, talks over the agent, or quietly loses trust. Every millisecond between "the caller stopped talking" and "the agent starts talking" is dead air, and dead air is exactly where a voice agent stops feeling human.

This post is about the architecture we built to close that gap. The short version: instead of running the whole voice agent as one big program, we split it into two - a lightweight edge that runs the fast, real-time loop the caller hears, and a heavier brain that does all the slow preparation and bookkeeping behind it - and we wire them together so the caller never waits on the slow part. If terms like that are new to you, don't worry: we'll build the whole picture up from scratch, starting with what actually happens during a call.

The voice pipeline, in four stages

Strip away the buzzwords and every voice agent is just a small loop that repeats, turn after turn, for the entire call:

1. Listening (turn detection / VAD). Work out when the caller starts talking and - just as importantly - when they have stopped. This is what tells the agent "okay, it's my turn now."
2. Speech-to-text (STT). Turn the caller's audio into text.
3. Text-to-text. Hand that text to an LLM and get back the reply text. This is the "thinking" step.
4. Text-to-speech (TTS). Turn the reply text back into audio and play it to the caller.

That is the whole thing. Listen, transcribe, think, speak - on repeat. And this little loop is sacred: it has to run in real time, every single turn, with nothing getting in its way. Every millisecond a stage stalls is a millisecond of dead air the caller actually hears. So the golden rule of the audio path is simple: keep it to these four stages, and nothing else.

The catch: the "text-to-text" step is hiding a monster

Stage 3 looks innocent - "just call the LLM" - but in a real product it is anything but. Before you can call the model, you have to build the prompt, and a good prompt for a serious production agent is enormous and, crucially, different on every single turn:

• A system prompt assembled from the business's policies, tone of voice, and rules
• Dynamic strings stitched in per conversation - the caller's name, their account details, what has happened so far in the call, which tools have run and what they returned
• Tool definitions the model is allowed to call (look something up, update a record, send a message)
• Retrieved knowledge relevant to whatever the caller just asked
• Compliance and guardrail instructions that change depending on the topic
• The right model, language, and voice settings for this specific business

Assembling all of that is genuine work: database reads, knowledge retrieval, string templating, version resolution. It is exactly the kind of slow, stateful processing you must not drop into the middle of a real-time audio loop. Do that, and stage 3 balloons, the pipeline stalls, and the caller sits there listening to the agent "think."

So we have a real tension:

• The pipeline has to stay tiny and fast - VAD → STT → text-to-text → TTS, and nothing else.
• But that innocent-looking "text-to-text" step secretly needs a big, slow, constantly-changing prompt to produce a correct, compliant, on-brand answer.

This is the question the whole architecture exists to answer: how do you orchestrate an enormously complex, always-changing prompt without ever interfering with the simple, sacred pipeline the caller depends on?

The split: a thin edge and a thinking brain

The answer is to stop doing both jobs in the same place. We cut the agent in two along that exact seam: one process that runs the sacred pipeline, and one process that does all the heavy prompt orchestration.

This split wasn't a whiteboard idea - it came from a bug. Early on, the audio process did some of the heavy lifting inline: a synchronous, CPU-bound computation running right next to the audio loop. Whenever it fired, it blocked the very thread that was supposed to be moving audio, and the caller heard it - tiny stutters and small delays creeping into the conversation. The fix wasn't to make that one computation faster; it was to get it out of the audio path entirely. That is the principle behind everything here: the audio server should do nothing but the critical pipeline, as fast as possible, while every heavy or slow operation - anything at all - happens somewhere else. Separation of concerns, enforced by a process boundary.

The edge (the fast loop)

The edge runs the four-stage pipeline - exactly that, and nothing more. Crucially, that pipeline includes the LLM call: the edge isn't just the mouth and ears, it does the talking-back too. It is deliberately, almost aggressively, thin - but thin means "no slow orchestration," not "no thinking":

• Receive caller audio frames
• Run voice isolation and turn detection (deciding whose turn it is)
• Transcribe the caller's speech (STT)
• Run the text-to-text step - call the LLM with a prompt the brain has already prepared
• Turn the reply back into audio and stream it to the caller (TTS)
• Handle barge-in - stop talking the instant the caller does

Notice what is not on that list: building the prompt, retrieving knowledge, executing tools, writing to the database. The edge holds no business logic, no rulebook, no knowledge base. It runs the model, but it does not even build the prompt it feeds it - it is handed a finished one. So "thin" doesn't mean it can't think; it means it is optimized for one metric only: how quickly it can turn a caller's words into the agent's words.

The brain (the "thinking")

The brain is where all the heavy prompt orchestration - the monster from stage 3 - actually lives:

• Building the full, dynamic prompt for the next turn (system prompt + dynamic strings + tool-call state)
• The system prompt and its versioning
• Tool definitions and tool execution - the actions that actually change things
• Conversation persistence, logs, and analytics
• Knowledge retrieval
• The compliance and guardrail state machine

The brain is allowed to be heavy. It is the source of truth for the conversation, and it is where auditability lives: every message, every tool call, every decision is recorded here. For anyone running agents in a high-stakes or regulated setting, that single-source-of-truth property is not a nice-to-have - it is the entire point.

A simple way to remember it: the edge runs each turn fast; the brain decides what each turn should know. (If you've heard the terms before, that's the classic "data plane" versus "control plane" split.) Keeping the two apart lets each be built for a completely different goal without fighting the other - one chases speed, the other chases correctness.

"But doesn't talking to a second service add latency?"

This is the obvious objection, and in the naive design it is correct. If, on every user turn, the edge had to make a network round-trip to the brain - "here is the transcript, please build a prompt, call the LLM, run the tools, and send me back the answer" - we would have added a network hop to the critical path. We would be slower, not faster.

We avoid that with three techniques: a persistent connection, an optimistic edge that speaks before the brain catches up, and edge-owned conversation state.

1) A persistent WebSocket, not request/response

The edge and the brain are connected by a single long-lived WebSocket that opens when the call starts and stays open for the entire conversation. Think of it as keeping someone on the line for the whole call, instead of hanging up and redialing every time you have something to say. This matters more than it sounds.

A fresh HTTP request pays for DNS, the TCP handshake, and the TLS handshake every single time - easily 100ms+ of pure overhead before one useful byte moves, and that tax lands on every turn. A persistent WebSocket pays that cost once, at call setup, while the caller is still hearing the first ring. After that, sending a message is just writing bytes to an already-warm, already-authenticated socket. We authenticate the connection a single time on upgrade and reuse it for the life of the call.

The connection is also bidirectional and full-duplex. The brain does not have to wait to be polled - it can push events to the edge the moment they happen: "a background workflow wants the agent to say something," "this conversation should escalate now," "here is an updated prompt." No polling loop, no waiting for the next request to deliver news. Both sides talk whenever they have something to say.

So talking between the two halves is about as cheap as it gets: no setup, no re-authenticating, no waiting to be asked - just a quick message down a pipe that is already open.

2) The edge speaks optimistically - the brain reconciles after

Here is the move that resolves the whole tension - the monster prompt versus the sacred pipeline: the edge runs the LLM itself, but the brain builds the prompt for it in advance.

This sounds like it contradicts everything above - the brain does the thinking! - but the trick is in what the brain sends and when. The brain does not wait for a user turn to build a prompt. Instead, it does the heavy orchestration continuously, in the background, and whenever conversation state changes it pushes a fully-resolved prompt snapshot to the edge ahead of time: a compact, ready-to-use package containing

• the complete system prompt,
• the tool definitions,
• tool-call status messages (what is running, what just ran),
• the resolved voice and language settings, and
• a short list of which tools actually change something in the real world (issuing a refund, sending a text), so the edge can handle those moments carefully.

The edge keeps the latest snapshot warm in memory, and the brain keeps it fresh - every time something changes (a tool runs, the policy updates, the call moves to a new topic), a new snapshot is pushed. So at any given moment the edge is holding a ready-to-go prompt that already reflects everything the brain knows. When the user finishes a turn, the edge does not call the brain and wait. It takes that warm snapshot, drops in the fresh transcript, calls the LLM directly, and starts streaming the answer to text-to-speech sentence by sentence the moment the first sentence is ready.

In other words: all the slow, complicated prompt work happens off to the side, ahead of time. By the time the caller stops talking, the hard part is already done and sitting in memory. The pipeline stays four stages long; the complexity lives next to it, never inside it.

Only after the caller is already hearing the answer does the edge report back to the brain: here is what I said, here are the tool calls the model wanted, here are the logs. The brain then does the slow, off-the-critical-path work - persist the messages, execute the real tools, record analytics.

This is optimistic generation with reconciliation. The caller hears a response built from local, warm state immediately; the authoritative system catches up in the background. The expensive part of "thinking" - assembling everything the model needs to know - is done before the turn, so it never sits on the clock.

It also fails gracefully. If the report back to the brain fails, we keep the local answer. The caller already heard it, and re-running it on the brain would make the agent repeat itself or execute a tool twice. The edge's spoken word is treated as having happened, because it did.

3) The edge owns the live transcript

There is a subtle trap in any two-process design. If the brain is the source of truth for the conversation, and the edge has to ask the brain for the history on every turn, the edge is always one round-trip behind.

So we don't do that. The brain seeds the edge's conversation history exactly once, at the start of the call (or on reconnect). After that, the edge owns the live transcript. It appends each user turn and each spoken assistant turn locally, the instant they happen. When it builds the next prompt, the most recent turn is already there - no waiting for the brain to persist a message and push it back.

The brain still persists everything for the record. But the edge never blocks on that persistence to keep the conversation moving. The system of record and the speaking loop are decoupled: one optimizes for durability, the other for speed.

‍

What the brain pushes, what the edge sends

Because the connection is full-duplex, the two sides have a clean division of labor over the same socket.

The brain pushes to the edge, unprompted, whenever it happens:

• Updated prompt snapshots when conversation state changes
• Proactive prompts - something running in the background wants the agent to speak up on its own (a reminder, a status update)
• Escalation signals - hand this call to a human, now
• Status updates - whether the agent is currently busy producing a reply

The edge sends to the brain, after the caller is already being served:

• Save this user message / assistant message
• Here is the result of the reply I just produced (logs, plus any tools that need to run)
• Persist these latency metrics
• The caller pressed a keypad digit; run the matching workflow

Notice the asymmetry: almost everything the edge sends to the brain happens after it has already responded to the caller. The brain's work is bookkeeping and action execution, deliberately pushed off the critical path. The caller's experience is gated only by the edge's local loop.

Why this design pays off

Speed where it counts. The latency-critical loop - audio in, turn detection, LLM, audio out - runs entirely on a thin process with a warm prompt and a local transcript. The heavy work runs on the brain, off the clock. The slow stuff no longer sets the pace.

Resilience. A persistent connection with regular health checks notices a dead connection quickly. And because the edge already holds the warm snapshot and the live transcript, it can keep a conversation going through a brain hiccup instead of going silent the instant the brain is slow.

Clean scaling. The two halves have completely different resource profiles. The edge is bound by real-time audio and concurrent calls; the brain is bound by LLM calls, database writes, and tool execution. Splitting them lets each scale independently, against its own bottleneck, instead of over-provisioning one giant service to satisfy two contradictory load shapes.

Auditability without the latency tax. The brain stays the single source of truth - every message, tool call, and decision recorded. But because persistence happens off the critical path, full auditability costs the caller nothing in latency. In regulated industries, that combination - fast and fully recorded - is exactly what you need and rarely what you get.

A clean security boundary. The thin edge handles raw audio, telephony, and the real-time model call; the brain holds the business logic, the rulebook, and the keys to systems that actually change customer data. If something goes wrong in the noisy, internet-facing audio half, the damage it can do is small by design.

Tradeoffs we accepted

No architecture is free. Being honest about the costs:

• Keeping the warm copy fresh is real work. That ready-to-go prompt sitting on the edge is a copy, and copies can fall out of date. We have to be careful about when the brain pushes a new one, and how the edge blends what the brain knows (tool results) with what only the edge knows (the latest thing the caller just said). Get it wrong and the agent answers from stale information.
• Optimism means occasional reconciliation work. Because the edge speaks first and reports later, the system has to handle the cases where reality diverges - a report that fails to land, a tool the brain must still execute after the words were already spoken. We treat the spoken word as ground truth and reconcile around it.
• Two processes is more operational surface. Two services, one more connection to monitor, health checks to tune, a protocol to version. We pay this for the latency and scaling wins, and on voice it is worth it.

What's next

The split between a thin edge and a thinking brain is not the end state - it is the foundation for everything we want to do next in voice. Once the speaking path is decoupled from the authoritative one, you can get more ambitious about both halves: richer pre-pushed context on the edge, smarter reconciliation on the brain, and tighter guardrails that never cost the caller a millisecond.

That future still needs people. It needs engineers who are comfortable treating a voice agent as a distributed system - weighing speed against consistency, freshness against simplicity - not just a model with a microphone in front of it. At Notch we are building that infrastructure for reliable AI in regulated industries, and we are hiring a top-notch team to help shape it.

Conclusion: turning a hard latency problem into an architectural advantage

The instinct in voice AI is to make a single service smarter. The better move is to make it thinner where it matters. By splitting the agent into a thin edge and a heavyweight brain, connecting them with a persistent WebSocket, and letting the edge speak optimistically from warm state, the round-trip to the brain stops being a tax on every word the caller hears. The brain still thinks, still records, still acts - it just does it without making the caller wait. And a caller who never waits is a caller who forgets they are talking to a machine.

‍

‍