There's a comfortable myth going around right now: that a good enough AI model is a voice agent. Connect a speech model to a phone number and presto, you've got a receptionist who never sleeps, a scheduler, a sales rep on call around the clock.
You don't. You've got a brain in a jar.
The distance between "a brilliant AI model" and "a phone agent that sounds human and actually gets things done" is the same distance as between a genius new hire and a functioning employee. The hire is sharp. But on their first morning they don't know where anything is, can't reach a single tool, have no idea when it's their turn to speak, and will cheerfully talk straight over a customer mid-sentence. The intelligence was never the hard part. Teaching it to behave like a person on a phone is.
We've spent a long time building voice agents. Long enough to tell you, with some feeling, that the model, the part everyone fixates on, is maybe a tenth of the work. This is the story of the other nine tenths, and why it's the whole difference between a slick demo and something you'd actually trust to answer your phone.
The first thing that broke our hearts
Nobody tells you this part. A model doesn't hear a phone call. It hears a raw, continuous river of audio, and something else entirely has to decide the precise instant a human stopped talking so the model can answer. It sounds trivial. It is one of the hardest problems we work on.
Wait a beat too long and the agent feels slow, lifeless, robotic. Jump in a fraction too early and you've just cut off your own customer mid-thought, which is exactly how a real conversation curdles. We tuned and re-tuned that single judgment for months, because it turns out a quick "yes" and a long, thoughtful sentence with a pause in the middle look completely identical from the outside: a short burst of sound, then silence. One is a finished answer. The other is a person who isn't done. A human knows the difference without thinking. A machine has to be taught, painstakingly, case by case.
And early on, before we'd taught it well enough, it got one wrong. A real caller paused mid-sentence. The agent heard the silence, decided the person was finished, caught the tail end of a half-spoken word, read it as "okay, that's everything, goodbye," and hung up on them.
That moment has stayed with us. Because the bug wasn't in the model. The model was brilliant. The bug lived in the sliver of space between the sound and the brain, in the part nobody thinks about until it fails. Closing it took weeks and a small pile of careful mechanisms whose entire job is to make sure the agent never, ever mistakes a breath for a farewell.
The line is never actually silent
We assumed, naively, that the hard audio problems would be on the human's end: background noise, a bad connection, a barking dog. Those are real. But the strangest one came from the model itself.
When some models pause to think, they don't go quiet. They leak sound. A breath, a half-formed syllable, and sometimes a faint electrical hum that swells from a whisper into a drone. One caller, mid-call, asked what the buzzing was. Another stretch sounded like thirty seconds of static. You cannot fix that by choosing a smarter model or writing a cleverer prompt. We had to roll up our sleeves and do real audio engineering: learn to recognize that hum by its signature and erase it, surgically, without ever touching a single real word the agent was saying.
That was one of dozens of small wars over the sound itself. The pops a phone line makes when a call connects. The agent hearing its own voice echo back and mistaking it for the caller. The thousand tiny clicks and seams that turn synthetic speech into something that subtly screams "robot." Each one was its own afternoon, or week, of obsession. None of them are things the model does. All of them are things you only notice when they're missing, because when they're missing, the call just feels wrong in a way most people can't name.
The lie we almost told ourselves
For a while we measured how fast our agent responded by reading our own logs, and we felt great about the numbers. They were comfortably wrong, by more than a second, because the logs quietly skipped the part the caller actually feels.
So we threw out the comfortable number and built an entirely separate way to measure the truth: the real, lived gap between the moment you stop speaking and the moment the agent starts. Then we built a second tool to double-check the first, because "it feels faster to me" is not an engineering standard, and we'd already been fooled once. You cannot make a call feel instant if you're willing to lie to yourself about how slow it is.
The most dangerous thing a voice agent can do
The whole point of an agent, as opposed to a chatbot that just chats, is that it does things. Checks the calendar. Looks up the order. Books the appointment. And here we ran straight into the problem that quietly sinks most voice "agents."
Doing real work takes time. On a chat screen, a few seconds of "thinking" is fine. On a live phone call, a few seconds of silence is a human being convinced the line went dead. We learned that the obvious way of giving a model its tools could leave a caller listening to nothing for twenty, forty, ninety seconds. Unacceptable. So we rebuilt how the agent reaches the real world entirely, so that a lookup lands in about a second and the caller never feels the machinery turning behind it.
But the part that genuinely scared us was honesty. When a model can't reach a tool in time, it doesn't say "one moment." It does something far more dangerous: it makes up an answer, fluently, confidently, in a voice that sounds exactly right. We had calls where the agent described an email so convincingly that we went and checked our own inbox, only to find the email never existed. It hadn't lied on purpose. It had simply filled the silence with something plausible, the way it's built to.
Stopping that took a great deal of care, and a firm rule we will not bend: the agent does not get to claim something happened unless it actually happened. It does not announce a booking it didn't make, or read back a confirmation number it invented. And anything that changes the world, sending, booking, paying, goes through a deliberate, old-fashioned ritual: it tells you exactly what it's about to do, in plain words, and only does it once you say yes. A caller can't trick it into doing something else. A glitch can't make it fire twice. It's slower to build than "just let the model do it." It's the only version we'd put our name on.
And then it has to survive a Tuesday at 2am
The last stretch of this work is the part you only discover on a real call, with a real person, at the worst possible moment.
The connection drops mid-sentence, and the agent has to reconnect and continue its thought rather than start the whole answer over, while somehow knowing whether it had already finished. A caller goes quiet, and instead of dropping them cold the agent has to gently check in, wait, reassure, and only then say a warm goodbye it never clips off mid-word, while a single word from the caller pulls the whole conversation right back. Someone calls with a heavy accent on a terrible line. Someone calls from a different timezone, and the agent had better know it's a different day where they're standing. Someone, inevitably, tries to talk it into misbehaving, and it has to shrug that off without losing its warmth.
We could keep going. Honestly, somewhere past fifty distinct mechanisms we stopped keeping a tidy count, and every one of them is something we had to build around the brain, not inside it. The remarkable, slightly humbling thing is that not one of them is the model. Swap in a smarter model tomorrow, and every single one of these problems is still sitting there, waiting, exactly as hard as it was the day before.
The point
A great model makes a great voice agent possible. It does not make one exist.
The model is the brain. Everything we've described, and a great deal we haven't, is the nervous system, the ears, the reflexes, the hands, and the hard-won judgment to use them. That isn't a thin wrapper around an AI. It's the actual product, built one buzzing hum and one almost-clipped goodbye at a time, over a very long stretch of caring about details nobody is supposed to notice. That's the job. When we've done it right, you forget there was any work at all.
This is the part you can't buy off a shelf or bolt on over a weekend. It's the part we've poured ourselves into. And it's the reason that, if you want a voice agent that sounds human, answers in real time, and actually does the work, you don't have to live through all of it yourself. That's what we make. It's called Talk To My Agent.
Give it a call sometime. We think you'll forget you're talking to software. That's the entire point.