Voice AI is having its moment. OpenAI shipped Realtime. ElevenLabs raised $11B. Hume, Deepgram, Cartesia - everyone is building pieces of the voice puzzle. And yet, if you've actually tried to have a real conversation with any voice AI in production today, you know: it still feels like talking to a machine.
We've spent the last year building Salesteq, an autonomous AI platform for commercial operations. When we started adding voice, we realized why nobody has cracked it yet. The problem isn't any single component. The problem is architecture.
The Two Camps (And Why Both Are Wrong)
Today's voice AI splits into two camps:
Camp 1: Speech-to-Speech (S2S). OpenAI Realtime, Hume EVI, Moshi. Audio goes in, audio comes out. No text in the middle. This preserves emotion and prosody. The AI "hears" your frustration, not just your words. Conversations feel natural. Latency is low (~200-300ms).
The problem: these models are inherently unstable. They're integrated black boxes. You can't test them against specific business requirements - information correctness, safety, cultural compliance. Ask them to look up an order, query a database, or process a return, and they fall apart. S2S models are great at talking. They're terrible at acting. And when they get something wrong, you can't diagnose why.
Camp 2: Cascaded pipelines. Vapi, Retell, and most production deployments. Speech-to-text, then LLM, then text-to-speech. Three separate systems chained together. This is reliable. The LLM in the middle can call tools, query databases, execute business logic. You can test each stage independently.
The problem: every boundary crossing destroys information. The ASR strips emotion from the audio. The LLM processes cold text. The TTS generates flat speech. The result: a robotic voice that technically answers your question but makes you want to hang up. Latency stacks up too. 500-800ms is typical, which feels noticeably slow.
Neither camp alone can build a voice AI that feels like talking to a person AND reliably takes action.
The Industry Is Chasing the Wrong Thing
There's a deeper problem. The voice AI industry is obsessed with state-of-the-art benchmarks - lowest latency, highest MOS score, most natural-sounding voice - while ignoring what actually matters in production: reliability, testability, and domain accuracy.
Voice systems have a hard time recognizing local brands, city names, and street names. Models aren't updated in real time. They have long training cycles, so they struggle with recent trending words and new products. Many languages are extremely variable in terms of accents; Arabic alone has dozens of distinct dialects. Only locally-trained models can compete in regional markets.
And here's the testing problem nobody talks about: integrated end-to-end systems are nearly impossible to test properly. Instead of a single metric, you need to verify accuracy, latency, robustness, safety, and cultural appropriateness simultaneously. Direct comparison of competing solutions is rarely possible and requires deep research. Everyone publishes demo videos. Nobody publishes production failure rates.
We think this is exactly where smaller, focused companies win. The giants (OpenAI, Google) are too focused on AGI. The startups are too focused on wrapping their APIs. The gap is in the middle: purpose-built systems that prioritize reliability over benchmarks.
Kol: A Hybrid Voice Orchestrator
We're building Kol, a voice-native hybrid orchestrator that operates in two modes simultaneously and switches between them mid-conversation, transparently.
Our approach draws on a concept that's well-established in ML but under-explored in voice: heterogeneous mixture of experts. While typical MoE architectures use uniform experts (20 similar sub-networks), we extend this to use fundamentally different technologies as experts. Different architectures, different model types, even different processing paradigms. Each expert is chosen for its strengths: a duplex model for natural conversation, a cascaded pipeline for reliable tool execution, specialized models for emotion detection, domain-specific ASR for local vocabulary.
This gives us something that monolithic systems can't achieve: modular reliability. Each component can be tested, replaced, and fine-tuned independently. When the ASR misrecognizes a brand name, we fix the ASR. When the TTS sounds flat, we swap the TTS. We don't retrain the entire system and hope nothing else broke.
In practice, this means two modes:
Voice Mode handles the human side. When a customer is talking, explaining a problem, or expressing frustration, Kol is in Voice Mode. Duplex speech-to-speech architecture. Prosody is preserved end-to-end. The AI hears emotion in the audio signal, not from sentiment analysis on a transcript. It backchannels ("mm-hmm", "I see") while the customer is still speaking. It handles barge-in. If the customer interrupts, Kol stops instantly and pivots.
Action Mode handles the execution side. When Kol needs to actually do something (look up an order, check inventory, process a return, book an appointment) it switches to a cascaded pipeline backed by our full tool set. 55+ tools, a 7M company database, email integration, CRM pipeline, product knowledge base. Reliable, observable, auditable.
The switch happens mid-conversation. The customer never notices.
Here's what that looks like in practice:
Customer: "I'm so frustrated, I've been waiting three weeks for my delivery and nobody-"
Kol (Voice Mode): "mm-hmm"
Customer: "-and nobody can tell me where it is!"
Kol (empathetic tone): "I completely understand your frustration. Let me find your order right now."
Kol switches to Action Mode, queries order system, finds shipment status
Kol (confident tone): "I found it. Your replacement shipped this morning. It'll arrive Thursday. I'm sorry about the wait."
The customer heard empathy, got a real answer, and the whole interaction took seconds. No hold music. No "let me transfer you." No robot.
The Hard Constraint: 400 Milliseconds
We have one non-negotiable engineering constraint: < 400ms end-to-end latency. From the moment a customer finishes speaking to the first audio of Kol's response.
This number isn't arbitrary. It's the boundary between "talking to a person" and "talking to a machine." Above 500ms, humans perceive a delay. Above 800ms, it feels broken. Below 400ms, it feels like a natural conversation.
Our latency budget:
| Stage | Target | |---|---| | Voice activity detection + semantic turn detection | < 60ms | | Speech recognition (streaming) | < 100ms | | LLM first token (streaming) | < 150ms | | Text-to-speech first audio chunk (streaming) | < 80ms | | Total | < 400ms |
The key word is streaming. Nothing waits for the previous stage to finish. ASR streams partial transcripts into the LLM. The LLM streams tokens into TTS. TTS streams audio chunks to the client. The first audio plays before the AI has finished generating its full response.
We also use speculative execution. We start generating a response before the turn detector is 100% confident the customer is done speaking. If they keep talking, we cancel and restart. This trades compute for perceived latency and shaves another 150-250ms off the experience.
Why Semantic Turn Detection Matters More Than You Think
The most underrated problem in voice AI is knowing when someone is done talking.
Traditional systems use Voice Activity Detection (VAD). They listen for silence. The moment you pause for 300ms, they assume you're done and start responding. This creates the most annoying behavior in all of voice AI: the system cuts you off mid-thought because you paused to think.
Semantic turn detection solves this. Instead of just listening for silence, a neural model analyzes the transcript and determines whether the utterance is semantically complete. "I wanted to ask about-" is clearly incomplete. "What's the status of my order?" is clearly complete. The silence threshold adjusts dynamically based on semantic signals.
The result: Kol never cuts you off when you're pausing to think. But it responds instantly when you're actually done. This single capability eliminates the most common complaint about every voice AI in production today.
Why We Build Our Own Models
Every voice AI startup in 2026 wraps the same third-party APIs: OpenAI for reasoning, Deepgram or Whisper for ASR, ElevenLabs or Cartesia for TTS. This works fine for English demos. It breaks down in production for three reasons:
1. Localization. Large general-purpose ASR models fail on local brand names, street names, product SKUs, and accents. Arabic alone has dozens of distinct dialects. German compound words break tokenizers. A model trained on American English stumbles on Swiss German. Only purpose-built, locally-trained models perform well enough for production customer service.
2. Control. When your voice pipeline is a chain of third-party APIs, you can't diagnose issues, guarantee latency, or tune for specific verticals. We need to know exactly why the ASR misheard "BMW 320i" as "BMW 3201." With our own models running on our own infrastructure, we can trace every failure to its source and fix it. Without waiting for a vendor's next model release.
3. Naturalness. Cascaded pipelines lose prosody at every boundary. The ASR transcribes "I'm so frustrated" as flat text. The TTS reads it back in a neutral voice. The emotional context, which was right there in the original audio, is gone. Speech-to-speech models preserve it, but can't do tool calling. Our hybrid architecture does both.
Our voice technology is led by Dr. Nickolay Shmyrev, who has spent over 20 years building speech recognition systems. He created recognition engines with millions of downloads worldwide, led voice research at Nexiwave, and serves as Chief Scientist at EnglishCentral, a language learning platform where voice recognition accuracy is the product, not a feature. His background spans from formal verification (PhD, Russian Academy of Sciences) to production speech systems at scale. He brings the rare combination of deep ML research and engineering pragmatism that this problem demands.
We're not wrapping APIs. We're building the voice intelligence layer from the ground up, starting with open-source foundations (streaming ASR via sherpa-onnx, Kokoro for TTS, Silero for VAD) and fine-tuning for our specific use cases. Our prototype is already running: WebRTC audio in, streaming STT and TTS as separate gRPC microservices, sentence-boundary splitting for real-time synthesis, full conversation loop. The architecture is live. Now we're optimizing.
Five Moats, Not One
Voice AI companies tend to pick one thing and hope it's enough. A better TTS voice. Lower latency. A novel turn detection model. The problem is that each of these is individually reproducible. Any well-funded team can match a single technical advantage within 12-18 months.
Our bet is on the combination. Five things that each take years to build, and together create a system that nobody can replicate by doing just one of them well.
1. The hybrid architecture itself. Two fundamentally different voice processing modes in one system, switching mid-conversation based on what the situation needs. This isn't a feature you bolt on. It's a ground-up architectural decision that affects every component: the orchestrator, the state machine, the audio routing, the context management. Every competitor has committed to either S2S or cascaded. Switching architectures after the fact is a rewrite, not an upgrade.
2. Business integration depth. Kol doesn't exist in isolation. It sits on top of Badger, our autonomous agent with 55+ tools connected to live production systems: a 7M company database, CRM pipelines, email, calendar, product knowledge bases, scheduled automation. When Kol needs to "look up your order," it actually looks up the order. This is not a voice company bolting on a few API integrations. This is a full commercial operations platform that added voice. The depth of business logic underneath the voice layer is something a standalone voice startup would need years to build.
3. Owned models on owned infrastructure. Every component runs on our machines. We can fine-tune ASR per customer, per vertical, per dialect. We can swap TTS voices per brand. We can adjust latency budgets per use case. We don't pay per-minute API fees that make unit economics impossible at scale. And we're not one vendor deprecation away from rebuilding our pipeline. When OpenAI changes their Realtime API pricing or Google discontinues a model, our system keeps running.
4. Regional language depth. Not just Arabic, though Arabic is where the advantage is most visible. The modular architecture means we can deeply tune each language module independently. Gulf Arabic ASR trained on real customer service recordings from automotive dealerships. Swiss German that handles dialect-switching. Spanish that distinguishes Mexican from Castilian. Every language we add compounds the moat because the training data, dialect expertise, and vertical tuning are specific, hard-won, and non-transferable. API wrappers get whatever the upstream model gives them. We build what each market actually needs.
5. Two decades of speech expertise. Our voice team is led by someone who has been building speech recognition systems since before deep learning existed. He's built engines that ship to millions of users, not research demos that work in controlled environments. This kind of experience means knowing which corners can be cut and which can't, which open-source models are production-ready and which will break under load, and where the real latency bottlenecks hide. You can't hire this. It takes 20 years to develop.
The moat isn't any one of these. It's all five together. A competitor can match our TTS quality or our ASR latency. Nobody can simultaneously match the hybrid architecture, the business integration, the owned infrastructure, the regional language depth, and the team. That's the bet.
What This Means for Businesses
The end state isn't "a better chatbot with voice." It's this:
A customer visits your website. They click the chat widget and start talking. An AI answers, in their language, in their accent, instantly. They explain a long, complex problem. The AI murmurs "mm-hmm" while they're speaking. It hears the frustration in their voice, not from a keyword, from the audio signal itself. It adjusts its tone to be empathetic. Then it takes action: looks up the order, finds the status, processes the return. It tells the customer what happened, in a confident voice. The whole thing takes 30 seconds.
The customer hangs up. They don't know they were talking to an AI. They just know it was the best customer service they've ever had.
That's Kol. That's what we're building at Salesteq.
Dr. Nickolay Shmyrev is Co-founder and Voice Product Lead at Salesteq. He has spent over 20 years building speech recognition systems, serves as Chief Scientist at EnglishCentral, and previously led voice research at Nexiwave. Kol is Salesteq's voice intelligence engine. We're currently in private beta. Reach out at [email protected].