What LiveKit Agents Actually Is
LiveKit started life as open-source WebRTC infrastructure. It handles the unglamorous part of real-time audio and video: signalling, codecs, jitter buffers, and getting a stream from a browser in Lagos to a server in Frankfurt without it sounding like a walkie-talkie. LiveKit Agents is the layer above that. It’s a framework for running AI agents inside a LiveKit room as full participants, meaning an agent can subscribe to a caller’s microphone, publish its own voice back, watch a video feed, and respond in a fraction of a second.
It ships in Python and Node.js. Python gets the deeper plugin ecosystem, but the TypeScript version covers the same core concepts if your stack lives in Node.
The mental model that matters: an agent isn’t a webhook or a chatbot endpoint. It’s a peer in the room, with a session, a state machine for listening and speaking, and a lifecycle tied to whoever it’s talking to.
Latency Is the Entire Product
Voice AI has a crueller usability threshold than almost any other software category. Text chats can take four seconds to answer and nobody minds. A voice agent that pauses for 1.2 seconds feels broken, and users start saying “hello? are you there?” which derails the transcription and the conversation along with it.
Roughly: under 500ms reads as natural. Around 800ms people begin talking over the agent. Past a second and a half, they assume the line dropped.
A naive loop, where you record a clip, POST it to a transcription API, wait for the full transcript, send it to an LLM, wait for the complete answer, then hand that to a TTS endpoint, usually lands between two and three seconds. LiveKit Agents gets under the threshold by streaming every stage and overlapping them:
- Partial transcripts flow to the LLM while the caller is still speaking, so the model is often ready before they stop.
- TTS starts synthesising on the first sentence chunk rather than the full reply.
- Audio moves over WebRTC via UDP, avoiding the retransmission stalls you get with TCP.
None of that is exotic. Doing all of it at once, reliably, is the hard part.
Inside the Voice Pipeline
Most deployments run a three-stage pipeline: speech-to-text, language model, text-to-speech. Each stage is a swappable plugin.
Speech-to-text
Plugins exist for Deepgram, AssemblyAI, Whisper, Azure, Google and others. Streaming matters more than raw accuracy here, because a model with brilliant final transcripts and a two-second finalisation delay loses to a slightly less accurate model that emits partials in 150 milliseconds.
Turn detection, where most projects quietly fail
Deciding that a human has finished talking is not the same as detecting silence. People pause mid-sentence to think, say “um” while they search for a word, and trail off expecting you to fill the gap. A pure silence timer either cuts them off after 400ms or makes the agent feel sluggish with an 1,800ms wait.
LiveKit’s approach layers a few signals. Silero VAD detects raw voice activity. A small transformer-based turn detector, published as an open-weight model you can run locally on CPU, judges whether an utterance sounds semantically complete. The agent session then decides whether to respond, keep listening, or treat incoming speech as an interruption.
That last part is barge-in. If the caller starts talking while the agent is mid-answer, the session cancels the TTS stream and the LLM generation, then pivots. Getting this right is the difference between a demo and something a customer tolerates for ten minutes.
The language model
Any function-calling model works: OpenAI, Anthropic, Google, Groq, Mistral, or something you host. You’re optimising for time to first token, because that’s what the caller hears. A 70B model that starts answering in 300ms beats a 400B model that thinks for two seconds.
Text-to-speech
Cartesia, ElevenLabs, Rime, PlayHT and Azure all have plugins. The metric that matters is time to first byte, not total synthesis speed, since most providers stream audio as they generate it. 24kHz mono Opus is a reasonable default for voice calls.
Speech-to-Speech, Skipping the Pipeline
Realtime APIs from OpenAI and Google accept audio and emit audio directly, collapsing three models into one. LiveKit Agents wraps these as realtime models inside the same session abstraction, so you can swap a pipeline for a speech-to-speech model without rewriting your agent’s logic.
The trade-off is control. Realtime models give you less visibility into the transcript, fewer knobs for voice identity, and pricing that scales with audio minutes rather than tokens. For a support bot where every word needs logging, a classic pipeline plus a good turn detector is often the safer build. For a latency-obsessed companion app, the loss of control may be worth it.
Tools, Memory, and MCP
An agent that can only talk is a novelty. The interesting work starts when it can look things up. In LiveKit Agents you define tools as ordinary functions, decorate them, and the framework exposes their schema to the model. Ask “when’s my next appointment?” and the agent calls your calendar function, gets JSON back, and speaks the result.
Model Context Protocol support means you can point an agent at an existing MCP server rather than writing bespoke integrations for every internal system, which is a real shortcut for teams already running MCP servers for developer tooling. Memory deserves a warning, though: stuffing a full transcript into every prompt balloons cost and latency over a long call. Summarise older turns, keep retrieval scoped, and cap what goes into context.
Real Phone Numbers
LiveKit has SIP integration built in, so agents can take inbound calls and place outbound ones through trunk providers like Twilio, Telnyx or regional carriers. From the agent’s perspective, a phone caller is just another participant in a room. That one detail removes a whole class of work, because you don’t maintain separate code paths for web and telephony.
Worth leaning on too: built-in noise cancellation. LiveKit Cloud offers background voice cancellation, and there’s Krisp-based noise filtering in the plugin set. An agent on a call centre floor with a barking dog nearby is a solved problem if you switch it on.
Running It: Workers, Dispatch and Scaling
Agents run as workers. You start a worker process, it registers with LiveKit, and jobs get dispatched when someone joins a room, either explicitly or through dispatch rules. Scaling is adding worker processes, same as any queue consumer. Under load you’ll want per-agent timeouts, a cap on concurrent sessions per process, and a plan for the worker that dies mid-call.
The CLI gives you hot reload in development and a production start command, which sounds trivial until you’ve tried iterating on prompt wording with a 20-second deploy cycle.
Testing and Observability
LiveKit Agents includes an evaluation harness that runs scripted conversations against your agent and scores them, often with an LLM as judge. This matters because voice bugs hide. A prompt change that fixes one caller’s edge case can wreck another, and you won’t spot it from logs alone.
Metrics to watch: end-of-utterance to agent-audio-out, interruption success rate, tool-call error rate, and how often the agent talks over the user. Traces export via OpenTelemetry, and LiveKit Cloud adds session recordings and dashboards if you’d rather not build them yourself.
Costs, and Where It Bites
LiveKit Cloud has a free tier and usage-based pricing from there, and self-hosting is genuinely viable since the core is open source. You’ll just own media server operations. The variable cost that surprises people is model spend. A three-minute call across STT, LLM and TTS can run anywhere from a few cents to twenty, depending on your vendors and how chatty the agent is. An agent that opens every reply with “Great question!” is paying for that filler three times over.
The Unsexy Work That Decides Whether It Ships
Tooling gets you to a working demo in an afternoon. What separates that from something you’d put in front of paying customers is a list nobody puts on a landing page: handling accents your test set missed, writing a barge-in policy for when the caller is wrong, deciding what the agent says when a tool times out, keeping PII out of logs, and building a loop where real calls become eval cases.
Pick boring defaults first. A pipeline you can inspect beats a speech-to-speech black box until you’ve proven the conversation design works. Instrument everything from day one, because “the agent feels weird sometimes” is not a bug report you can act on. And budget honest time for turn detection tuning, since a two-hundred-millisecond improvement there buys the caller more than a smarter model ever will.