Build a LiveKit voice agent

Use Dialog-RSN-1 as the model in a LiveKit Agents session: it replaces the speech-to-text plugin, the turn detector and the LLM, and any LiveKit TTS plugin speaks the reply.

Build a voice agent on LiveKit Agents where one model hears the caller, decides when they’ve finished, and writes the reply. LiveKit carries the audio and a text-to-speech plugin speaks the reply. In about ten minutes you’ll be talking to it, in your terminal or in a LiveKit room.

Full code on GitHub

A hotel front desk agent with two tools, ElevenLabs speech, and a scripted caller that runs a whole call, barge-in included.

Building with a coding agent? The coding agent version of this guide gives you the prompts, what to check, and what to tell your agent when it goes wrong.

What you’ll build

1. A front desk agent for a fictional hotel with two tools: one looks up a booking, the other changes its checkout time.
2. An AgentSession that uses Dialog-RSN-1 as the model and ElevenLabs as the voice. Dialog-RSN-1 replaces the speech-to-text plugin, the turn detector and the LLM.
3. Barge-in. Talk over the agent and it stops, and the model’s history keeps only the words the caller heard.
4. A scripted caller that joins the room and runs a whole call, barge-in included, so you can test without a microphone.

What changes from a standard LiveKit pipeline

A standard LiveKit pipeline chains speech-to-text, an LLM and text-to-speech, with a turn detector deciding when the caller has finished. Dialog-RSN-1 takes the caller’s audio and handles every job except the voice:

Job	Standard LiveKit pipeline	With Dialog-RSN-1
Speech-to-text	An STT plugin, such as Deepgram or AssemblyAI	None. Dialog-RSN-1 hears the audio and sends the transcript
End of turn	VAD plus LiveKit’s turn detector model	Dialog-RSN-1 decides. You tune it through the instructions
Reply	An LLM, called with the transcript	Dialog-RSN-1 writes it from the audio it heard
Voice	A TTS plugin	Any TTS plugin LiveKit supports

Dialog-RSN-1 speaks the OpenAI Realtime protocol, so it plugs into LiveKit through LiveKit’s own OpenAI plugin: you point openai.realtime.RealtimeModel at Dialog-RSN-1’s URL and key. Settings that matter lists the few options to set or avoid.

Before you begin

1. A Dialog-RSN-1 API key. Sign up or log in at studio.poly.ai and create one in the API Keys tab. The quickstart walks through it.
2. An ElevenLabs API key. The free tier works.
3. Python 3.10 or later, and uv.
4. The LiveKit CLI: brew install livekit-cli.
5. For Step 6: a LiveKit Cloud project, or a local server (brew install livekit).

Step 1: Get the code

git clone https://github.com/PolyAI-LDN/dialog-rsn-1-livekit
cd dialog-rsn-1-livekit
uv sync

The repo has three files you’ll work with:

File	What it is
agent.py	The agent: instructions, two tools, and the AgentSession
hotel.py	The instructions and in-memory demo bookings
caller.py	A scripted caller for Step 6

Step 2: Configure your environment

cp .env.example .env

Open .env and set two keys:

POLY_API_KEY=YOUR_DIALOG_RSN_1_KEY
ELEVEN_API_KEY=YOUR_ELEVENLABS_KEY

The ElevenLabs plugin reads ELEVEN_API_KEY, not ELEVENLABS_API_KEY. Leave the LIVEKIT_* values alone until Step 6.

Step 3: Point the session at Dialog-RSN-1

agent.py builds the whole session in a few lines:

session = AgentSession(
    # Dialog-RSN-1 hears the caller, decides when they've finished, and streams the reply
    # as text. It replaces the STT, the LLM and the turn detector.
    llm=openai.realtime.RealtimeModel(
        base_url=os.environ.get("DIALOG_REALTIME_URL", "wss://api.studio.poly.ai/v1"),
        api_key=os.environ["POLY_API_KEY"],
        model="dialog-rsn-1",
        # Text out, so the TTS below voices each reply as it streams.
        modalities=["text"],
    ),
    tts=elevenlabs.TTS(voice_id="EXAVITQu4vr4xnSDxMaL", model="eleven_flash_v2_5"),
)

await session.start(agent=FrontDesk(), room=ctx.room)
await session.generate_reply(instructions="Greet the caller.")

Compared with the LiveKit starter template, the session leaves out two things:

• No stt. Dialog-RSN-1 takes the room’s audio as it arrives. You still see the caller’s transcript in LiveKit, because Dialog-RSN-1 sends it with each reply. It comes at the end of a completed reply, so a turn whose reply is cancelled before it finishes generating has no transcript.
• No turn_handling. You have no turn detector to configure. Dialog-RSN-1 judges each pause in the context of the conversation. To make it more or less patient, say so in the instructions. The prompt guide shows how.

The model connects to wss://api.studio.poly.ai/v1/realtime; the plugin adds /realtime to base_url. For a key from a regional Studio, set DIALOG_REALTIME_URL, for example wss://api.us.poly.ai/v1. modalities=["text"] asks Dialog-RSN-1 for text, and the TTS plugin speaks each reply as the text streams in. Leave it out and Dialog-RSN-1 speaks in its own voice instead, and LiveKit skips your TTS.

The two tools are ordinary LiveKit function tools:

@function_tool
async def look_up_reservation(
    self, context: RunContext, confirmation_code: str = "", last_name: str = ""
) -> str:
    """Find a reservation by confirmation code or by the guest's last name.

    Args:
        confirmation_code: Confirmation code, for example GM40912.
        last_name: The guest's last name.
    """
    return json.dumps(look_up_reservation(confirmation_code, last_name))

Each tool returns its result as a JSON string, which gives the model structured data to read. Dialog-RSN-1 accepts function tools only, so don’t add OpenAI’s provider tools such as web search.

Step 4: Run the agent

lk agent console agent.py

The console runs the agent on your machine and connects it to your microphone and speakers. You don’t need a LiveKit server for this step. Wear headphones so the agent doesn’t hear itself.

To type instead of talk, add --text. Text mode skips TTS and runs only Dialog-RSN-1 and the tools:

lk agent console agent.py --text

Step 5: Test the conversation

The agent greets you with “Hello, Grand Meridian Hotel. How can I help you today?” Then try four things:

1. Say “Can you look up booking GM40912?” The agent calls look_up_reservation and reads back the booking.
2. Say “Could I check out at two instead?” The agent tells you whether there’s a fee and asks you to confirm before it calls change_checkout_time.
3. Ask something with a long answer, then talk over the reply. The agent stops and answers your new question.
4. Pause mid-sentence, for example while you read out the booking code. The agent should wait for you to finish.

If the agent doesn’t start, see Troubleshooting.

Step 6: Run it in a LiveKit room

To use LiveKit Cloud, put your project’s URL, API key and secret in .env. To stay on your machine, start a local server and keep the defaults from .env.example:

livekit-server --dev

Start the agent. It joins every new room on the server:

uv run agent.py dev

On LiveKit Cloud, you can talk to the agent from your project’s Agent Console. On a local server, or without a microphone, run the scripted caller in a second terminal:

uv run caller.py

The caller joins a new room and plays five recorded lines, each one after the agent finishes its reply. The third line, “Sorry, stop, stop”, plays 1.5 seconds into a reply instead. The first three exchanges look like this:

[  7.4s] agent : Hello, Grand Meridian Hotel. How can I help you today?
[ 14.8s] caller: Hi, can you look up booking G M four zero nine one two please?
[ 22.0s] agent : I've found your booking, Amara. You're currently scheduled to check out at 11:00 on September 30th. What would you like to change?
[ 31.6s] caller: Can you tell me everything about my booking and describe the hotel for me in detail?
[ 34.3s] agent : I can tell you about your booking - you have a Deluxe King room, checking out
[ 39.4s] caller: Sorry, stop, stop. Actually, what time is checkout?
[ 44.0s] agent : Your current checkout time is 11:00 AM on September 30th. Would you like to change that?

The agent stops at “checking out”, where the caller cut in, and Dialog-RSN-1’s history stops there too. When the caller interrupts, LiveKit deletes the full reply from the server and adds back only the words the caller heard. The model builds its next reply on that shorter version.

When the call ends, the caller saves the agent’s side to call.wav so you can listen back.

Add Dialog-RSN-1 to an existing LiveKit agent

1. Install LiveKit’s OpenAI plugin:

   uv add "livekit-agents[openai]~=1.8.3"
   

2. Import it: from livekit.plugins import openai.

3. Add POLY_API_KEY to your environment.

4. In your AgentSession, set llm to openai.realtime.RealtimeModel(base_url="wss://api.studio.poly.ai/v1", api_key=os.environ["POLY_API_KEY"], model="dialog-rsn-1", modalities=["text"]). Remove stt and turn_handling, and keep your tts.

Settings that matter

LiveKit’s OpenAI plugin sends some OpenAI settings Dialog-RSN-1 doesn’t use, such as a voice, semantic VAD tuning and a transcription model. Dialog-RSN-1 accepts and ignores them, so the defaults work. A few options do matter:

Option	What to do
modalities	Set ["text"] to voice replies with your TTS plugin. The default, ["text", "audio"], has Dialog-RSN-1 speak them
turn_detection	Leave it at the default. Dialog-RSN-1 runs turn detection on the server whatever the VAD type, and refuses interrupt_response=False
AgentSession(turn_handling=...), vad, stt	Leave them out, so Dialog-RSN-1 decides when the caller has finished
tool_choice	Leave it at auto. Dialog-RSN-1 refuses any other value

A refused session.update comes back as an error event, and LiveKit logs it.

Dialog-RSN-1’s own fields

Fields such as poly_response_language and poly_knowledge aren’t plugin options. Send them on a raw session.update, and read poly_cited_topics from response.done in the raw server events:

rt = session.current_agent.realtime_llm_session  # the OpenAI plugin's RealtimeSession

def send_poly_settings() -> None:
    rt.send_event({"type": "session.update", "session": {
        "type": "realtime",
        "poly_response_language": "es",
        "poly_knowledge": [{"name": "spa-hours", "content": "The spa is open 7am to 9pm."}],
    }})

send_poly_settings()
# The plugin replays only its own settings after a reconnect, so send these again.
rt.on("session_reconnected", lambda _: send_poly_settings())
rt.on("openai_server_event_received",
      lambda ev: ev["type"] == "response.done" and print(ev["response"].get("poly_cited_topics")))

Later session.updates from the plugin don’t clear these fields. To mark the argument a tool speaks while it runs with poly_spoken, declare that tool with @function_tool(raw_schema=...), which the plugin sends unchanged.

Choose a different voice

To change the ElevenLabs voice, pass a different voice_id. The repo uses Sarah, which works on the free tier. Some library voices, such as Rachel, don’t.

To switch provider, swap the plugin. For Cartesia, install its plugin:

uv add "livekit-agents[cartesia]~=1.8.3"

Add CARTESIA_API_KEY to .env, then change the tts line:

from livekit.plugins import cartesia

tts=cartesia.TTS(),

Nothing else in agent.py changes. LiveKit’s TTS models page lists every provider it supports.

Troubleshooting

Symptom	Likely cause	Fix
KeyError: 'POLY_API_KEY'	The Dialog-RSN-1 key is missing	Copy .env.example to .env and set POLY_API_KEY
OpenAI Realtime API connection failed after 3 attempts (… 401 …)	The Dialog-RSN-1 key is wrong, or it’s from a regional Studio such as studio.us.poly.ai	Create a key in the API Keys tab at studio.poly.ai and set it as POLY_API_KEY in .env. For a studio.us.poly.ai key, set DIALOG_REALTIME_URL=wss://api.us.poly.ai/v1
OpenAI Realtime API connection failed after 3 attempts (… 403 …)	It’s a personal access token	Use an API key from the API Keys tab
The agent speaks in a voice you didn’t choose	modalities is missing, so Dialog-RSN-1 sends audio and LiveKit skips your TTS	Pass modalities=["text"] to openai.realtime.RealtimeModel
ElevenLabs API key is required	The key is under the wrong name	Rename it to ELEVEN_API_KEY
ws_url is required, or set LIVEKIT_URL	agent.py dev has no LiveKit server to join	Set the LIVEKIT_* values in .env, or use lk agent console
caller.py prints “no agent spoke within 30 seconds”	The agent isn’t running, or it’s on a different server	Start uv run agent.py dev with the same LIVEKIT_URL as the caller
The agent keeps interrupting itself	The microphone picks up the agent’s voice	Wear headphones

What doesn’t carry over

• Turn-handling options. LiveKit’s turn detector, endpointing delays and preemptive generation don’t apply, because Dialog-RSN-1 decides when the turn ends.
• STT features. With no STT plugin, options such as keyterm prompting have nothing to configure. Dialog-RSN-1 transcribes the caller as part of each turn.
• Playback reports. LiveKit doesn’t send Dialog-RSN-1’s poly.output_audio.* events, so Dialog-RSN-1 estimates what was heard. The streaming text-to-speech guide explains what they add.

Where to go next

• The OpenAI Realtime migration guide lists every field Dialog-RSN-1 accepts and rejects.
• The prompt guide covers turn-taking instructions in depth.
• The API reference lists every client and server event.
• LiveKit’s Agents docs cover deployment, telephony and the rest of the framework.