Dialog-RSN-1 API

Connect to the Dialog-RSN-1 real-time voice model over WebSocket: endpoints, authentication, the session contract, events, PolyAI extensions and error codes.

Dialog-RSN-1 is a real-time voice model. You stream the caller’s audio over one WebSocket and the model decides when the caller has finished speaking. The API speaks the reply and streams its transcript, or sends the reply as text for your own text-to-speech.

The wire format is the OpenAI Realtime API (GA schema). A stock OpenAI Realtime client works by changing the base URL and the key. The model name on the wire is dialog-rsn-1.

Limits and pricing

Concurrency, session duration, context window, audio formats and the connected-minute rate.

Dialog-RSN-1 docs

The overview, quickstart, prompt guide, cookbook and integration guides.

OpenAI Realtime reference

Field-level documentation for every event. This page lists what we accept and where we differ.

Connect

Connect to wss://api.studio.poly.ai/v1/realtime with an API key from studio.poly.ai. Sign up or log in there and create the key in the API Keys tab. The quickstart has the steps.

Enterprise accounts on studio.us.poly.ai connect to wss://api.us.poly.ai/v1/realtime instead, with a key from that Studio. A key only works on the host that matches the Studio that issued it, and a mismatch returns 401. Dialog-RSN-1 runs in the US only.

Check the service is up with the health route, which takes the same key:

curl -s -H "X-API-KEY: $POLY_API_KEY" https://api.studio.poly.ai/v1/realtime/health

Open a connection and read the first event. The OpenAI SDK appends /realtime to the base URL and sends your key as a bearer token.

Python

import asyncio
import os
from openai import AsyncOpenAI

client = AsyncOpenAI(
    api_key=os.environ["POLY_API_KEY"],
    websocket_base_url="wss://api.studio.poly.ai/v1",
)

async def main():
    async with client.realtime.connect(model="dialog-rsn-1") as conn:
        async for event in conn:
            print(event.type)  # session.created
            break

asyncio.run(main())

The optional query parameter session_id ([A-Za-z0-9_.-], up to 128 characters) names the session for your own logs. It comes back as session.poly_session_id. The model query parameter that OpenAI clients send is accepted and ignored.

The server pings every 20 seconds, which keeps a quiet connection open through the gateway. A connection whose client sends no event for 5 minutes is closed with code 4408; a client that is streaming audio never meets this.

Authenticate

Send your API key from studio.poly.ai in either header. They are equivalent.

X-API-KEY: <key>
Authorization: Bearer <key>

There is no query-string key and no WebSocket subprotocol credential. Browsers cannot set either header on a WebSocket, so put a relay in front of browser clients.

A refused handshake answers with an HTTP status and a JSON body. The close codes below are sent only by a server without the websocket.http.response extension, which can’t answer the handshake with a status. A client library that can’t read the HTTP response sees a failed handshake, not a close code.

HTTP	Close code	Meaning	What to do
401	4401	Missing, unknown or inactive key	Check the key, or create a new one in the API Keys tab at studio.poly.ai
402	4402	The account is not enrolled for billing, or has no balance left	Add funds at studio.poly.ai, then reconnect. A session already open is not cut off when the balance reaches zero.
403	4403	A personal access token	Use an API key from the API Keys tab at studio.poly.ai
429		Too many new connections this minute for this key, refused at the gateway	Honour Retry-After
429	4429	The account is at its concurrent-session cap (body "code": "session_limit_reached")	There is no Retry-After. Close a session before opening another. See Limits.
503	4503	Authentication service unavailable	Retry with backoff. The key is fine.
503	1013	No capacity for a new session on this server (body "code": "capacity_exceeded")	Retry after Retry-After (1 second). The retry very likely lands on another server.

Configure the session

Send session.update after session.created. Updates are merged into the session, not replaced. A rejected update applies nothing and sends no session.updated, so what comes back in session.updated is exactly what is in effect.

session.update

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You work the front desk of a hotel. Keep replies to one sentence.",
    "output_modalities": ["text"],
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "turn_detection": { "type": "server_vad", "create_response": true }
      }
    },
    "tools": []
  }
}

Honoured

Field	Notes
instructions	The system prompt
tools	Function tools only. Mark at most one string parameter per tool with "poly_spoken": true and the model says that argument to the caller while the tool runs: as response.output_text.delta on a text session, for your text-to-speech to speak, or aloud on an audio session.
tool_choice	"auto" only
max_output_tokens	An integer or "inf". The deployment’s own cap is the ceiling. The model reasons before it answers and reasoning tokens come out of the same budget, so a small cap can leave a truncated or empty reply.
audio.input.format	See Audio in
audio.input.turn_detection.create_response	true (default): the model confirms the end of each turn and the server generates a reply. false: you send response.create, and turns are taken at face value with no model confirmation. The voice-activity detector still commits the turn at each pause it hears, and input_audio_buffer.commit commits at once.
output_modalities	["audio"], the default for a new session, has the API speak the reply: audio arrives as response.output_audio.delta and the text as response.output_audio_transcript.delta. ["text"] sends text replies you voice yourself, as response.output_text.delta.
audio.output.voice	The voice for spoken replies: a Cartesia voice id, as a string or {"id": "..."}. Unset, the deployment’s default voice speaks. An id Cartesia doesn’t know is accepted here, and each response then fails with audio_synthesis_failed. Kept on a text session, and used if the session switches to ["audio"].
poly_knowledge, poly_response_language, poly_input_rate	See PolyAI extensions

Accepted and ignored

Fields that stock clients send by default and that would change nothing a caller notices are accepted. Most are left out of session.updated; audio.input.transcription and parallel_tool_calls are echoed back but change nothing:

• model (the session stays dialog-rsn-1), tracing, truncation, include
• audio.input.noise_reduction; audio.input.transcription (transcripts are always sent)
• Turn-detection timing: threshold, silence_duration_ms, prefix_padding_ms, eagerness. type: "semantic_vad" is reported back as server_vad. The server finds the end of the turn: a voice-activity detector proposes a boundary and the model confirms it.
• turn_detection.interrupt_response: true, which is already how the service behaves
• audio.output.voice set to an OpenAI voice name (alloy, marin and the rest): the deployment’s default voice speaks instead. audio.output.speed.
• parallel_tool_calls (the model makes one call per turn)

Refused

Fields that would change what you get if silently ignored are refused with an error naming the field in param. The whole update is dropped.

Field	error.code	param
Any key not in the OpenAI schema and not prefixed poly_, at any depth	unknown_parameter	the path, e.g. session.audio.input.transcrption
prompt, reasoning	unsupported_parameter	prompt or reasoning
tool_choice other than "auto"	unsupported_parameter	tool_choice
tools[i].type other than "function" (no MCP)	unsupported_parameter	tools[i].type
audio.input.turn_detection: null (push-to-talk)	unsupported_parameter	audio.input.turn_detection
turn_detection.interrupt_response: false	unsupported_parameter	audio.input.turn_detection.interrupt_response
turn_detection.idle_timeout_ms	unsupported_parameter	audio.input.turn_detection.idle_timeout_ms
audio.output.format.type other than "audio/pcm"	unsupported_parameter	audio.output.format.type
output_modalities asking for text and audio together	invalid_request_error
max_output_tokens below 1	invalid_request_error	max_output_tokens
poly_session_id, expires_at (set by the server)	invalid_request_error	the field
poly_knowledge that isn’t a list of { "name", "content", "actions"? } with actions a string	invalid_request_error	poly_knowledge, or the entry, e.g. poly_knowledge[2]
poly_response_language that isn’t a string, is blank, or is over 64 characters	invalid_request_error	poly_response_language
poly_spoken set to anything but true, on a non-string property, or on more than one property of a tool	invalid_request_error	the path under tools[i].parameters.properties

Audio in

Audio travels as base64 inside input_audio_buffer.append. Binary WebSocket frames are refused.

audio.input.format.type	Rate	Notes
audio/pcm	24000 (default), 16000 (native, no resampling), or any other rate	PCM16, little-endian, mono
audio/pcmu	8 kHz, fixed	G.711 mu-law, as a telephony trunk or Twilio media stream sends it
audio/pcma	8 kHz, fixed	G.711 A-law

• Send about 20 ms per frame (640 bytes at 16 kHz, 960 at 24 kHz). One frame may not exceed 256 KiB; a larger frame closes the connection with code 1009.
• Keep streaming during silence. The server decides where the turn ends from the audio; a client that stops sending looks idle, not quiet.
• One turn may hold up to 300 seconds of audio; anything longer is dropped.
• Audio in conversation.item.create is refused with unsupported_item_type. Stream caller audio; use conversation.item.create for text, tool results and history.

Run a turn

With create_response: true the sequence for one caller turn is:

1. You stream input_audio_buffer.append frames.
2. input_audio_buffer.speech_started, then input_audio_buffer.speech_stopped and input_audio_buffer.committed when the model confirms the turn is over.
3. conversation.item.added and conversation.item.done for the caller’s turn.
4. response.created, response.output_item.added, response.content_part.added, then response.output_text.delta frames until response.output_text.done, response.content_part.done, response.output_item.done and response.done. conversation.item.input_audio_transcription.delta and .completed, carrying what the model heard, arrive during the reply, usually after its first text delta.

With create_response: false, the model doesn’t confirm the turn: step 2 follows the first pause the detector hears, or your input_audio_buffer.commit. Step 4 starts when you send response.create.

The samples below set ["text"] so they can print the reply. Leave it out and the API speaks the reply instead: read response.output_audio_transcript.delta for the text and response.output_audio.delta for the audio.

Python

"""Stream a WAV file to Dialog-RSN-1 and print the reply."""
import asyncio
import base64
import json
import os
import wave

import websockets

URL = "wss://api.studio.poly.ai/v1/realtime"
KEY = os.environ["POLY_API_KEY"]


async def main(path: str) -> None:
    with wave.open(path) as w:
        rate, pcm = w.getframerate(), w.readframes(w.getnframes())

    async with websockets.connect(URL, additional_headers={"X-API-KEY": KEY}) as ws:
        print((await recv(ws))["type"])  # session.created

        await ws.send(json.dumps({"type": "session.update", "session": {
            "instructions": "You work the front desk of a hotel. Keep replies to one sentence.",
            "output_modalities": ["text"],
            "audio": {"input": {"format": {"type": "audio/pcm", "rate": rate}}},
        }}))
        print((await recv(ws))["type"])  # session.updated

        # 20 ms of PCM16 mono per frame, as a microphone would deliver it.
        step = rate * 2 // 50
        for i in range(0, len(pcm), step):
            await ws.send(json.dumps({"type": "input_audio_buffer.append",
                                      "audio": base64.b64encode(pcm[i:i + step]).decode()}))
            await asyncio.sleep(0.02)
        # Keep sending silence: the server finds the end of the turn itself.
        silence = base64.b64encode(bytes(step)).decode()

        reply = ""
        while True:
            try:
                event = await asyncio.wait_for(recv(ws), 0.02)
            except asyncio.TimeoutError:
                await ws.send(json.dumps({"type": "input_audio_buffer.append", "audio": silence}))
                continue
            t = event["type"]
            if t == "response.output_text.delta":
                reply += event["delta"]
            elif t == "conversation.item.input_audio_transcription.completed":
                print("heard:", event["transcript"])
            elif t == "response.done":
                print("reply:", reply)
                break
            elif t == "error":
                print("error:", event["error"]["code"], event["error"]["message"])
                break


async def recv(ws):
    return json.loads(await ws.recv())


asyncio.run(main("booking_request.wav"))

Rules that catch people out:

• One response at a time. A response.create while a response is open is refused with conversation_already_has_active_response. Send response.cancel first. Two requests are held rather than refused: one sent while the caller is still speaking is answered by the response to their turn, and one sent while a reply the server speaks is still playing is answered when that audio ends. Only one request is held at a time.
• response.create may set instructions, max_output_tokens and metadata. parallel_tool_calls is ignored. Every other field must match the session or is refused with unsupported_parameter, naming response.<field>. That includes conversation: "none" (no out-of-band responses), a per-response output_modalities, tools or tool_choice, input, audio, prompt and reasoning. Change the session with session.update instead.
• A confirmed barge-in always cancels the reply. interrupt_response cannot be turned off. The cancelled response ends with status cancelled.
• Tell the server what the caller heard. If you speak the text through your own TTS, send the poly.output_audio.* events below so that an interrupted reply is kept in history as the words that were actually played.

Events

Every event is a JSON text frame with a type and an event_id. Fields follow the OpenAI Realtime reference; only the differences are listed here.

Client to server

Event	Notes
session.update	See Configure the session
input_audio_buffer.append	Base64 audio in the session’s format
input_audio_buffer.commit	An empty buffer gets input_audio_buffer_commit_empty
input_audio_buffer.clear
conversation.item.create	Messages, function_call and function_call_output items. Answered with conversation.item.added then .done.
conversation.item.retrieve	Returns the item. For a caller’s audio turn that is the transcript, not the audio itself.
conversation.item.delete
conversation.item.truncate	Cuts history only for a reply spoken by the server; otherwise validates the item and changes nothing
response.create
response.cancel	With nothing active, response_cancel_not_active
output_audio_buffer.clear	Always refused: WebRTC only. Send response.cancel.
poly.output_audio.started, .delta, .stopped	Playback reports, see below

Server to client

session.created, session.updated, conversation.item.added, .done, .deleted, .retrieved, .truncated, conversation.item.input_audio_transcription.delta, .completed, input_audio_buffer.speech_started, .speech_stopped, .committed, .cleared, response.created, response.done, response.output_item.added, .done, response.content_part.added, .done, response.output_text.delta, .done, response.function_call_arguments.delta, .done, error. On a session with output_modalities: ["audio"], response.output_audio_transcript.delta and .done replace response.output_text.*, and response.output_audio.delta and .done carry the speech as base64 PCM16 mono at 24 kHz.

Not sent: rate_limits.updated, conversation.created, MCP events. Token usage arrives on response.done as response.usage.

Differences a stock client notices

• Text and function-argument deltas arrive per model chunk, not per token. The concatenation is the same.
• Transcription deltas are true increments.
• previous_item_id is set on conversation.item.* events, not on response.output_item.*.
• speech_started.audio_start_ms counts audio actually received, not wall-clock time.
• session.created and response.created omit optional fields that are not in effect (usage, max_output_tokens, metadata and others).

PolyAI extensions

Extensions are additive fields on existing events, all prefixed poly_. Nothing is sent as a new server event type.

Where	Field	What it does
session	poly_knowledge	A list of { "name", "content", "actions"? } the model may cite in replies
session	poly_response_language	An ISO 639 tag (es, pt-BR) or words ("Singlish"). The model answers in that language and no other. null clears it.
session	poly_input_rate	Set for you when audio.input.format.rate is not 24000
session	poly_session_id	Read-only echo of the session_id query parameter
tools[].parameters.properties.<name>	poly_spoken: true	The model says this string argument to the caller while the tool runs: as response.output_text.delta on a text session, aloud on an audio session. One per tool, string parameters only.
conversation.item.create with function_call_output	poly_knowledge	Knowledge attached to one tool result
response.done.response	poly_cited_topics	Knowledge entries the reply cited; absent when none
response.done.response	poly_out_of_domain	true when the model judged the request outside its instructions
response.done.response	poly_compacted_item_ids	Items the prompt now leaves out, because older history was summarised or dropped to fit the context window. Sent on the first response after that changes. The items stay in history and can still be retrieved.
response.done.response	poly_probe_usage	The token cost of turn detection

Playback reports. When you voice the reply yourself, send these so the server knows what the caller heard:

{"type": "poly.output_audio.started", "response_id": "resp_123"}
{"type": "poly.output_audio.delta",   "response_id": "resp_123", "audio_ms": 1250, "delta": "Sure, what's"}
{"type": "poly.output_audio.stopped", "response_id": "resp_123", "audio_ms": 2100, "dropped_ms": 900, "spoken_text": "Sure, what's your"}

Send a delta every 250 ms or so with the cumulative milliseconds played and, in delta, the text spoken since the last report. The server builds what the caller has heard so far from those increments, so it knows where to cut the reply before stopped arrives. On a barge-in, stopped carries what was dropped and the words that were actually spoken, and the reply is kept in history as those words.

A client that sends no reports gets an estimate instead: the server treats a reply as audible for as long as it takes to say at 15 characters a second, and after a barge-in keeps the words it estimates were heard. On a reply the server speaks itself, it tracks playback and ignores these reports.

Errors

{
  "type": "error",
  "event_id": "event_9b71...",
  "error": {
    "type": "invalid_request_error",
    "code": "unknown_parameter",
    "param": "session.bogus_field",
    "event_id": "evt_2",
    "message": "Unknown parameter: 'session.bogus_field'. Extensions to this protocol must be namespaced 'poly_*'; anything else is rejected rather than silently dropped."
  }
}

Branch on code, never on message. error.event_id names the client frame the error is about, where there is one. type is invalid_request_error for something you sent and server_error for something that failed on our side. A server_error that ends a response also arrives on response.done with status failed. There, status_details.error carries the code in both code and type, for example generation_failed.

code	type	Retry?
unsupported_event_type	invalid_request_error	No. This event is not implemented.
invalid_request_error	invalid_request_error	No. A malformed frame, undecodable audio, a binary frame, or a bad session_id.
unknown_parameter	invalid_request_error	No. The path is in param.
unsupported_parameter	invalid_request_error	No. The field is real and this service does not honour it; path in param.
item_not_found	invalid_request_error	No
unsupported_item_type	invalid_request_error	No. An MCP item or audio in conversation.item.create.
input_audio_buffer_commit_empty	invalid_request_error	No. Nothing was lost.
conversation_already_has_active_response	invalid_request_error	Yes, after response.done, or send response.cancel first
conversation_item_limit_exceeded	invalid_request_error	Not as sent. The conversation already holds as many client-created items, or as many bytes of them, as the deployment allows. Delete some with conversation.item.delete first. A function_call_output answering the outstanding call is never refused.
response_cancel_not_active	invalid_request_error	No
session_expired	invalid_request_error	Reconnect now. The session reached its maximum duration; the socket closes with 4410 straight after.
context_length_exceeded	server_error	Not as sent. The conversation is too long for the context window; shorten instructions, tools or knowledge, or start a new session. The connection stays open.
capacity_exceeded	server_error	Yes, after a short backoff. No inference capacity for this conversation.
policy_violation	server_error	No
generation_failed	server_error	Yes
audio_synthesis_failed	server_error	Yes. A spoken reply’s audio could not be synthesised, so the response fails. Check audio.output.voice names a voice that exists.
server_error	server_error	Yes

Close codes

Code	Why	What to do
1008	Invalid session_id query parameter, after an error	Fix the id
1009	A frame over 256 KiB	Send smaller frames
1011	The session could not be started, after server_error	Retry
4408	No client event for 5 minutes, or the client stopped reading for 5 seconds	Reconnect when you next have audio to send
4410	The session reached its maximum duration, after session_expired. session.created.expires_at tells you when.	Reconnect now and carry on
1013	No capacity for a new session on this server, from a server that can’t answer the handshake with a status	Retry after a second
4401, 4402, 4403, 4429, 4503	A refused handshake, from a server that can’t answer it with an HTTP status	See Authenticate

Handling a refused handshake and a timed-out session:

Python

import json
import os

import websockets
from websockets.exceptions import ConnectionClosed, InvalidStatus

URL = "wss://api.studio.poly.ai/v1/realtime"
KEY = os.environ["POLY_API_KEY"]


async def run_session() -> None:
    try:
        async with websockets.connect(URL, additional_headers={"X-API-KEY": KEY}) as ws:
            async for frame in ws:
                event = json.loads(frame)
                if event["type"] == "error" and event["error"]["code"] == "session_expired":
                    break  # 4410 follows; open a new connection and continue the call
    except InvalidStatus as e:
        if e.response.status_code == 402:
            print("No balance: add funds at studio.poly.ai")  # body: {"detail": "..."}
        elif e.response.status_code == 429:
            # The gateway's rate limit sends Retry-After; the session cap doesn't, so close a session first.
            print("Rate limited or at the session cap; Retry-After:", e.response.headers.get("Retry-After"))
        else:
            raise
    except ConnectionClosed as e:
        if e.rcvd and e.rcvd.code == 4408:
            pass  # idle; reconnect when there is audio to send

Examples

There is no PolyAI SDK for this API; any OpenAI Realtime client works. Each guide has a public repo with its full example:

Repo	What it is	Guide
dialog-rsn-1-python-from-scratch	Voice agent on plain websockets, with tool calling and a live microphone	Python from scratch
dialog-rsn-1-javascript-browser	Browser microphone through a small key-holding relay	Browser with JavaScript
dialog-rsn-1-openai-migration	The same agent on the openai SDK, before and after the switch	OpenAI Realtime API
dialog-rsn-1-streaming-tts	Replies spoken through your own TTS as they stream, with barge-in	Streaming to your TTS
dialog-rsn-1-speech-to-text	Transcriber and support agent in Python and Node	Speech to text
dialog-rsn-1-voice-agent	FastAPI voice agent with Cartesia speech and a browser page	Browser voice agent with Cartesia
livekit-plugins-polyai and dialog-rsn-1-livekit	Dialog-RSN-1 as a LiveKit Agents realtime model, and a hotel agent built on it	LiveKit
dialog-rsn-1-realtime-agents	Multi-agent handoffs, a chat supervisor and a knowledge agent, in plain JavaScript