Many sessions on one WebSocket
A multiplexed WebSocket carries many live speech or realtime transcription sessions at once. Each is a context with its own id, events, billing, and close code.
WS /v1/tts/multi and WS /v1/stt/multi each carry many sessions on one long-lived WebSocket. Each session is a context. A context is exactly one WS /v1/tts/live or WS /v1/stt/live connection, with the same frames, events, billing, and limits. Both endpoints are built for an application that speaks or listens on many calls at once. For one conversation at a time, use the single sockets.
| One socket per call | Multiplexed socket | |
|---|---|---|
| Endpoints | /v1/tts/live, /v1/stt/live | /v1/tts/multi, /v1/stt/multi |
| A new call | One upgrade | One start frame |
| JSON frames and events | As the live references list them | The same, each with context_id |
| Binary frames | Audio | A header naming the context, then audio |
| End of a call | The connection closes with a code | A closed event carries the code |
Open a socket and start contexts#
- ConnectOpen the WebSocket with the API key in the
Authorizationorx-api-keyheader, exactly as on HTTP requests. A refused upgrade answers ordinary problem+json. - Name the contextChoose a
context_idfor each call, for example your own call id. It holds 1 to 64 characters of letters, digits, and.,_,:, and-. Every JSON frame you send and every event you receive carries it. - startA
startframe that names an id which is not open opens a context. It takes the fields the single socket'sstarttakes. Thestartedevent answers withconnection, the context's id on our side and the one to quote in support requests. - Send framesSend the context's other frames with its
context_id. Speech takestext,flush,cancel, andend, and transcription takesplaybackandend. Audio travels in binary frames.
// Runs on Bun as-is; on Node, `import WebSocket from "ws"` accepts
// the same constructor options.
const socket = new WebSocket("wss://api.paxalabs.com/v1/tts/multi", {
headers: { Authorization: `Bearer ${process.env.PAXA_API_KEY}` },
});
socket.binaryType = "arraybuffer";
const send = (frame: object) => socket.send(JSON.stringify(frame));
// One context per call, named by your own call id.
function speak(contextId: string, text: string) {
send({
type: "start",
context_id: contextId,
model: "paxa-tts-flash-v2",
voice: "khanomkrok",
});
send({ type: "text", context_id: contextId, text });
send({ type: "end", context_id: contextId });
}
socket.onopen = () => {
speak("call-1042", "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs");
speak("call-1043", "ขอบคุณที่รอสายครับ");
};
const ascii = new TextDecoder();
socket.onmessage = (event) => {
if (typeof event.data !== "string") {
// One byte counts the id, then the id in ASCII, then the audio.
const frame = new Uint8Array(event.data);
const audioAt = 1 + frame[0];
const contextId = ascii.decode(frame.subarray(1, audioAt));
players.get(contextId)?.feed(frame.subarray(audioAt));
return;
}
const message = JSON.parse(event.data);
switch (message.type) {
case "draining":
// The server is restarting: start new contexts on another socket
// and move each call there at its next turn boundary.
break;
case "closed":
// The context's last event. Its id is free for a new start.
players.delete(message.context_id);
}
};import json
import os
from websockets.sync.client import connect # pip install websockets
def speak(socket, context_id, text):
# One context per call, named by your own call id.
socket.send(json.dumps({
"type": "start",
"context_id": context_id,
"model": "paxa-tts-flash-v2",
"voice": "khanomkrok",
}))
socket.send(json.dumps({"type": "text", "context_id": context_id, "text": text}))
socket.send(json.dumps({"type": "end", "context_id": context_id}))
with connect(
"wss://api.paxalabs.com/v1/tts/multi",
additional_headers={"Authorization": f"Bearer {os.environ['PAXA_API_KEY']}"},
) as socket:
speak(socket, "call-1042", "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs")
speak(socket, "call-1043", "ขอบคุณที่รอสายครับ")
audio = {"call-1042": bytearray(), "call-1043": bytearray()}
running = set(audio)
for frame in socket:
if isinstance(frame, bytes):
# One byte counts the id, then the id in ASCII, then the audio.
audio_at = 1 + frame[0]
audio[frame[1:audio_at].decode("ascii")] += frame[audio_at:]
continue
event = json.loads(frame)
if event["type"] == "draining":
# The server is restarting: start new contexts on another socket
# and move each call there at its next turn boundary.
continue
if event["type"] == "closed":
# The context's last event. Its id is free for a new start.
running.discard(event["context_id"])
if not running:
break
for context_id, data in audio.items():
with open(f"{context_id}.mp3", "wb") as f:
f.write(data)A speech context that speaks one line sends these events, each naming its context:
{"type":"started","context_id":"call-1042","connection":"7d9c2b7e-4a41-4c8e-9a35-1f26d8c04b7a","model":"paxa-tts-flash-v2","voice":"khanomkrok","format":"mp3","sample_rate":44100}
{"type":"charged","context_id":"call-1042","seq":1,"chars":36,"credits":0.36,"total_credits":0.36}
{"type":"flushed","context_id":"call-1042","seq":1}
{"type":"done","context_id":"call-1042","total_chars":36,"total_credits":0.36}
{"type":"closed","context_id":"call-1042","code":1000,"reason":"session complete"}The binary frame header#
Every binary frame opens with one byte that counts the context id, then the id in ASCII. The audio follows. On /v1/tts/multi the server sends these frames, and each carries audio in its context's format. On /v1/stt/multi the client sends them, in the encoding and sample rate that context's start declared. A frame holds at most 64 KiB with its header, and audio may follow start at once.
offset bytes content 0 1 9, the length of the id 1 9 "call-1042" in ASCII 10 rest audio
A transcription client writes the header in front of every audio chunk it sends:
import { open } from "node:fs/promises";
// Runs on Bun as-is; on Node, `import WebSocket from "ws"` accepts
// the same constructor options.
const socket = new WebSocket("wss://api.paxalabs.com/v1/stt/multi", {
headers: { Authorization: `Bearer ${process.env.PAXA_API_KEY}` },
});
const send = (frame: object) => socket.send(JSON.stringify(frame));
const ascii = new TextEncoder();
// One byte counts the id, then the id in ASCII, then the audio.
function audioFrame(contextId: string, audio: Uint8Array) {
const id = ascii.encode(contextId);
const frame = new Uint8Array(1 + id.length + audio.length);
frame[0] = id.length;
frame.set(id, 1);
frame.set(audio, 1 + id.length);
return frame;
}
// One context per call, named by your own call id. Audio may follow
// start at once.
async function listen(contextId: string) {
send({
type: "start",
context_id: contextId,
model: "paxa-stt-lite-realtime-v1-preview",
audio: { encoding: "pcm_s16le", sample_rate: 16000 },
});
// 16 kHz signed 16-bit mono is 32,000 bytes a second. A call delivers
// frames at this pace on its own; a file has to be paced.
const file = await open(`${contextId}.pcm`);
const chunk = new Uint8Array(640); // 20 ms
for (;;) {
const { bytesRead } = await file.read(chunk, 0, chunk.length);
if (bytesRead === 0) break;
socket.send(audioFrame(contextId, chunk.subarray(0, bytesRead)));
await new Promise((resolve) => setTimeout(resolve, 20));
}
send({ type: "end", context_id: contextId });
}
const calls = new Set(["call-1042", "call-1043"]);
socket.onopen = () => {
for (const contextId of calls) void listen(contextId);
};
socket.onmessage = (event) => {
const message = JSON.parse(String(event.data));
switch (message.type) {
case "transcript":
if (message.is_final) console.log(`${message.context_id}: ${message.text}`);
break;
case "draining":
// The server is restarting: start new contexts on another socket
// and move each call there at its next turn boundary.
break;
case "closed":
// The context's last event. Its id is free for a new start.
calls.delete(message.context_id);
if (calls.size === 0) socket.close();
}
};import json
import os
import time
from websockets.sync.client import connect # pip install websockets
CALLS = ["call-1042", "call-1043"]
def audio_frame(context_id, audio):
# One byte counts the id, then the id in ASCII, then the audio.
header = context_id.encode("ascii")
return bytes([len(header)]) + header + audio
with connect(
"wss://api.paxalabs.com/v1/stt/multi",
additional_headers={"Authorization": f"Bearer {os.environ['PAXA_API_KEY']}"},
) as socket:
# One context per call, named by your own call id.
for context_id in CALLS:
socket.send(json.dumps({
"type": "start",
"context_id": context_id,
"model": "paxa-stt-lite-realtime-v1-preview",
"audio": {"encoding": "pcm_s16le", "sample_rate": 16000},
}))
# Audio may follow start at once.
# 16 kHz signed 16-bit mono is 32,000 bytes a second; a file
# has to be paced the way a call paces itself.
files = {context_id: open(f"{context_id}.pcm", "rb") for context_id in CALLS}
while files:
for context_id, audio in list(files.items()):
if chunk := audio.read(640): # 20 ms
socket.send(audio_frame(context_id, chunk))
continue
audio.close()
del files[context_id]
socket.send(json.dumps({"type": "end", "context_id": context_id}))
time.sleep(0.02)
running = set(CALLS)
for message in socket:
event = json.loads(message)
if event["type"] == "draining":
# The server is restarting: start new contexts on another socket
# and move each call there at its next turn boundary.
continue
if event["type"] == "transcript" and event["is_final"]:
print(event["context_id"], event["text"])
elif event["type"] == "closed":
# The context's last event. Its id is free for a new start.
running.discard(event["context_id"])
if not running:
breakHow a context ends#
A context ends with one closed event, the last thing it sends, unless the socket closes first. Its code is the context's close code, and reason states the same in words. After closed the id is free for a new start. One context closing leaves the others running.
A context reports 1000, 1001, 4400, 4401, 4402, 4403, 4408, 4413, 4429, 4500, or 4503, and a transcription context also reports 4502. The speech and transcription references state what each code means. A context closes with 4408 after 300 seconds with no frame for it and nothing sent from it.
A JSON frame other than start that names an id which is not open answers error with live_protocol and that context_id. Nothing closes. Audio for an id that is not open is dropped. A start on an id that is already open is a protocol violation: the context sends error, then closed with 4400. A start that fails validation answers error with validation, then closed with 4400. Send the corrected start under the same id or a new one.
Errors that name no context#
Some frames name no context. The frame is not a JSON object, its context_id is missing or malformed, or a binary frame's header is malformed. Any binary frame sent to /v1/tts/multi counts too, because that socket takes none. Each answers an error with validation and no context_id, and nothing closes.
Server restarts#
Before a restart the server sends a draining event with no context_id to every socket with an open context. A socket with none closes with 1001 at once. The event's grace field gives the seconds until the server closes the contexts still open on the socket. From that event on, a new start on the socket answers error with provider_unavailable, then closed with 1001. Contexts already open keep running.
Open another socket when draining arrives and start new contexts there. Move each call over at its next turn boundary. A context still open after the grace closes with 1001. A speech leg in flight finishes or is refunded, and transcription charges the audio received.
Limits and billing#
Each open context holds one of the plan's concurrent slots for its product, from its start frame until closed. Contexts and single connections count together against the plan, and a socket holds no slot of its own. A start past the plan answers error with concurrency_limited, then closed with 4429.
Each speech leg spends one request of the account's per-minute rate limit, as on /v1/tts/live. A transcription context spends one request at its start. A start past the limit answers error with rate_limited and a retry_after in seconds, then closed with 4429.
A context is billed as one connection of its own. The request floor applies once per context, and charged receipts and done totals count that context alone. The rates are those of live speech and realtime transcription.
The account's plan and the key's spending cap are read again while the socket stays open. A change applies to new contexts without reconnecting.
Closes that end the whole socket#
A socket close ends every context still open on it, with no closed event except on 4401 and 4403. The socket closes with these codes:
| Code | When the socket closes with it |
|---|---|
| 1001 | The server is restarting and the socket has no context left open. A 1001 can also close a socket with contexts open. |
| 1009 | A frame exceeded 64 KiB. |
| 4401 | The API key stopped being valid. Every open context reports closed with 4401 first. |
| 4403 | The key's spending cap was reached. Every open context reports closed with 4403 first. |
| 4408 | 300 seconds passed with no open context and no frame. |
| 4413 | The client read too slowly. |