Many sessions on one WebSocket

A multiplexed WebSocket carries many live speech or realtime transcription sessions at once. Each is a context with its own id, events, billing, and close code.

View as Markdown

WS /v1/tts/multi and WS /v1/stt/multi each carry many sessions on one long-lived WebSocket. Each session is a context. A context is exactly one WS /v1/tts/live or WS /v1/stt/live connection, with the same frames, events, billing, and limits. Both endpoints are built for an application that speaks or listens on many calls at once. For one conversation at a time, use the single sockets.

One socket per callMultiplexed socket
Endpoints/v1/tts/live, /v1/stt/live/v1/tts/multi, /v1/stt/multi
A new callOne upgradeOne start frame
JSON frames and eventsAs the live references list themThe same, each with context_id
Binary framesAudioA header naming the context, then audio
End of a callThe connection closes with a codeA closed event carries the code

Open a socket and start contexts#

  1. ConnectOpen the WebSocket with the API key in the Authorization or x-api-key header, exactly as on HTTP requests. A refused upgrade answers ordinary problem+json.
  2. Name the contextChoose a context_id for each call, for example your own call id. It holds 1 to 64 characters of letters, digits, and ., _, :, and -. Every JSON frame you send and every event you receive carries it.
  3. startA start frame that names an id which is not open opens a context. It takes the fields the single socket's start takes. The started event answers with connection, the context's id on our side and the one to quote in support requests.
  4. Send framesSend the context's other frames with its context_id. Speech takes text, flush, cancel, and end, and transcription takes playback and end. Audio travels in binary frames.
// Runs on Bun as-is; on Node, `import WebSocket from "ws"` accepts
// the same constructor options.
const socket = new WebSocket("wss://api.paxalabs.com/v1/tts/multi", {
  headers: { Authorization: `Bearer ${process.env.PAXA_API_KEY}` },
});
socket.binaryType = "arraybuffer";

const send = (frame: object) => socket.send(JSON.stringify(frame));

// One context per call, named by your own call id.
function speak(contextId: string, text: string) {
  send({
    type: "start",
    context_id: contextId,
    model: "paxa-tts-flash-v2",
    voice: "khanomkrok",
  });
  send({ type: "text", context_id: contextId, text });
  send({ type: "end", context_id: contextId });
}

socket.onopen = () => {
  speak("call-1042", "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs");
  speak("call-1043", "ขอบคุณที่รอสายครับ");
};

const ascii = new TextDecoder();
socket.onmessage = (event) => {
  if (typeof event.data !== "string") {
    // One byte counts the id, then the id in ASCII, then the audio.
    const frame = new Uint8Array(event.data);
    const audioAt = 1 + frame[0];
    const contextId = ascii.decode(frame.subarray(1, audioAt));
    players.get(contextId)?.feed(frame.subarray(audioAt));
    return;
  }
  const message = JSON.parse(event.data);
  switch (message.type) {
    case "draining":
      // The server is restarting: start new contexts on another socket
      // and move each call there at its next turn boundary.
      break;
    case "closed":
      // The context's last event. Its id is free for a new start.
      players.delete(message.context_id);
  }
};

A speech context that speaks one line sends these events, each naming its context:

Server events
{"type":"started","context_id":"call-1042","connection":"7d9c2b7e-4a41-4c8e-9a35-1f26d8c04b7a","model":"paxa-tts-flash-v2","voice":"khanomkrok","format":"mp3","sample_rate":44100}
{"type":"charged","context_id":"call-1042","seq":1,"chars":36,"credits":0.36,"total_credits":0.36}
{"type":"flushed","context_id":"call-1042","seq":1}
{"type":"done","context_id":"call-1042","total_chars":36,"total_credits":0.36}
{"type":"closed","context_id":"call-1042","code":1000,"reason":"session complete"}

The binary frame header#

Every binary frame opens with one byte that counts the context id, then the id in ASCII. The audio follows. On /v1/tts/multi the server sends these frames, and each carries audio in its context's format. On /v1/stt/multi the client sends them, in the encoding and sample rate that context's start declared. A frame holds at most 64 KiB with its header, and audio may follow start at once.

one binary frame of context call-1042
offset  bytes  content
0       1      9, the length of the id
1       9      "call-1042" in ASCII
10      rest   audio

A transcription client writes the header in front of every audio chunk it sends:

import { open } from "node:fs/promises";

// Runs on Bun as-is; on Node, `import WebSocket from "ws"` accepts
// the same constructor options.
const socket = new WebSocket("wss://api.paxalabs.com/v1/stt/multi", {
  headers: { Authorization: `Bearer ${process.env.PAXA_API_KEY}` },
});

const send = (frame: object) => socket.send(JSON.stringify(frame));

const ascii = new TextEncoder();
// One byte counts the id, then the id in ASCII, then the audio.
function audioFrame(contextId: string, audio: Uint8Array) {
  const id = ascii.encode(contextId);
  const frame = new Uint8Array(1 + id.length + audio.length);
  frame[0] = id.length;
  frame.set(id, 1);
  frame.set(audio, 1 + id.length);
  return frame;
}

// One context per call, named by your own call id. Audio may follow
// start at once.
async function listen(contextId: string) {
  send({
    type: "start",
    context_id: contextId,
    model: "paxa-stt-lite-realtime-v1-preview",
    audio: { encoding: "pcm_s16le", sample_rate: 16000 },
  });
  // 16 kHz signed 16-bit mono is 32,000 bytes a second. A call delivers
  // frames at this pace on its own; a file has to be paced.
  const file = await open(`${contextId}.pcm`);
  const chunk = new Uint8Array(640); // 20 ms
  for (;;) {
    const { bytesRead } = await file.read(chunk, 0, chunk.length);
    if (bytesRead === 0) break;
    socket.send(audioFrame(contextId, chunk.subarray(0, bytesRead)));
    await new Promise((resolve) => setTimeout(resolve, 20));
  }
  send({ type: "end", context_id: contextId });
}

const calls = new Set(["call-1042", "call-1043"]);
socket.onopen = () => {
  for (const contextId of calls) void listen(contextId);
};

socket.onmessage = (event) => {
  const message = JSON.parse(String(event.data));
  switch (message.type) {
    case "transcript":
      if (message.is_final) console.log(`${message.context_id}: ${message.text}`);
      break;
    case "draining":
      // The server is restarting: start new contexts on another socket
      // and move each call there at its next turn boundary.
      break;
    case "closed":
      // The context's last event. Its id is free for a new start.
      calls.delete(message.context_id);
      if (calls.size === 0) socket.close();
  }
};

How a context ends#

A context ends with one closed event, the last thing it sends, unless the socket closes first. Its code is the context's close code, and reason states the same in words. After closed the id is free for a new start. One context closing leaves the others running.

A context reports 1000, 1001, 4400, 4401, 4402, 4403, 4408, 4413, 4429, 4500, or 4503, and a transcription context also reports 4502. The speech and transcription references state what each code means. A context closes with 4408 after 300 seconds with no frame for it and nothing sent from it.

A JSON frame other than start that names an id which is not open answers error with live_protocol and that context_id. Nothing closes. Audio for an id that is not open is dropped. A start on an id that is already open is a protocol violation: the context sends error, then closed with 4400. A start that fails validation answers error with validation, then closed with 4400. Send the corrected start under the same id or a new one.

Errors that name no context#

Some frames name no context. The frame is not a JSON object, its context_id is missing or malformed, or a binary frame's header is malformed. Any binary frame sent to /v1/tts/multi counts too, because that socket takes none. Each answers an error with validation and no context_id, and nothing closes.

Server restarts#

Before a restart the server sends a draining event with no context_id to every socket with an open context. A socket with none closes with 1001 at once. The event's grace field gives the seconds until the server closes the contexts still open on the socket. From that event on, a new start on the socket answers error with provider_unavailable, then closed with 1001. Contexts already open keep running.

Open another socket when draining arrives and start new contexts there. Move each call over at its next turn boundary. A context still open after the grace closes with 1001. A speech leg in flight finishes or is refunded, and transcription charges the audio received.

Limits and billing#

Each open context holds one of the plan's concurrent slots for its product, from its start frame until closed. Contexts and single connections count together against the plan, and a socket holds no slot of its own. A start past the plan answers error with concurrency_limited, then closed with 4429.

Each speech leg spends one request of the account's per-minute rate limit, as on /v1/tts/live. A transcription context spends one request at its start. A start past the limit answers error with rate_limited and a retry_after in seconds, then closed with 4429.

A context is billed as one connection of its own. The request floor applies once per context, and charged receipts and done totals count that context alone. The rates are those of live speech and realtime transcription.

The account's plan and the key's spending cap are read again while the socket stays open. A change applies to new contexts without reconnecting.

Closes that end the whole socket#

A socket close ends every context still open on it, with no closed event except on 4401 and 4403. The socket closes with these codes:

CodeWhen the socket closes with it
1001The server is restarting and the socket has no context left open. A 1001 can also close a socket with contexts open.
1009A frame exceeded 64 KiB.
4401The API key stopped being valid. Every open context reports closed with 4401 first.
4403The key's spending cap was reached. Every open context reports closed with 4403 first.
4408300 seconds passed with no open context and no frame.
4413The client read too slowly.