# Many sessions on one WebSocket

A multiplexed WebSocket carries many live speech or realtime transcription sessions at once. Each is a context with its own id, events, billing, and close code.

[WS /v1/tts/multi](https://paxalabs.com/docs/api/tts-multi) and [WS /v1/stt/multi](https://paxalabs.com/docs/api/stt-multi) each carry many sessions on one long-lived WebSocket. Each session is a context. A context is exactly one [WS /v1/tts/live](https://paxalabs.com/docs/api/live) or [WS /v1/stt/live](https://paxalabs.com/docs/api/stt-live) connection, with the same frames, events, billing, and limits. Both endpoints are built for an application that speaks or listens on many calls at once. For one conversation at a time, use the single sockets.

|  | One socket per call | Multiplexed socket |
| --- | --- | --- |
| Endpoints | `/v1/tts/live`, `/v1/stt/live` | `/v1/tts/multi`, `/v1/stt/multi` |
| A new call | One upgrade | One `start` frame |
| JSON frames and events | As the live references list them | The same, each with `context_id` |
| Binary frames | Audio | A header naming the context, then audio |
| End of a call | The connection closes with a code | A `closed` event carries the code |

## Open a socket and start contexts

1. Connect: Open the WebSocket with the API key in the `Authorization` or `x-api-key` header, exactly as on HTTP requests. A refused upgrade answers ordinary problem+json.
2. Name the context: Choose a `context_id` for each call, for example your own call id. It holds 1 to 64 characters of letters, digits, and `.`, `_`, `:`, and `-`. Every JSON frame you send and every event you receive carries it.
3. start: A `start` frame that names an id which is not open opens a context. It takes the fields the single socket's `start` takes. The `started` event answers with `connection`, the context's id on our side and the one to quote in support requests.
4. Send frames: Send the context's other frames with its `context_id`. Speech takes `text`, `flush`, `cancel`, and `end`, and transcription takes `playback` and `end`. Audio travels in binary frames.

TypeScript:

```typescript
// Runs on Bun as-is; on Node, `import WebSocket from "ws"` accepts
// the same constructor options.
const socket = new WebSocket("wss://api.paxalabs.com/v1/tts/multi", {
  headers: { Authorization: `Bearer ${process.env.PAXA_API_KEY}` },
});
socket.binaryType = "arraybuffer";

const send = (frame: object) => socket.send(JSON.stringify(frame));

// One context per call, named by your own call id.
function speak(contextId: string, text: string) {
  send({
    type: "start",
    context_id: contextId,
    model: "paxa-tts-flash-v2",
    voice: "khanomkrok",
  });
  send({ type: "text", context_id: contextId, text });
  send({ type: "end", context_id: contextId });
}

socket.onopen = () => {
  speak("call-1042", "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs");
  speak("call-1043", "ขอบคุณที่รอสายครับ");
};

const ascii = new TextDecoder();
socket.onmessage = (event) => {
  if (typeof event.data !== "string") {
    // One byte counts the id, then the id in ASCII, then the audio.
    const frame = new Uint8Array(event.data);
    const audioAt = 1 + frame[0];
    const contextId = ascii.decode(frame.subarray(1, audioAt));
    players.get(contextId)?.feed(frame.subarray(audioAt));
    return;
  }
  const message = JSON.parse(event.data);
  switch (message.type) {
    case "draining":
      // The server is restarting: start new contexts on another socket
      // and move each call there at its next turn boundary.
      break;
    case "closed":
      // The context's last event. Its id is free for a new start.
      players.delete(message.context_id);
  }
};
```

Python:

```python
import json
import os

from websockets.sync.client import connect  # pip install websockets


def speak(socket, context_id, text):
    # One context per call, named by your own call id.
    socket.send(json.dumps({
        "type": "start",
        "context_id": context_id,
        "model": "paxa-tts-flash-v2",
        "voice": "khanomkrok",
    }))
    socket.send(json.dumps({"type": "text", "context_id": context_id, "text": text}))
    socket.send(json.dumps({"type": "end", "context_id": context_id}))


with connect(
    "wss://api.paxalabs.com/v1/tts/multi",
    additional_headers={"Authorization": f"Bearer {os.environ['PAXA_API_KEY']}"},
) as socket:
    speak(socket, "call-1042", "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs")
    speak(socket, "call-1043", "ขอบคุณที่รอสายครับ")

    audio = {"call-1042": bytearray(), "call-1043": bytearray()}
    running = set(audio)
    for frame in socket:
        if isinstance(frame, bytes):
            # One byte counts the id, then the id in ASCII, then the audio.
            audio_at = 1 + frame[0]
            audio[frame[1:audio_at].decode("ascii")] += frame[audio_at:]
            continue
        event = json.loads(frame)
        if event["type"] == "draining":
            # The server is restarting: start new contexts on another socket
            # and move each call there at its next turn boundary.
            continue
        if event["type"] == "closed":
            # The context's last event. Its id is free for a new start.
            running.discard(event["context_id"])
            if not running:
                break

for context_id, data in audio.items():
    with open(f"{context_id}.mp3", "wb") as f:
        f.write(data)
```

A speech context that speaks one line sends these events, each naming its context:

Server events:

```json
{"type":"started","context_id":"call-1042","connection":"7d9c2b7e-4a41-4c8e-9a35-1f26d8c04b7a","model":"paxa-tts-flash-v2","voice":"khanomkrok","format":"mp3","sample_rate":44100}
{"type":"charged","context_id":"call-1042","seq":1,"chars":36,"credits":0.36,"total_credits":0.36}
{"type":"flushed","context_id":"call-1042","seq":1}
{"type":"done","context_id":"call-1042","total_chars":36,"total_credits":0.36}
{"type":"closed","context_id":"call-1042","code":1000,"reason":"session complete"}
```

## The binary frame header

Every binary frame opens with one byte that counts the context id, then the id in ASCII. The audio follows. On `/v1/tts/multi` the server sends these frames, and each carries audio in its context's format. On `/v1/stt/multi` the client sends them, in the encoding and sample rate that context's `start` declared. A frame holds at most 64 KiB with its header, and audio may follow `start` at once.

one binary frame of context call-1042:

```
offset  bytes  content
0       1      9, the length of the id
1       9      "call-1042" in ASCII
10      rest   audio
```

A transcription client writes the header in front of every audio chunk it sends:

TypeScript:

```typescript
import { open } from "node:fs/promises";

// Runs on Bun as-is; on Node, `import WebSocket from "ws"` accepts
// the same constructor options.
const socket = new WebSocket("wss://api.paxalabs.com/v1/stt/multi", {
  headers: { Authorization: `Bearer ${process.env.PAXA_API_KEY}` },
});

const send = (frame: object) => socket.send(JSON.stringify(frame));

const ascii = new TextEncoder();
// One byte counts the id, then the id in ASCII, then the audio.
function audioFrame(contextId: string, audio: Uint8Array) {
  const id = ascii.encode(contextId);
  const frame = new Uint8Array(1 + id.length + audio.length);
  frame[0] = id.length;
  frame.set(id, 1);
  frame.set(audio, 1 + id.length);
  return frame;
}

// One context per call, named by your own call id. Audio may follow
// start at once.
async function listen(contextId: string) {
  send({
    type: "start",
    context_id: contextId,
    model: "paxa-stt-lite-realtime-v1-preview",
    audio: { encoding: "pcm_s16le", sample_rate: 16000 },
  });
  // 16 kHz signed 16-bit mono is 32,000 bytes a second. A call delivers
  // frames at this pace on its own; a file has to be paced.
  const file = await open(`${contextId}.pcm`);
  const chunk = new Uint8Array(640); // 20 ms
  for (;;) {
    const { bytesRead } = await file.read(chunk, 0, chunk.length);
    if (bytesRead === 0) break;
    socket.send(audioFrame(contextId, chunk.subarray(0, bytesRead)));
    await new Promise((resolve) => setTimeout(resolve, 20));
  }
  send({ type: "end", context_id: contextId });
}

const calls = new Set(["call-1042", "call-1043"]);
socket.onopen = () => {
  for (const contextId of calls) void listen(contextId);
};

socket.onmessage = (event) => {
  const message = JSON.parse(String(event.data));
  switch (message.type) {
    case "transcript":
      if (message.is_final) console.log(`${message.context_id}: ${message.text}`);
      break;
    case "draining":
      // The server is restarting: start new contexts on another socket
      // and move each call there at its next turn boundary.
      break;
    case "closed":
      // The context's last event. Its id is free for a new start.
      calls.delete(message.context_id);
      if (calls.size === 0) socket.close();
  }
};
```

Python:

```python
import json
import os
import time

from websockets.sync.client import connect  # pip install websockets

CALLS = ["call-1042", "call-1043"]


def audio_frame(context_id, audio):
    # One byte counts the id, then the id in ASCII, then the audio.
    header = context_id.encode("ascii")
    return bytes([len(header)]) + header + audio


with connect(
    "wss://api.paxalabs.com/v1/stt/multi",
    additional_headers={"Authorization": f"Bearer {os.environ['PAXA_API_KEY']}"},
) as socket:
    # One context per call, named by your own call id.
    for context_id in CALLS:
        socket.send(json.dumps({
            "type": "start",
            "context_id": context_id,
            "model": "paxa-stt-lite-realtime-v1-preview",
            "audio": {"encoding": "pcm_s16le", "sample_rate": 16000},
        }))
    # Audio may follow start at once.
    # 16 kHz signed 16-bit mono is 32,000 bytes a second; a file
    # has to be paced the way a call paces itself.
    files = {context_id: open(f"{context_id}.pcm", "rb") for context_id in CALLS}
    while files:
        for context_id, audio in list(files.items()):
            if chunk := audio.read(640):  # 20 ms
                socket.send(audio_frame(context_id, chunk))
                continue
            audio.close()
            del files[context_id]
            socket.send(json.dumps({"type": "end", "context_id": context_id}))
        time.sleep(0.02)

    running = set(CALLS)
    for message in socket:
        event = json.loads(message)
        if event["type"] == "draining":
            # The server is restarting: start new contexts on another socket
            # and move each call there at its next turn boundary.
            continue
        if event["type"] == "transcript" and event["is_final"]:
            print(event["context_id"], event["text"])
        elif event["type"] == "closed":
            # The context's last event. Its id is free for a new start.
            running.discard(event["context_id"])
            if not running:
                break
```

## How a context ends

A context ends with one `closed` event, the last thing it sends, unless the socket closes first. Its `code` is the context's close code, and `reason` states the same in words. After `closed` the id is free for a new `start`. One context closing leaves the others running.

A context reports 1000, 1001, 4400, 4401, 4402, 4403, 4408, 4413, 4429, 4500, or 4503, and a transcription context also reports 4502. The [speech](https://paxalabs.com/docs/api/tts-multi) and [transcription](https://paxalabs.com/docs/api/stt-multi) references state what each code means. A context closes with 4408 after 300 seconds with no frame for it and nothing sent from it.

A JSON frame other than `start` that names an id which is not open answers `error` with `live_protocol` and that `context_id`. Nothing closes. Audio for an id that is not open is dropped. A `start` on an id that is already open is a protocol violation: the context sends `error`, then `closed` with 4400. A `start` that fails validation answers `error` with `validation`, then `closed` with 4400. Send the corrected `start` under the same id or a new one.

## Errors that name no context

Some frames name no context. The frame is not a JSON object, its `context_id` is missing or malformed, or a binary frame's header is malformed. Any binary frame sent to `/v1/tts/multi` counts too, because that socket takes none. Each answers an `error` with `validation` and no `context_id`, and nothing closes.

## Server restarts

Before a restart the server sends a `draining` event with no `context_id` to every socket with an open context. A socket with none closes with 1001 at once. The event's `grace` field gives the seconds until the server closes the contexts still open on the socket. From that event on, a new `start` on the socket answers `error` with `provider_unavailable`, then `closed` with 1001. Contexts already open keep running.

Open another socket when `draining` arrives and start new contexts there. Move each call over at its next turn boundary. A context still open after the grace closes with 1001. A speech leg in flight finishes or is refunded, and transcription charges the audio received.

## Limits and billing

Each open context holds one of the plan's [concurrent slots](https://paxalabs.com/docs/rate-limits#concurrency) for its product, from its `start` frame until `closed`. Contexts and single connections count together against the plan, and a socket holds no slot of its own. A `start` past the plan answers `error` with `concurrency_limited`, then `closed` with 4429.

Each speech leg spends one request of the account's per-minute [rate limit](https://paxalabs.com/docs/rate-limits), as on `/v1/tts/live`. A transcription context spends one request at its `start`. A `start` past the limit answers `error` with `rate_limited` and a `retry_after` in seconds, then `closed` with 4429.

A context is billed as one connection of its own. The request floor applies once per context, and `charged` receipts and `done` totals count that context alone. The rates are those of [live speech](https://paxalabs.com/docs/live-streaming#billing) and [realtime transcription](https://paxalabs.com/docs/speech-to-text#realtime).

The account's plan and the key's [spending cap](https://paxalabs.com/docs/authentication#spending-caps) are read again while the socket stays open. A change applies to new contexts without reconnecting.

## Closes that end the whole socket

A socket close ends every context still open on it, with no `closed` event except on 4401 and 4403. The socket closes with these codes:

| Code | When the socket closes with it |
| --- | --- |
| 1001 | The server is restarting and the socket has no context left open. A 1001 can also close a socket with contexts open. |
| 1009 | A frame exceeded 64 KiB. |
| 4401 | The API key stopped being valid. Every open context reports `closed` with 4401 first. |
| 4403 | The key's spending cap was reached. Every open context reports `closed` with 4403 first. |
| 4408 | 300 seconds passed with no open context and no frame. |
| 4413 | The client read too slowly. |
