# Multiplexed realtime transcription

`WS wss://api.paxalabs.com/v1/stt/multi`

WS /v1/stt/multi carries many realtime transcriptions on one WebSocket. Each is a context with its own id, billed and limited as one /v1/stt/live connection.

Connect once. Each call then opens a context with a `start` frame that carries a `context_id` you choose, and the context takes the frames of [WS /v1/stt/live](https://paxalabs.com/docs/api/stt-live) and sends its events. Every event names its context. Stream each call's audio in binary frames that open with a header naming the context. A context ends with a `closed` event unless the socket closes first, and the others keep running. Each open context holds one of the plan's [concurrent transcription slots](https://paxalabs.com/docs/rate-limits#concurrency) from its `start` frame until `closed`. The [Multiplexing guide](https://paxalabs.com/docs/multiplexing) covers the frame header, restarts, and limits.

## Headers

- `authorization` (Bearer pxa_..., required): Authorization: Bearer pxa_... (recommended).
- `x-api-key` (pxa_..., optional): Alternative to the Authorization header. Ignored when Authorization is present.

See authentication at https://paxalabs.com/docs/authentication.

## Connect

`wss://api.paxalabs.com/v1/stt/multi`. Binary frames carry audio in. JSON text frames carry control in, and turn, transcript, and billing events out, each naming its context. Where a message below says connection, read context.

## Client messages

**start**: Configures the session. Must be the first frame; exactly one per connection.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("start", required)
- `model` (string, required, example "paxa-stt-lite-realtime-v1-preview"): Realtime transcription model id, for example paxa-stt-lite-realtime-v1-preview. GET /v1/models lists the served catalog; a recording model answers unknown_model here.
- `audio` (object, required): What the binary frames carry, fixed for the connection. The bytes are read exactly as declared. A wrong declaration transcribes noise.
  - `audio.encoding` ("pcm_s16le" or "mulaw", required): "pcm_s16le" is signed 16-bit little-endian PCM, "mulaw" is G.711 mu-law as telephony delivers it. Mono.
  - `audio.sample_rate` (16000 or 48000 or 8000, required): Samples per second of the binary frames: 16000 or 48000 for pcm_s16le, 8000 for mulaw. Any other pairing answers validation.
- `language` (string, optional, up to 16 characters, pattern ^[A-Za-z]{2,3}(-[A-Za-z0-9]{2,8})*$, example "th"): The language the speech is expected to be in, as a BCP 47 tag such as "th", "en", or "zh". The hint filters nothing. Speech in any other language is still transcribed as spoken, and Thai that switches into English mid-sentence needs no setting. Omitted, the model detects the language itself. Set the tag when detection has picked the wrong language.
- `timestamps` ("word", optional): Adds a words array to every final transcript, one span per word with its start and end in seconds on the connection's clock. Thai words are segmented for you. Costs nothing extra.
- `style` ("verbatim" or "clean", optional, default "verbatim"): How faithfully the transcript follows the speech. "verbatim", the default, writes exactly what was said: fillers, false starts, and self-corrections included. "clean" drops fillers and merges broken phrases into one, for a transcript that is easier to read. It is a readability choice. A clean transcript departs further from what was said, and our own measurement found it less exact. Any other value answers 400 validation.
- `convention` ("spoken" or "written", optional, default "spoken"): How the transcript writes what was said. "spoken", the default, writes numbers as they were said (สามสิบเปอร์เซ็นต์) and a word said twice as two words, which is what a verbatim transcript, a subtitle, or a language model reading the text needs. "written" keeps display conventions: digits and units (30%), the repetition mark ๆ, for dictated text that will be read on a page. The word timestamps are the same under both. Only Thai is respelled: a transcript in another language, English and Mandarin among them, reads the same under both.
- `vocabulary` (array of strings, optional, example ["Paxa Labs","กรุงเทพฯ","quarterly review"]): Terms the speech is likely to contain that the model should recognize as written: product names, people, places, jargon, in Thai or any other language. Up to 50 terms of 50 characters each, and a longer list or term answers 400 validation. The terms bias recognition and are never inserted; a term that was not said does not appear. Costs nothing extra and combines with every other option.

**playback**: Tells the turn detector when your agent is speaking. Its own voice reaching the microphone then cannot open a turn. While speaking is true, opening a turn takes more sustained speech from the user; a real interruption still opens one and arrives as speech_started. Send it on every change. A session that never sends it treats the room as quiet.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("playback", required)
- `speaking` (boolean, required): true while your agent's audio is playing into the same room as the microphone, false once it stops.

**end**: Closes the turn in progress, if any, and delivers its transcript, then the server sends done and closes with code 1000. Audio after end is ignored.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("end", required)

## Server events

Audio frames: Each frame opens with one byte counting the context id, 1 to 64, then the id in ASCII. The rest is audio in the encoding and sample rate that context's start frame declared, in any chunking and at any cadence, up to 64 KiB a frame with its header. Audio may follow the start frame at once. Audio that names a context which is not open is dropped.

**started**: The session is configured and listening.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("started", required)
- `connection` (string, required): Connection id; quote it in support requests (the x-request-id analog).
- `model` (string, required): The model serving this session.

**speech_started**: The user started speaking. This is the interruption signal. Stop your agent's audio and cancel the reply it was generating. It arrives once enough speech has been heard to be sure. The time it reports is therefore earlier than the moment it arrives.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("speech_started", required)
- `turn` (integer, required, min 1): The turn this event belongs to, counting from 1 per connection. A higher number is a later turn: an agent that already replied to turn 3 discards turn 3 and answers turn 4.
- `time` (number, required): Where the user started speaking. In seconds of audio received since the connection's first binary frame, exact to a thousandth.

**transcript**: A transcript of one turn. The final transcript is the turn: send its text to your language model. An interim gives an earlier look at the same turn and never needs handling on its own.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("transcript", required)
- `turn` (integer, required, min 1): The turn this event belongs to, counting from 1 per connection. A higher number is a later turn: an agent that already replied to turn 3 discards turn 3 and answers turn 4.
- `is_final` (boolean, required): true for the turn's finished transcript, sent exactly once per turn. false for an interim transcript of the turn so far, sent at a natural pause and replaced by the final.
- `text` (string, required): The verbatim transcript of the turn, or of the turn so far. Empty on a final when the turn held no speech the model could read.
- `start` (number, required): Where the transcribed audio starts. In seconds of audio received since the connection's first binary frame, exact to a thousandth.
- `end` (number, required): Where the transcribed audio ends. In seconds of audio received since the connection's first binary frame, exact to a thousandth.
- `words` (array of objects, optional): Word spans on the connection's clock, present on a final when the session set timestamps.
  - `words[].text` (string, required): The spoken input text this span covers. An emotion tag takes no span.
  - `words[].start` (number, required): Where the span begins in the audio, in seconds.
  - `words[].end` (number, required): Where the span ends in the audio, in seconds.

**speech_ended**: The user stopped speaking and the turn is closed. Its final transcript follows. Use it to show that the agent is thinking.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("speech_ended", required)
- `turn` (integer, required, min 1): The turn this event belongs to, counting from 1 per connection. A higher number is a later turn: an agent that already replied to turn 3 discards turn 3 and answers turn 4.
- `time` (number, required): Where the turn ended. In seconds of audio received since the connection's first binary frame, exact to a thousandth.

**charged**: Billing receipt. The connection is charged for the audio it receives, silence included. A period closes at the end of each turn, after a minute of audio with no turn, and at the end of the session. Receipts replace the x-credits-charged header.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("charged", required)
- `seq` (integer, required): The billing period this receipt covers, starting at 1.
- `seconds` (number, required): Audio received in this period, in seconds, exact to a thousandth.
- `credits` (number, required): Credits charged for this period, exact to a hundredth. The 0.1-credit request floor applies once per connection, on the first receipt.
- `total_credits` (number, required): Credits charged on this connection so far, the audio of failed turns excluded, exact to a hundredth.

**error**: Something failed. A recoverable error leaves the context open. After a fatal one the context sends closed with the code listed below, and the socket itself closes on 4401 and 4403.

- `context_id` (string, optional): The context the error belongs to. Absent when the frame named no context. That is a frame that was not a JSON object, one whose context_id was missing or malformed, or a binary frame the socket does not take or whose header was malformed. Nothing closes on such an error.
- `type` ("error", required)
- `code` (string, required): Stable machine-readable code from the error catalog. Branch on this field; the errors reference documents every code.
- `turn` (integer, optional): The turn the error belongs to, when it belongs to one. That turn delivers no transcript; the session keeps listening.
- `not_billed` (number, optional): Credits this turn's audio is held back from the bill, exact to a hundredth. It comes off what the periods after this turn owe. A turn that fails with little audio behind it returns less than this. Sent with a turn.
- `retry_after` (integer, optional): Seconds to wait before sending start again. Sent with rate_limited.

**done**: Every turn is delivered; the server closes with code 1000 after this event. A connection that closes without done did not finish delivering.

- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `type` ("done", required)
- `total_seconds` (number, required): Audio received on this connection, in seconds, exact to a thousandth.
- `turns` (integer, required): Turns the connection delivered or failed.
- `total_credits` (number, required): Credits charged on this connection, the audio of failed turns excluded, exact to a hundredth. Zero when no turn was delivered, which returns the connection's charge in full.

**closed**: The context is closed, and no audio follows. A context ends with one closed event unless the socket closes first. The id is free for a new start once it arrives.

- `type` ("closed", required)
- `context_id` (string, required, pattern ^[A-Za-z0-9._:-]{1,64}$): The context this message belongs to. You choose it: 1 to 64 characters of letters, digits, and . _ : -, such as your own call id.
- `code` (integer, required): The context's close code, one of those listed below.
- `reason` (string, required): The same in words, for logs.

**draining**: The server is restarting. Open another socket now and start every new context there. Contexts already open keep running. Move each call at its next turn boundary. A start sent here from now on answers provider_unavailable and closed. The socket closes with 1001 once its last context has closed.

- `type` ("draining", required)
- `grace` (integer, required): Seconds until the server closes the contexts still open on this socket.

## Close codes

| Code | Meaning |
| --- | --- |
| 1000 | A context reports it after done. The socket closes with it only when you close it. |
| 1001 | The server is restarting. A context reports it when it was still open after the grace the draining event named. The socket closes with it once its last context has closed. It can also close the socket while contexts are open, and those end with no closed event. Open another socket and start again there. |
| 1009 | The socket closes: a frame exceeded 64 KiB. |
| 4400 | A context reports it: a frame violated the session protocol, or the start frame failed validation. It also reports it when the start frame named an unknown model or a sample rate its encoding does not take. Any other frame that fails validation answers an error event, and the context stays open. |
| 4401 | The API key stopped being valid. Every open context reports it, then the socket closes with it. |
| 4402 | A context reports it: the account cannot cover the request floor at start, or the next billing period. |
| 4403 | The key's spending cap was reached. Every open context reports it, then the socket closes with it. |
| 4408 | A context reports it when no frame arrived for it and no event was sent for 300 seconds. The socket closes with it after 300 seconds with no open context and no frame. |
| 4413 | A context reports it when its frames arrived faster than they could be handled, and a speech context also when its client read a leg too slowly for the leg's time budget, which keeps that leg's charge. The socket closes with it when the client read too slowly and the outbound buffer overflowed. |
| 4429 | A context reports it: the account holds the plan's full count of concurrent transcription sessions, or the start passed the plan's requests per minute. End another context or connection, or wait the error's retry_after seconds, then start again. |
| 4500 | A context reports it: internal server error. |
| 4502 | A context reports it: transcription failed for the whole context. What its delivered periods charged stays charged; a context that delivered no turn is refunded in full. Start it again. |
| 4503 | A context reports it: transcription was unavailable at start. Nothing was charged; retry. |

## Errors

- `validation` (400): The request body, query, or headers failed validation against the endpoint schema, or broke a rule the endpoint's reference states. A field the schema refused adds two fields to the problem body. path names the first failing field by location and name in dot form, such as body.voice or headers.idempotency-key. A body that could not be read gives the path body alone. reason is one stable code from the validation reasons list.
- `live_protocol` (400): A frame on a live WebSocket violated the session protocol. The frame arrived before start or repeated start. On a multiplexed socket, it also answers a frame for a context that is not open.
- `unknown_model` (400): The model field does not name a served model.
- `unauthorized` (401): The request carried no API key, or the key is invalid or disabled.
- `insufficient_credits` (402): The account does not have enough credits for this request. Nothing was charged.
- `key_limit` (403): This API key reached its spending cap. Nothing was charged.
- `concurrency_limited` (429): The account holds the plan's full count of concurrent requests for this product. Nothing was charged. Each product is limited separately, and an open live connection, or an open context of a multiplexed socket, holds one slot of its product for its lifetime.
- `internal` (500): Request state was inconsistent on the server.
- `provider_error` (502): Model inference failed after the request was charged.
- `provider_unavailable` (503): The model behind this endpoint is not available right now. Nothing was charged.
- `rate_limited` (429): Requests per minute for the plan are exhausted. One window covers the whole account, across every product and every key, and voice design calls also have one of their own at 20 per minute.

## Example

TypeScript:

```typescript
import { open } from "node:fs/promises";

// Runs on Bun as-is; on Node, `import WebSocket from "ws"` accepts
// the same constructor options.
const socket = new WebSocket("wss://api.paxalabs.com/v1/stt/multi", {
  headers: { Authorization: `Bearer ${process.env.PAXA_API_KEY}` },
});

const send = (frame: object) => socket.send(JSON.stringify(frame));

const ascii = new TextEncoder();
// One byte counts the id, then the id in ASCII, then the audio.
function audioFrame(contextId: string, audio: Uint8Array) {
  const id = ascii.encode(contextId);
  const frame = new Uint8Array(1 + id.length + audio.length);
  frame[0] = id.length;
  frame.set(id, 1);
  frame.set(audio, 1 + id.length);
  return frame;
}

// One context per call, named by your own call id. Audio may follow
// start at once.
async function listen(contextId: string) {
  send({
    type: "start",
    context_id: contextId,
    model: "paxa-stt-lite-realtime-v1-preview",
    audio: { encoding: "pcm_s16le", sample_rate: 16000 },
  });
  // 16 kHz signed 16-bit mono is 32,000 bytes a second. A call delivers
  // frames at this pace on its own; a file has to be paced.
  const file = await open(`${contextId}.pcm`);
  const chunk = new Uint8Array(640); // 20 ms
  for (;;) {
    const { bytesRead } = await file.read(chunk, 0, chunk.length);
    if (bytesRead === 0) break;
    socket.send(audioFrame(contextId, chunk.subarray(0, bytesRead)));
    await new Promise((resolve) => setTimeout(resolve, 20));
  }
  send({ type: "end", context_id: contextId });
}

const calls = new Set(["call-1042", "call-1043"]);
socket.onopen = () => {
  for (const contextId of calls) void listen(contextId);
};

socket.onmessage = (event) => {
  const message = JSON.parse(String(event.data));
  switch (message.type) {
    case "transcript":
      if (message.is_final) console.log(`${message.context_id}: ${message.text}`);
      break;
    case "draining":
      // The server is restarting: start new contexts on another socket
      // and move each call there at its next turn boundary.
      break;
    case "closed":
      // The context's last event. Its id is free for a new start.
      calls.delete(message.context_id);
      if (calls.size === 0) socket.close();
  }
};
```

Python:

```python
import json
import os
import time

from websockets.sync.client import connect  # pip install websockets

CALLS = ["call-1042", "call-1043"]


def audio_frame(context_id, audio):
    # One byte counts the id, then the id in ASCII, then the audio.
    header = context_id.encode("ascii")
    return bytes([len(header)]) + header + audio


with connect(
    "wss://api.paxalabs.com/v1/stt/multi",
    additional_headers={"Authorization": f"Bearer {os.environ['PAXA_API_KEY']}"},
) as socket:
    # One context per call, named by your own call id.
    for context_id in CALLS:
        socket.send(json.dumps({
            "type": "start",
            "context_id": context_id,
            "model": "paxa-stt-lite-realtime-v1-preview",
            "audio": {"encoding": "pcm_s16le", "sample_rate": 16000},
        }))
    # Audio may follow start at once.
    # 16 kHz signed 16-bit mono is 32,000 bytes a second; a file
    # has to be paced the way a call paces itself.
    files = {context_id: open(f"{context_id}.pcm", "rb") for context_id in CALLS}
    while files:
        for context_id, audio in list(files.items()):
            if chunk := audio.read(640):  # 20 ms
                socket.send(audio_frame(context_id, chunk))
                continue
            audio.close()
            del files[context_id]
            socket.send(json.dumps({"type": "end", "context_id": context_id}))
        time.sleep(0.02)

    running = set(CALLS)
    for message in socket:
        event = json.loads(message)
        if event["type"] == "draining":
            # The server is restarting: start new contexts on another socket
            # and move each call there at its next turn boundary.
            continue
        if event["type"] == "transcript" and event["is_final"]:
            print(event["context_id"], event["text"])
        elif event["type"] == "closed":
            # The context's last event. Its id is free for a new start.
            running.discard(event["context_id"])
            if not running:
                break
```

Server events:

```json
{"type":"started","context_id":"call-1042","connection":"7d9c2b7e-4a41-4c8e-9a35-1f26d8c04b7a","model":"paxa-stt-lite-realtime-v1-preview"}
{"type":"speech_started","context_id":"call-1042","turn":1,"time":0.3}
{"type":"speech_ended","context_id":"call-1042","turn":1,"time":12.252}
{"type":"charged","context_id":"call-1042","seq":1,"seconds":12.252,"credits":2.56,"total_credits":2.56}
{"type":"transcript","context_id":"call-1042","turn":1,"is_final":true,"text":"สวัสดีค่ะคุณสมชายยอดชำระของท่านจำนวน พันสองร้อยห้าสิบ บาทครบกำหนดวันที่ สิบห้า สิงหาคมนี้ชำระผ่านแอปได้ตลอด ยี่สิบสี่ ชั่วโมงค่ะ","start":0.3,"end":12.252}
{"type":"done","context_id":"call-1042","total_seconds":12.252,"turns":1,"total_credits":2.56}
{"type":"closed","context_id":"call-1042","code":1000,"reason":"session complete"}
```
