Multi-speaker dialogue

POST /v1/tts/dialogue speaks up to 100 lines, each in its own voice, as one audio file with a pause between lines. One charge covers the script, and a failure keeps only the price of the lines delivered whole.

View as Markdown

Send the lines of a conversation to POST /v1/tts/dialogue, each with its own voice, and receive one audio file of the whole exchange. Each line follows every text rule of POST /v1/tts, with the tags its voice reads, but has no length ceiling of its own. A request holds up to 100 lines and 20,000 characters in all.

curl -X POST https://api.paxalabs.com/v1/tts/dialogue \
  -H "Authorization: Bearer $PAXA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "paxa-tts-flash-v2",
  "lines": [
    {
      "text": "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs",
      "voice": "khanomkrok"
    },
    {
      "text": "สวัสดีค่ะ วันนี้ให้ช่วยเรื่องอะไรดีคะ",
      "voice": "nomyen"
    }
  ]
}' \
  --output dialogue.ndjson

import { writeFile } from "node:fs/promises";
import { createInterface } from "node:readline";
import { Readable } from "node:stream";

const response = await fetch("https://api.paxalabs.com/v1/tts/dialogue", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.PAXA_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "model": "paxa-tts-flash-v2",
    "lines": [
      {
        "text": "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs",
        "voice": "khanomkrok"
      },
      {
        "text": "สวัสดีค่ะ วันนี้ให้ช่วยเรื่องอะไรดีคะ",
        "voice": "nomyen"
      }
    ]
  }),
});

if (!response.ok) throw new Error(`Request failed: ${response.status}`);
const audio: Buffer[][] = [];
let failure;
for await (const line of createInterface({ input: Readable.fromWeb(response.body!) })) {
  if (!line) continue;
  const event = JSON.parse(line);
  if (event.type === "error") failure = event;
  if (event.type === "chunk" && event.audio) {
    (audio[event.line] ??= []).push(Buffer.from(event.audio, "base64"));
  }
}
const kept = failure ? audio.slice(0, failure.lines.length) : audio;
await writeFile("dialogue.mp3", Buffer.concat(kept.flat()));
if (failure) throw new Error(failure.error.title);

import base64
import json
import os
import requests

response = requests.post(
    "https://api.paxalabs.com/v1/tts/dialogue",
    headers={"Authorization": f"Bearer {os.environ['PAXA_API_KEY']}"},
    json={
        "model": "paxa-tts-flash-v2",
        "lines": [
            {
                "text": "สวัสดีครับ ยินดีต้อนรับสู่ Paxa Labs",
                "voice": "khanomkrok",
            },
            {
                "text": "สวัสดีค่ะ วันนี้ให้ช่วยเรื่องอะไรดีคะ",
                "voice": "nomyen",
            },
        ],
    },
    stream=True,
)
response.raise_for_status()
audio = {}
failure = None
for line in response.iter_lines():
    if not line:
        continue
    event = json.loads(line)
    if event["type"] == "error":
        failure = event
    if event["type"] == "chunk" and event.get("audio"):
        audio.setdefault(event["line"], []).append(base64.b64decode(event["audio"]))
kept = len(failure["lines"]) if failure else len(audio)
with open("dialogue.mp3", "wb") as f:
    for index in range(kept):
        f.write(b"".join(audio.get(index, [])))
if failure:
    raise RuntimeError(failure["error"]["title"])

Read the stream#

The response always streams NDJSON. Each chunk event names its line and carries base64 audio, and the chunks decoded in arrival order are one file in the requested format. Lines are synthesized in parallel and delivered in order, and playback can start at the first chunk. The final done event gives each line's start and end in seconds, for captions or a seek bar.

NDJSON stream

{"type":"chunk","line":0,"audio":"//NkxAAaOAAAAAABLBQAAAAAAABVVVVV..."}
{"type":"chunk","line":1,"audio":"//NkxAAaOAAAAAABLBQAAAAAAABVVVVV..."}
{"type":"done","lines":[{"start":0,"end":2.35},{"start":2.65,"end":4.9}]}

A WAV dialogue opens with one header whose size fields read 0xFFFFFFFF. Its length is known only at the end, and players read such a file to its end.

Pauses between lines#

gap_ms sets the silence between one line and the next, from 0 to 5,000 ms, 300 ms by default. It never changes the charge. Opus rounds the gap to 20 ms and MP3 to one frame. For a pause inside a line, write a pause tag.

Billing and failures#

One charge covers every line's characters together at the model's rate, with the 0.1-credit minimum applied once. A line in a designed voice bills at that voice's rate, and timestamps apply to every line.
Every line is checked before the charge. A refusal's problem body carries line, the index of the line at fault. A 400 tag_invalid names the line to fix.
A failure after the charge but before any audio answers a problem whose body carries line, and refunds the whole charge.
A failure after audio began ends the stream with an error event. Its lines lists the lines delivered whole, which are the chunks whose line is below that count, and credits is what they cost as a dialogue of their own. The request keeps that charge and the rest is refunded. A stream read more slowly than it is made can run out of the request's time budget. It then ends with no done event, and the charge stands, as it does when the client disconnects.
A dialogue is one request to the rate window and holds one speech concurrency slot while it runs. An Idempotency-Key works as it does on POST /v1/tts.

PreviousText to speech NextStreaming