path: root/app/docs/backend-sglomni.md

# SGLang-Omni Backend

`--backend sglomni` talks to an [SGLang-Omni](https://github.com/sgl-project/sglang-omni)
server (`sgl-omni serve`), which hosts **one TTS model per server process** behind the
OpenAI-compatible `POST /v1/audio/speech` endpoint. The backend is installed as a pip
package into its own managed venv (`app/envs/sglomni`) and its models are pre-downloaded
into the standard HuggingFace cache — the same flow as the `qwen` and `faster` backends.

## Requirements

- **Linux with an NVIDIA GPU** (recent driver). SGLang-Omni's serving stack is
  CUDA-only: there are no Windows builds, and Apple Silicon is limited to ASR.
  The setup wizard warns loudly when `nvidia-smi` does not answer.
- **Compute capability matters per model.** ZONOS2's default pipeline
  quantizes its MoE experts to FP8 at load time, and the FP8 Triton kernels
  only compile on compute capability 8.9+ (RTX 4090/5090, Hopper). On older
  GPUs (Ampere: RTX 30xx, A100) the managed spec automatically launches the
  vendored bf16 config (`app/backends/sglomni/configs/zonos2_bf16.yaml`)
  instead, which runs ZONOS2 in bf16 at roughly twice the MoE VRAM — the
  status line tags the model `zonos2 (bf16 fallback)` and the install/boot
  output says so. That config also raises the engine's
  `mem_fraction_static` to 0.70: the builder's 0.5 default leaves no room
  for the KV cache once the bf16 weights (~11.5 GB) are resident on a
  24 GB card, and the server aborts with "Loaded weights leave no GPU
  memory for the KV cache". Practical floor: a ~16 GB-class card; all
  stages stay colocated on GPU 0.
- **Python 3.10, 3.11 or 3.12** for the backend venv. `sglang-omni` requires
  `>=3.10,<3.13` while the app itself is version-agnostic, so the wizard
  resolves a compatible interpreter automatically:
  1. the app venv's interpreter when it is already 3.10-3.12 (no download);
  2. a `python3.12`/`python3.11`/`python3.10` found on `PATH`;
  3. otherwise it pip-installs `uv` into the app venv and provisions a managed
     standalone CPython 3.12 into `app/envs/pythons` (checksum-verified, no root
     required — this is the zero-prerequisites path on hosts like stock Arch).
- **Disk space**: the venv alone is ~10 GB (torch, sglang, flash-attn, flashinfer,
  CUDA-13 wheels); each model adds 1-10 GB of weights in `~/.cache/huggingface/hub`.
- `ffmpeg` (already a project prerequisite).

The venv pip-installs `sglang-omni` with `--pre` (its dependency stack includes
prerelease components). The verified version at the time of writing is **0.1.4**.

## Model Catalog

Each catalog entry is one installable model. Both the setup wizard and the
hub's **Configure Backends… → Configure SGLang-Omni** screen use the same
checkbox tree (grouped by upstream org, like the audio.cpp one), with the
already-installed models pre-checked: checking a model installs it — running
its companion-package recipe, then pre-downloading its weights with the
venv's `hf` CLI (resumable, cancelable) — and unchecking one removes its
cached weights after a confirm (a managed server hosting that model is
stopped first). The whole diff runs as one task-view pass, removals before
downloads, and the tree re-opens afterwards reflecting the state on disk.
Models can also be left uninstalled — the first server start for one fetches
its weights implicitly, but a pre-download keeps the managed server's boot
inside the start timeout.

| Catalog key | Model | Voice | Notes |
| --- | --- | --- | --- |
| `qwen3_tts_0_6b_customvoice` | Qwen3-TTS 0.6B CustomVoice | built-in speakers | lightest model |
| `qwen3_tts_0_6b_base` | Qwen3-TTS 0.6B Base | clone (reference required) | |
| `qwen3_tts_1_7b_base` | Qwen3-TTS 1.7B Base | clone (reference required) | higher quality |
| `qwen3_tts_1_7b_voicedesign` | Qwen3-TTS 1.7B VoiceDesign | `--instructions` | |
| `higgs_audio_v3_tts` | Higgs Audio v3 TTS | default voice or clone | launches with a vendored config: VRAM headroom for 24 GB cards + a raised generation cap (requests carry `max_new_tokens=3000` — upstream pins the thinker engine's context length at 4096, so prompt + generation must fit 4095 tokens, ~40 s of speech — and sub-requests cap at 80 words, with a pre-run popup offering the CHUNK_SIZE clamp) |
| `moss_tts` | MOSS-TTS v1.5 | clone (reference required) | |
| `moss_tts_local` | MOSS-TTS Local v1.5 | default voice or clone | 48 kHz |
| `voxtral_tts` | Voxtral TTS 4B | preset named voices | e.g. `default`, `casual_male` |
| `dots_tts_mf` | dots.tts (MeanFlow) | clone (reference required) | |
| `fish_s2_pro` | Fish Speech S2-Pro | default voice or clone | needs ~24 GB VRAM (known OOM on a single RTX 3090, upstream issue #359) |
| `zonos2` | ZONOS2 | clone (reference required) | 44.1 kHz; FP8 pipeline falls back to bf16 on GPUs below compute capability 8.9; requests carry `max_new_tokens=12288` because the engine's 1024-frame default caps one request at ~12 s of speech |

In the hub's **Generate Audiobooks** form the Model picker reads as a table,
like the audio.cpp one: each entry's label is padded to the widest one and its
capabilities are rendered as fixed columns so every capability word lines up
down its own column — `tts` in the first column (entries that voice plain
text with a preset or built-in default voice), `clone` in the second (the
entry clones a reference clip), `design` in the third (Qwen3-TTS VoiceDesign).
Clone entries that also narrate without a reference (Higgs Audio v3, MOSS-TTS
Local, Fish Speech S2-Pro) carry both `tts  clone`.

### Companion packages

Some models need extra packages in the backend venv before their server
starts; installing a model through this tool runs its recipe automatically.

| Model family | Companion packages (as upstream instructs) | System packages |
| --- | --- | --- |
| Qwen3-TTS (all four) | `sox`, `einops`, `qwen-tts==0.1.1` — all with `--no-deps`: the qwen-tts demo pins Transformers 4, which would replace sglang-omni's pinned 5.x stack (sglang-omni shims the API differences) | the `sox` **binary** (e.g. `sudo pacman -S sox`, `sudo apt install sox`) |
| Fish Speech S2-Pro, ZONOS2 | `descript-audiotools==0.7.2`, `descript-audio-codec==1.0.0` (Descript DAC codec) | `ffmpeg` on the server's `PATH` |

A missing system package never blocks a download — the wizard prints the
remediation and the weights install anyway — but the server will fail to
synthesize with that model until the package is present.

## How voice cloning works

Unlike `audio.cpp` (server-side voice presets) the reference clip travels
**with each request** as `ref_audio` + `ref_text`:

- The transcript comes from `--transcription`, or a local Whisper
  transcription of the clip (the qwen backend's flow). Without any
  transcript the request is sent without `ref_text`, which lowers quality
  for models that use it.
- On a locally-managed (or loopback `--api-url`) server the clip is sent as
  its **local path** — the server reads the file directly.
- Against a remote server the clip is **inlined as a base64 data URL**, so
  `--api-url` runs need no server-side file setup.

Reference clips live in the project's `voices/` directory (10-20 seconds of
clean speech recommended).

### Higgs Audio v3's request window

The Higgs server pins its thinker engine's context length at 4096 tokens
(not configurable), and the scheduler rejects any request whose prompt —
including the reference-audio tokens — plus `max_new_tokens` exceeds that
window ("Request requires more tokens than the thinker KV cache can hold",
on every GPU: the KV pool side is never the binding constraint). One
request can therefore narrate at most ~40 s of speech, so:

- Requests carry `max_new_tokens=3000` (the vendored config raises the
  engine's 2048-frame default to match), and sub-requests cap at 80 words.
  Before a run with Higgs, a popup offers to clamp CHUNK_SIZE for the run
  ("Set Chunk to 80"), keep the configured size (audio may cut off
  mid-chunk), or cancel; CLI runs answer on the console (`[S/t/c]`, and a
  closed stdin clamps so unattended runs never truncate silently).
- When a request is still rejected (a long reference clip, for example),
  the client refits `max_new_tokens` to the capacity the server reports
  and resends once — the same request never retries as-is.
- Voice cloning with a *long* reference clip eats into the same window:
  keep references around 10-20 seconds (as recommended above) for the
  most usable generation headroom.

## Server lifecycle

The hub and the CLI start and stop the managed instance around each run
(like every backend): without `--api-url` the CLI boots the selected model's
server, converts, and stops it again; a server already answering at the
configured endpoint is used as-is and left running. Because one process
hosts one model, a run whose selected model differs from the hosted one
restarts a server this tool started — a foreign server hosting another
model refuses the run with an actionable message instead.

The hub's **Start/Stop Backend Servers** menu asks which *downloaded* model
to load when starting the server fresh (with none downloaded the entry says
so and points at the Configure screen; a single model starts without
asking). A start also auto-installs a model's missing companion packages
first, so a boot never dies on their import even when the weights arrived
via the shared HuggingFace cache or an earlier install's pip run failed.

The server boots a multi-stage pipeline (preprocessing → TTS generation →
vocoder) and may pull companion weights on first start, so its start
timeout is larger than the other backends' (20 minutes). Pre-downloading
models keeps cold boots well inside it.

External servers: point the **Settings → SGLang-Omni remote URL** (or the
CLI's `--api-url`) at an `sgl-omni` instance. The hub discovers it via
`GET /health` (`{"status": "healthy", "stages": [...]}`) and lists its
model from `GET /v1/models`; models uploaded to that server via
`POST /v1/audio/voices` appear in its Voice menu.

## Troubleshooting

**The generate screen sits on "Status: starting", and the server log says
`Port 8100 is already in use ... Using port 37183 instead`.** Something
else already holds the configured port (often a stale `sgl-omni` from an
earlier attempt). The upstream launcher does not fail — it silently moves
the server to a random port, where no client ever looks for it. The app
detects this in the boot log, aborts the boot, and names both ports.
Find and stop the process holding the port
(`ss -tlnp 'sport = :8100'`), or move this server to a free port
(Settings → SGLang-Omni port), then start again.

**The first request dies with `HTTP 500: CUDA out of memory. Tried to
allocate ~100 MiB` (and every retry fails identically) on a 24 GB card,
with the model far smaller than the card.** The model size is not the
problem: upstream's Qwen3-TTS, MOSS-TTS and Voxtral pipelines colocate
their AR engine and vocoder on GPU 0 with the engine's sglang
`mem_fraction_static` unset, so the static pool (weights + KV cache) is
auto-sized to nearly all free VRAM at boot. By the time a real request
runs, the engine's CUDA graphs (~1.5 GB) and the colocated vocoder are
resident too and only tens of MiB are left — the transient allocations
generation needs do not fit. The vendored configs for those models pin
the pool exactly for this — 0.70 for the small ones (~7 GB of headroom
on a 24 GB card); Higgs (`gpu_memory_fraction: 0.80`) and ZONOS2's
bf16 fallback (`0.70`) carry the same treatment. MOSS-TTS is the
exception: its ~17.1 GB of v1.5 bf16 weights alone exceed the 0.70
budget on the ~20.7 GB the engine profiler sees at that point (its
colocated audio-tokenizer stage is already resident), so a 0.70 pin
aborts the boot outright with "Loaded weights leave no GPU memory for
the KV cache" (the floor is ~0.828 on a clean 24 GB card) instead of
fixing the request-time OOM — it runs `mem_fraction_static: 0.87`,
keeping a ~0.8 GB KV pool with ~2.7 GB of slack for the CUDA graphs
and the colocated vocoder. If you hit the failure
on a server started by hand or an older checkout, pin the budget on the
command line and let this tool use the running server as-is:

```bash
app/envs/sglomni/bin/sgl-omni serve \
  --model-path Qwen/Qwen3-TTS-12Hz-0.6B-Base \
  --config app/backends/sglomni/configs/qwen3_tts_0_6b.yaml \
  --port 8100 --mem-fraction-static 0.70
```

(MOSS-TTS Local needs none of this: its upstream config already budgets
its colocated stages explicitly. Fish Speech S2-Pro's OOM is a different
problem — the model itself needs ~24 GB, upstream issue #359.)

## Manual setup

```bash
# 1. A compatible venv (any of: system python3.10-3.12, or uv-managed)
uv venv --seed --python 3.12 app/envs/sglomni

# 2. The package (into that venv)
app/envs/sglomni/bin/python -m pip install --pre sglang-omni

# 3. A model's companion packages (example: Qwen3-TTS)
app/envs/sglomni/bin/python -m pip install --no-deps sox einops
app/envs/sglomni/bin/python -m pip install --no-deps qwen-tts==0.1.1

# 4. Model weights (example)
app/envs/sglomni/bin/hf download bosonai/higgs-audio-v3-tts-4b

# 5. Start a server manually (the managed flow does this for you)
app/envs/sglomni/bin/sgl-omni serve \
  --model-path bosonai/higgs-audio-v3-tts-4b \
  --config app/backends/sglomni/configs/higgs_audio_v3_tts.yaml \
  --port 8100

# 6. Convert
python audiobook.py --backend sglomni --model higgs_audio_v3_tts \
  --api-url http://127.0.0.1:8100
```

Models that take a vendored config file (all but ZONOS2) add
`--config app/backends/sglomni/configs/<key>.yaml` to the serve command —
the managed spec builds this from the catalog automatically. Higgs's config
also exists for a reason: the upstream pipeline budgets 98% of the card
across its colocated stages, which OOMs on a 24 GB GPU once anything else
uses VRAM, and its engine caps requests at ~27 s of speech.

## CLI examples

```bash
# Zero-shot narration with Higgs Audio v3
python audiobook.py --backend sglomni --model higgs_audio_v3_tts

# Voice cloning from a reference clip (transcript transcribed locally)
python audiobook.py --backend sglomni --model moss_tts_local --clone voices/narrator.wav

# Built-in speakers (Qwen3-TTS CustomVoice)
python audiobook.py --backend sglomni --model qwen3_tts_0_6b_customvoice --voice Vivian

# Voice design
python audiobook.py --backend sglomni --model qwen3_tts_1_7b_voicedesign \
  --instructions "A warm adult female narrator with a British accent"
```