# SGLang-Omni Backend

`--backend sglomni` talks to an [SGLang-Omni](https://github.com/sgl-project/sglang-omni)
server (`sgl-omni serve`), which hosts **one TTS model per server process** behind the
OpenAI-compatible `POST /v1/audio/speech` endpoint. The backend is installed as a pip
package into its own managed venv (`app/envs/sglomni`) and its models are pre-downloaded
into the standard HuggingFace cache — the same flow as the `qwen` and `faster` backends.

## Requirements

- **Linux with an NVIDIA GPU** (recent driver). SGLang-Omni's serving stack is
  CUDA-only: there are no Windows builds, and Apple Silicon is limited to ASR.
  The setup wizard warns loudly when `nvidia-smi` does not answer.
- **Compute capability matters per model.** ZONOS2's default pipeline
  quantizes its MoE experts to FP8 at load time, and the FP8 Triton kernels
  only compile on compute capability 8.9+ (RTX 4090/5090, Hopper). On older
  GPUs (Ampere: RTX 30xx, A100) the managed spec automatically launches the
  vendored bf16 config (`app/backends/sglomni/configs/zonos2_bf16.yaml`)
  instead, which runs ZONOS2 in bf16 at roughly twice the MoE VRAM — the
  status line tags the model `zonos2 (bf16 fallback)` and the install/boot
  output says so. That config also raises the engine's
  `mem_fraction_static` to 0.70: the builder's 0.5 default leaves no room
  for the KV cache once the bf16 weights (~11.5 GB) are resident on a
  24 GB card, and the server aborts with "Loaded weights leave no GPU
  memory for the KV cache". Practical floor: a ~16 GB-class card; all
  stages stay colocated on GPU 0.
- **Python 3.10, 3.11 or 3.12** for the backend venv. `sglang-omni` requires
  `>=3.10,<3.13` while the app itself is version-agnostic, so the wizard
  resolves a compatible interpreter automatically:
  1. the app venv's interpreter when it is already 3.10-3.12 (no download);
  2. a `python3.12`/`python3.11`/`python3.10` found on `PATH`;
  3. otherwise it pip-installs `uv` into the app venv and provisions a managed
     standalone CPython 3.12 into `app/envs/pythons` (checksum-verified, no root
     required — this is the zero-prerequisites path on hosts like stock Arch).
- **Disk space**: the venv alone is ~10 GB (torch, sglang, flash-attn, flashinfer,
  CUDA-13 wheels); each model adds 1-10 GB of weights in `~/.cache/huggingface/hub`.
- `ffmpeg` (already a project prerequisite).

The venv pip-installs `sglang-omni` with `--pre` (its dependency stack includes
prerelease components). The verified version at the time of writing is **0.1.4**.

## Model Catalog

Each catalog entry is one installable model. Both the setup wizard and the
hub's **Configure Backends… → Configure SGLang-Omni** screen use the same
checkbox tree (grouped by upstream org, like the audio.cpp one), with the
already-installed models pre-checked: checking a model installs it — running
its companion-package recipe, then pre-downloading its weights with the
venv's `hf` CLI (resumable, cancelable) — and unchecking one removes its
cached weights after a confirm (a managed server hosting that model is
stopped first). The whole diff runs as one task-view pass, removals before
downloads, and the tree re-opens afterwards reflecting the state on disk.
Models can also be left uninstalled — the first server start for one fetches
its weights implicitly, but a pre-download keeps the managed server's boot
inside the start timeout.

| Catalog key | Model | Voice | Notes |
| --- | --- | --- | --- |
| `qwen3_tts_0_6b_customvoice` | Qwen3-TTS 0.6B CustomVoice | built-in speakers | lightest model |
| `qwen3_tts_0_6b_base` | Qwen3-TTS 0.6B Base | clone (reference required) | |
| `qwen3_tts_1_7b_base` | Qwen3-TTS 1.7B Base | clone (reference required) | higher quality |
| `qwen3_tts_1_7b_voicedesign` | Qwen3-TTS 1.7B VoiceDesign | `--instructions` | |
| `higgs_audio_v3_tts` | Higgs Audio v3 TTS | default voice or clone | no config file needed |
| `moss_tts` | MOSS-TTS v1.5 | clone (reference required) | |
| `moss_tts_local` | MOSS-TTS Local v1.5 | default voice or clone | 48 kHz |
| `voxtral_tts` | Voxtral TTS 4B | preset named voices | e.g. `default`, `casual_male` |
| `dots_tts_mf` | dots.tts (MeanFlow) | clone (reference required) | |
| `fish_s2_pro` | Fish Speech S2-Pro | default voice or clone | needs ~24 GB VRAM (known OOM on a single RTX 3090, upstream issue #359) |
| `zonos2` | ZONOS2 | clone (reference required) | 44.1 kHz; FP8 pipeline falls back to bf16 on GPUs below compute capability 8.9; requests carry `max_new_tokens=12288` because the engine's 1024-frame default caps one request at ~12 s of speech |

In the hub's **Generate Audiobooks** form the Model picker reads as a table,
like the audio.cpp one: each entry's label is padded to the widest one and its
capabilities are rendered as fixed columns so every capability word lines up
down its own column — `tts` in the first column (entries that voice plain
text with a preset or built-in default voice), `clone` in the second (the
entry clones a reference clip), `design` in the third (Qwen3-TTS VoiceDesign).
Clone entries that also narrate without a reference (Higgs Audio v3, MOSS-TTS
Local, Fish Speech S2-Pro) carry both `tts  clone`.

### Companion packages

Some models need extra packages in the backend venv before their server
starts; installing a model through this tool runs its recipe automatically.

| Model family | Companion packages (as upstream instructs) | System packages |
| --- | --- | --- |
| Qwen3-TTS (all four) | `sox`, `einops`, `qwen-tts==0.1.1` — all with `--no-deps`: the qwen-tts demo pins Transformers 4, which would replace sglang-omni's pinned 5.x stack (sglang-omni shims the API differences) | the `sox` **binary** (e.g. `sudo pacman -S sox`, `sudo apt install sox`) |
| Fish Speech S2-Pro, ZONOS2 | `descript-audiotools==0.7.2`, `descript-audio-codec==1.0.0` (Descript DAC codec) | `ffmpeg` on the server's `PATH` |

A missing system package never blocks a download — the wizard prints the
remediation and the weights install anyway — but the server will fail to
synthesize with that model until the package is present.

## How voice cloning works

Unlike `audio.cpp` (server-side voice presets) the reference clip travels
**with each request** as `ref_audio` + `ref_text`:

- The transcript comes from `--transcription`, or a local Whisper
  transcription of the clip (the qwen backend's flow). Without any
  transcript the request is sent without `ref_text`, which lowers quality
  for models that use it.
- On a locally-managed (or loopback `--api-url`) server the clip is sent as
  its **local path** — the server reads the file directly.
- Against a remote server the clip is **inlined as a base64 data URL**, so
  `--api-url` runs need no server-side file setup.

Reference clips live in the project's `voices/` directory (10-20 seconds of
clean speech recommended).

## Server lifecycle

The hub and the CLI start and stop the managed instance around each run
(like every backend): without `--api-url` the CLI boots the selected model's
server, converts, and stops it again; a server already answering at the
configured endpoint is used as-is and left running. Because one process
hosts one model, a run whose selected model differs from the hosted one
restarts a server this tool started — a foreign server hosting another
model refuses the run with an actionable message instead.

The server boots a multi-stage pipeline (preprocessing → TTS generation →
vocoder) and may pull companion weights on first start, so its start
timeout is larger than the other backends' (20 minutes). Pre-downloading
models keeps cold boots well inside it.

External servers: point the **Settings → SGLang-Omni remote URL** (or the
CLI's `--api-url`) at an `sgl-omni` instance. The hub discovers it via
`GET /health` (`{"status": "healthy", "stages": [...]}`) and lists its
model from `GET /v1/models`; models uploaded to that server via
`POST /v1/audio/voices` appear in its Voice menu.

## Manual setup

```bash
# 1. A compatible venv (any of: system python3.10-3.12, or uv-managed)
uv venv --seed --python 3.12 app/envs/sglomni

# 2. The package (into that venv)
app/envs/sglomni/bin/python -m pip install --pre sglang-omni

# 3. A model's companion packages (example: Qwen3-TTS)
app/envs/sglomni/bin/python -m pip install --no-deps sox einops
app/envs/sglomni/bin/python -m pip install --no-deps qwen-tts==0.1.1

# 4. Model weights (example)
app/envs/sglomni/bin/hf download bosonai/higgs-audio-v3-tts-4b

# 5. Start a server manually (the managed flow does this for you)
app/envs/sglomni/bin/sgl-omni serve \
  --model-path bosonai/higgs-audio-v3-tts-4b \
  --port 8100

# 6. Convert
python audiobook.py --backend sglomni --model higgs_audio_v3_tts \
  --api-url http://127.0.0.1:8100
```

Models that take a vendored config file (all but Higgs and ZONOS2) add
`--config app/backends/sglomni/configs/<key>.yaml` to the serve command —
the managed spec builds this from the catalog automatically.

## CLI examples

```bash
# Zero-shot narration with Higgs Audio v3
python audiobook.py --backend sglomni --model higgs_audio_v3_tts

# Voice cloning from a reference clip (transcript transcribed locally)
python audiobook.py --backend sglomni --model moss_tts_local --clone voices/narrator.wav

# Built-in speakers (Qwen3-TTS CustomVoice)
python audiobook.py --backend sglomni --model qwen3_tts_0_6b_customvoice --voice Vivian

# Voice design
python audiobook.py --backend sglomni --model qwen3_tts_1_7b_voicedesign \
  --instructions "A warm adult female narrator with a British accent"
```