1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
|
# SGLang-Omni Backend
`--backend sglomni` talks to an [SGLang-Omni](https://github.com/sgl-project/sglang-omni)
server (`sgl-omni serve`), which hosts **one TTS model per server process** behind the
OpenAI-compatible `POST /v1/audio/speech` endpoint. The backend is installed as a pip
package into its own managed venv (`app/envs/sglomni`) and its models are pre-downloaded
into the standard HuggingFace cache — the same flow as the `qwen` and `faster` backends.
## Requirements
- **Linux with an NVIDIA GPU** (recent driver). SGLang-Omni's serving stack is
CUDA-only: there are no Windows builds, and Apple Silicon is limited to ASR.
The setup wizard warns loudly when `nvidia-smi` does not answer.
- **Compute capability matters per model.** ZONOS2's default pipeline
quantizes its MoE experts to FP8 at load time, and the FP8 Triton kernels
only compile on compute capability 8.9+ (RTX 4090/5090, Hopper). On older
GPUs (Ampere: RTX 30xx, A100) the managed spec automatically launches the
vendored bf16 config (`app/backends/sglomni/configs/zonos2_bf16.yaml`)
instead, which runs ZONOS2 in bf16 at roughly twice the MoE VRAM — the
status line tags the model `zonos2 (bf16 fallback)` and the install/boot
output says so. That config also raises the engine's
`mem_fraction_static` to 0.70: the builder's 0.5 default leaves no room
for the KV cache once the bf16 weights (~11.5 GB) are resident on a
24 GB card, and the server aborts with "Loaded weights leave no GPU
memory for the KV cache". Practical floor: a ~16 GB-class card; all
stages stay colocated on GPU 0.
- **Python 3.10, 3.11 or 3.12** for the backend venv. `sglang-omni` requires
`>=3.10,<3.13` while the app itself is version-agnostic, so the wizard
resolves a compatible interpreter automatically:
1. the app venv's interpreter when it is already 3.10-3.12 (no download);
2. a `python3.12`/`python3.11`/`python3.10` found on `PATH`;
3. otherwise it pip-installs `uv` into the app venv and provisions a managed
standalone CPython 3.12 into `app/envs/pythons` (checksum-verified, no root
required — this is the zero-prerequisites path on hosts like stock Arch).
- **Disk space**: the venv alone is ~10 GB (torch, sglang, flash-attn, flashinfer,
CUDA-13 wheels); each model adds 1-10 GB of weights in `~/.cache/huggingface/hub`.
- `ffmpeg` (already a project prerequisite).
The venv pip-installs `sglang-omni` with `--pre` (its dependency stack includes
prerelease components). The verified version at the time of writing is **0.1.4**.
## Model Catalog
Each catalog entry is one installable model. Both the setup wizard and the
hub's **Configure Backends… → Configure SGLang-Omni** screen use the same
checkbox tree (grouped by upstream org, like the audio.cpp one), with the
already-installed models pre-checked: checking a model installs it — running
its companion-package recipe, then pre-downloading its weights with the
venv's `hf` CLI (resumable, cancelable) — and unchecking one removes its
cached weights after a confirm (a managed server hosting that model is
stopped first). The whole diff runs as one task-view pass, removals before
downloads, and the tree re-opens afterwards reflecting the state on disk.
Models can also be left uninstalled — the first server start for one fetches
its weights implicitly, but a pre-download keeps the managed server's boot
inside the start timeout.
| Catalog key | Model | Voice | Notes |
| --- | --- | --- | --- |
| `qwen3_tts_0_6b_customvoice` | Qwen3-TTS 0.6B CustomVoice | built-in speakers | lightest model |
| `qwen3_tts_0_6b_base` | Qwen3-TTS 0.6B Base | clone (reference required) | |
| `qwen3_tts_1_7b_base` | Qwen3-TTS 1.7B Base | clone (reference required) | higher quality |
| `qwen3_tts_1_7b_voicedesign` | Qwen3-TTS 1.7B VoiceDesign | `--instructions` | |
| `higgs_audio_v3_tts` | Higgs Audio v3 TTS | default voice or clone | no config file needed |
| `moss_tts` | MOSS-TTS v1.5 | clone (reference required) | |
| `moss_tts_local` | MOSS-TTS Local v1.5 | default voice or clone | 48 kHz |
| `voxtral_tts` | Voxtral TTS 4B | preset named voices | e.g. `default`, `casual_male` |
| `dots_tts_mf` | dots.tts (MeanFlow) | clone (reference required) | |
| `fish_s2_pro` | Fish Speech S2-Pro | default voice or clone | needs ~24 GB VRAM (known OOM on a single RTX 3090, upstream issue #359) |
| `zonos2` | ZONOS2 | clone (reference required) | 44.1 kHz; FP8 pipeline falls back to bf16 on GPUs below compute capability 8.9; requests carry `max_new_tokens=12288` because the engine's 1024-frame default caps one request at ~12 s of speech |
In the hub's **Generate Audiobooks** form the Model picker reads as a table,
like the audio.cpp one: each entry's label is padded to the widest one and its
capabilities are rendered as fixed columns so every capability word lines up
down its own column — `tts` in the first column (entries that voice plain
text with a preset or built-in default voice), `clone` in the second (the
entry clones a reference clip), `design` in the third (Qwen3-TTS VoiceDesign).
Clone entries that also narrate without a reference (Higgs Audio v3, MOSS-TTS
Local, Fish Speech S2-Pro) carry both `tts clone`.
### Companion packages
Some models need extra packages in the backend venv before their server
starts; installing a model through this tool runs its recipe automatically.
| Model family | Companion packages (as upstream instructs) | System packages |
| --- | --- | --- |
| Qwen3-TTS (all four) | `sox`, `einops`, `qwen-tts==0.1.1` — all with `--no-deps`: the qwen-tts demo pins Transformers 4, which would replace sglang-omni's pinned 5.x stack (sglang-omni shims the API differences) | the `sox` **binary** (e.g. `sudo pacman -S sox`, `sudo apt install sox`) |
| Fish Speech S2-Pro, ZONOS2 | `descript-audiotools==0.7.2`, `descript-audio-codec==1.0.0` (Descript DAC codec) | `ffmpeg` on the server's `PATH` |
A missing system package never blocks a download — the wizard prints the
remediation and the weights install anyway — but the server will fail to
synthesize with that model until the package is present.
## How voice cloning works
Unlike `audio.cpp` (server-side voice presets) the reference clip travels
**with each request** as `ref_audio` + `ref_text`:
- The transcript comes from `--transcription`, or a local Whisper
transcription of the clip (the qwen backend's flow). Without any
transcript the request is sent without `ref_text`, which lowers quality
for models that use it.
- On a locally-managed (or loopback `--api-url`) server the clip is sent as
its **local path** — the server reads the file directly.
- Against a remote server the clip is **inlined as a base64 data URL**, so
`--api-url` runs need no server-side file setup.
Reference clips live in the project's `voices/` directory (10-20 seconds of
clean speech recommended).
## Server lifecycle
The hub and the CLI start and stop the managed instance around each run
(like every backend): without `--api-url` the CLI boots the selected model's
server, converts, and stops it again; a server already answering at the
configured endpoint is used as-is and left running. Because one process
hosts one model, a run whose selected model differs from the hosted one
restarts a server this tool started — a foreign server hosting another
model refuses the run with an actionable message instead.
The hub's **Start/Stop Backend Servers** menu asks which *downloaded* model
to load when starting the server fresh (with none downloaded the entry says
so and points at the Configure screen; a single model starts without
asking). A start also auto-installs a model's missing companion packages
first, so a boot never dies on their import even when the weights arrived
via the shared HuggingFace cache or an earlier install's pip run failed.
The server boots a multi-stage pipeline (preprocessing → TTS generation →
vocoder) and may pull companion weights on first start, so its start
timeout is larger than the other backends' (20 minutes). Pre-downloading
models keeps cold boots well inside it.
External servers: point the **Settings → SGLang-Omni remote URL** (or the
CLI's `--api-url`) at an `sgl-omni` instance. The hub discovers it via
`GET /health` (`{"status": "healthy", "stages": [...]}`) and lists its
model from `GET /v1/models`; models uploaded to that server via
`POST /v1/audio/voices` appear in its Voice menu.
## Manual setup
```bash
# 1. A compatible venv (any of: system python3.10-3.12, or uv-managed)
uv venv --seed --python 3.12 app/envs/sglomni
# 2. The package (into that venv)
app/envs/sglomni/bin/python -m pip install --pre sglang-omni
# 3. A model's companion packages (example: Qwen3-TTS)
app/envs/sglomni/bin/python -m pip install --no-deps sox einops
app/envs/sglomni/bin/python -m pip install --no-deps qwen-tts==0.1.1
# 4. Model weights (example)
app/envs/sglomni/bin/hf download bosonai/higgs-audio-v3-tts-4b
# 5. Start a server manually (the managed flow does this for you)
app/envs/sglomni/bin/sgl-omni serve \
--model-path bosonai/higgs-audio-v3-tts-4b \
--port 8100
# 6. Convert
python audiobook.py --backend sglomni --model higgs_audio_v3_tts \
--api-url http://127.0.0.1:8100
```
Models that take a vendored config file (all but Higgs and ZONOS2) add
`--config app/backends/sglomni/configs/<key>.yaml` to the serve command —
the managed spec builds this from the catalog automatically.
## CLI examples
```bash
# Zero-shot narration with Higgs Audio v3
python audiobook.py --backend sglomni --model higgs_audio_v3_tts
# Voice cloning from a reference clip (transcript transcribed locally)
python audiobook.py --backend sglomni --model moss_tts_local --clone voices/narrator.wav
# Built-in speakers (Qwen3-TTS CustomVoice)
python audiobook.py --backend sglomni --model qwen3_tts_0_6b_customvoice --voice Vivian
# Voice design
python audiobook.py --backend sglomni --model qwen3_tts_1_7b_voicedesign \
--instructions "A warm adult female narrator with a British accent"
```
|
