diff options
Diffstat (limited to 'app/docs/backend-sglomni.md')
| -rw-r--r-- | app/docs/backend-sglomni.md | 177 |
1 files changed, 177 insertions, 0 deletions
diff --git a/app/docs/backend-sglomni.md b/app/docs/backend-sglomni.md new file mode 100644 index 0000000..13c8876 --- /dev/null +++ b/app/docs/backend-sglomni.md @@ -0,0 +1,177 @@ +# SGLang-Omni Backend + +`--backend sglomni` talks to an [SGLang-Omni](https://github.com/sgl-project/sglang-omni) +server (`sgl-omni serve`), which hosts **one TTS model per server process** behind the +OpenAI-compatible `POST /v1/audio/speech` endpoint. The backend is installed as a pip +package into its own managed venv (`app/envs/sglomni`) and its models are pre-downloaded +into the standard HuggingFace cache — the same flow as the `qwen` and `faster` backends. + +## Requirements + +- **Linux with an NVIDIA GPU** (recent driver). SGLang-Omni's serving stack is + CUDA-only: there are no Windows builds, and Apple Silicon is limited to ASR. + The setup wizard warns loudly when `nvidia-smi` does not answer. +- **Compute capability matters per model.** ZONOS2's default pipeline + quantizes its MoE experts to FP8 at load time, and the FP8 Triton kernels + only compile on compute capability 8.9+ (RTX 4090/5090, Hopper). On older + GPUs (Ampere: RTX 30xx, A100) the managed spec automatically launches the + vendored bf16 config (`app/backends/sglomni/configs/zonos2_bf16.yaml`) + instead, which runs ZONOS2 in bf16 at roughly twice the MoE VRAM — the + status line tags the model `zonos2 (bf16 fallback)` and the install/boot + output says so. That config also raises the engine's + `mem_fraction_static` to 0.70: the builder's 0.5 default leaves no room + for the KV cache once the bf16 weights (~11.5 GB) are resident on a + 24 GB card, and the server aborts with "Loaded weights leave no GPU + memory for the KV cache". Practical floor: a ~16 GB-class card; all + stages stay colocated on GPU 0. +- **Python 3.10, 3.11 or 3.12** for the backend venv. `sglang-omni` requires + `>=3.10,<3.13` while the app itself is version-agnostic, so the wizard + resolves a compatible interpreter automatically: + 1. the app venv's interpreter when it is already 3.10-3.12 (no download); + 2. a `python3.12`/`python3.11`/`python3.10` found on `PATH`; + 3. otherwise it pip-installs `uv` into the app venv and provisions a managed + standalone CPython 3.12 into `app/envs/pythons` (checksum-verified, no root + required — this is the zero-prerequisites path on hosts like stock Arch). +- **Disk space**: the venv alone is ~10 GB (torch, sglang, flash-attn, flashinfer, + CUDA-13 wheels); each model adds 1-10 GB of weights in `~/.cache/huggingface/hub`. +- `ffmpeg` (already a project prerequisite). + +The venv pip-installs `sglang-omni` with `--pre` (its dependency stack includes +prerelease components). The verified version at the time of writing is **0.1.4**. + +## Model Catalog + +Each catalog entry is one installable model. Both the setup wizard and the +hub's **Configure Backends… → Configure SGLang-Omni** screen use the same +checkbox tree (grouped by upstream org, like the audio.cpp one), with the +already-installed models pre-checked: checking a model installs it — running +its companion-package recipe, then pre-downloading its weights with the +venv's `hf` CLI (resumable, cancelable) — and unchecking one removes its +cached weights after a confirm (a managed server hosting that model is +stopped first). The whole diff runs as one task-view pass, removals before +downloads, and the tree re-opens afterwards reflecting the state on disk. +Models can also be left uninstalled — the first server start for one fetches +its weights implicitly, but a pre-download keeps the managed server's boot +inside the start timeout. + +| Catalog key | Model | Voice | Notes | +| --- | --- | --- | --- | +| `qwen3_tts_0_6b_customvoice` | Qwen3-TTS 0.6B CustomVoice | built-in speakers | lightest model | +| `qwen3_tts_0_6b_base` | Qwen3-TTS 0.6B Base | clone (reference required) | | +| `qwen3_tts_1_7b_base` | Qwen3-TTS 1.7B Base | clone (reference required) | higher quality | +| `qwen3_tts_1_7b_voicedesign` | Qwen3-TTS 1.7B VoiceDesign | `--instructions` | | +| `higgs_audio_v3_tts` | Higgs Audio v3 TTS | default voice or clone | no config file needed | +| `moss_tts` | MOSS-TTS v1.5 | clone (reference required) | | +| `moss_tts_local` | MOSS-TTS Local v1.5 | default voice or clone | 48 kHz | +| `voxtral_tts` | Voxtral TTS 4B | preset named voices | e.g. `default`, `casual_male` | +| `dots_tts_mf` | dots.tts (MeanFlow) | clone (reference required) | | +| `fish_s2_pro` | Fish Speech S2-Pro | default voice or clone | needs ~24 GB VRAM (known OOM on a single RTX 3090, upstream issue #359) | +| `zonos2` | ZONOS2 | clone (reference required) | 44.1 kHz; FP8 pipeline falls back to bf16 on GPUs below compute capability 8.9 | + +In the hub's **Generate Audiobooks** form the Model picker reads as a table, +like the audio.cpp one: each entry's label is padded to the widest one and its +capabilities are rendered as fixed columns so every capability word lines up +down its own column — `tts` in the first column (entries that voice plain +text with a preset or built-in default voice), `clone` in the second (the +entry clones a reference clip), `design` in the third (Qwen3-TTS VoiceDesign). +Clone entries that also narrate without a reference (Higgs Audio v3, MOSS-TTS +Local, Fish Speech S2-Pro) carry both `tts clone`. + +### Companion packages + +Some models need extra packages in the backend venv before their server +starts; installing a model through this tool runs its recipe automatically. + +| Model family | Companion packages (as upstream instructs) | System packages | +| --- | --- | --- | +| Qwen3-TTS (all four) | `sox`, `einops`, `qwen-tts==0.1.1` — all with `--no-deps`: the qwen-tts demo pins Transformers 4, which would replace sglang-omni's pinned 5.x stack (sglang-omni shims the API differences) | the `sox` **binary** (e.g. `sudo pacman -S sox`, `sudo apt install sox`) | +| Fish Speech S2-Pro, ZONOS2 | `descript-audiotools==0.7.2`, `descript-audio-codec==1.0.0` (Descript DAC codec) | `ffmpeg` on the server's `PATH` | + +A missing system package never blocks a download — the wizard prints the +remediation and the weights install anyway — but the server will fail to +synthesize with that model until the package is present. + +## How voice cloning works + +Unlike `audio.cpp` (server-side voice presets) the reference clip travels +**with each request** as `ref_audio` + `ref_text`: + +- The transcript comes from `--transcription`, or a local Whisper + transcription of the clip (the qwen backend's flow). Without any + transcript the request is sent without `ref_text`, which lowers quality + for models that use it. +- On a locally-managed (or loopback `--api-url`) server the clip is sent as + its **local path** — the server reads the file directly. +- Against a remote server the clip is **inlined as a base64 data URL**, so + `--api-url` runs need no server-side file setup. + +Reference clips live in the project's `voices/` directory (10-20 seconds of +clean speech recommended). + +## Server lifecycle + +The hub and the CLI start and stop the managed instance around each run +(like every backend): without `--api-url` the CLI boots the selected model's +server, converts, and stops it again; a server already answering at the +configured endpoint is used as-is and left running. Because one process +hosts one model, a run whose selected model differs from the hosted one +restarts a server this tool started — a foreign server hosting another +model refuses the run with an actionable message instead. + +The server boots a multi-stage pipeline (preprocessing → TTS generation → +vocoder) and may pull companion weights on first start, so its start +timeout is larger than the other backends' (20 minutes). Pre-downloading +models keeps cold boots well inside it. + +External servers: point the **Settings → SGLang-Omni remote URL** (or the +CLI's `--api-url`) at an `sgl-omni` instance. The hub discovers it via +`GET /health` (`{"status": "healthy", "stages": [...]}`) and lists its +model from `GET /v1/models`; models uploaded to that server via +`POST /v1/audio/voices` appear in its Voice menu. + +## Manual setup + +```bash +# 1. A compatible venv (any of: system python3.10-3.12, or uv-managed) +uv venv --seed --python 3.12 app/envs/sglomni + +# 2. The package (into that venv) +app/envs/sglomni/bin/python -m pip install --pre sglang-omni + +# 3. A model's companion packages (example: Qwen3-TTS) +app/envs/sglomni/bin/python -m pip install --no-deps sox einops +app/envs/sglomni/bin/python -m pip install --no-deps qwen-tts==0.1.1 + +# 4. Model weights (example) +app/envs/sglomni/bin/hf download bosonai/higgs-audio-v3-tts-4b + +# 5. Start a server manually (the managed flow does this for you) +app/envs/sglomni/bin/sgl-omni serve \ + --model-path bosonai/higgs-audio-v3-tts-4b \ + --port 8100 + +# 6. Convert +python audiobook.py --backend sglomni --model higgs_audio_v3_tts \ + --api-url http://127.0.0.1:8100 +``` + +Models that take a vendored config file (all but Higgs and ZONOS2) add +`--config app/backends/sglomni/configs/<key>.yaml` to the serve command — +the managed spec builds this from the catalog automatically. + +## CLI examples + +```bash +# Zero-shot narration with Higgs Audio v3 +python audiobook.py --backend sglomni --model higgs_audio_v3_tts + +# Voice cloning from a reference clip (transcript transcribed locally) +python audiobook.py --backend sglomni --model moss_tts_local --clone voices/narrator.wav + +# Built-in speakers (Qwen3-TTS CustomVoice) +python audiobook.py --backend sglomni --model qwen3_tts_0_6b_customvoice --voice Vivian + +# Voice design +python audiobook.py --backend sglomni --model qwen3_tts_1_7b_voicedesign \ + --instructions "A warm adult female narrator with a British accent" +``` |