diff options
Diffstat (limited to 'app/docs/backend-sglomni.md')
| -rw-r--r-- | app/docs/backend-sglomni.md | 271 |
1 files changed, 42 insertions, 229 deletions
diff --git a/app/docs/backend-sglomni.md b/app/docs/backend-sglomni.md index 7af6a72..1d5dcda 100644 --- a/app/docs/backend-sglomni.md +++ b/app/docs/backend-sglomni.md @@ -1,257 +1,70 @@ -# SGLang-Omni Backend +# SGLang-Omni (`--backend sglomni`) -`--backend sglomni` talks to an [SGLang-Omni](https://github.com/sgl-project/sglang-omni) -server (`sgl-omni serve`), which hosts **one TTS model per server process** behind the -OpenAI-compatible `POST /v1/audio/speech` endpoint. The backend is installed as a pip -package into its own managed venv (`app/envs/sglomni`) and its models are pre-downloaded -into the standard HuggingFace cache — the same flow as the `qwen` and `faster` backends. +Runs an [SGLang-Omni](https://github.com/sgl-project/sglang-omni) server (`sgl-omni serve`), one TTS model per server process. Installs into its own managed venv, `app/envs/sglomni`. -## Requirements - -- **Linux with an NVIDIA GPU** (recent driver). SGLang-Omni's serving stack is - CUDA-only: there are no Windows builds, and Apple Silicon is limited to ASR. - The setup wizard warns loudly when `nvidia-smi` does not answer. -- **Compute capability matters per model.** ZONOS2's default pipeline - quantizes its MoE experts to FP8 at load time, and the FP8 Triton kernels - only compile on compute capability 8.9+ (RTX 4090/5090, Hopper). On older - GPUs (Ampere: RTX 30xx, A100) the managed spec automatically launches the - vendored bf16 config (`app/backends/sglomni/configs/zonos2_bf16.yaml`) - instead, which runs ZONOS2 in bf16 at roughly twice the MoE VRAM — the - status line tags the model `zonos2 (bf16 fallback)` and the install/boot - output says so. That config also raises the engine's - `mem_fraction_static` to 0.70: the builder's 0.5 default leaves no room - for the KV cache once the bf16 weights (~11.5 GB) are resident on a - 24 GB card, and the server aborts with "Loaded weights leave no GPU - memory for the KV cache". Practical floor: a ~16 GB-class card; all - stages stay colocated on GPU 0. -- **Python 3.10, 3.11 or 3.12** for the backend venv. `sglang-omni` requires - `>=3.10,<3.13` while the app itself is version-agnostic, so the wizard - resolves a compatible interpreter automatically: - 1. the app venv's interpreter when it is already 3.10-3.12 (no download); - 2. a `python3.12`/`python3.11`/`python3.10` found on `PATH`; - 3. otherwise it pip-installs `uv` into the app venv and provisions a managed - standalone CPython 3.12 into `app/envs/pythons` (checksum-verified, no root - required — this is the zero-prerequisites path on hosts like stock Arch). -- **Disk space**: the venv alone is ~10 GB (torch, sglang, flash-attn, flashinfer, - CUDA-13 wheels); each model adds 1-10 GB of weights in `~/.cache/huggingface/hub`. -- `ffmpeg` (already a project prerequisite). - -The venv pip-installs `sglang-omni` with `--pre` (its dependency stack includes -prerelease components). The verified version at the time of writing is **0.1.4**. +Run all commands from the project root (next to `audiobook.py`). -## Model Catalog +## Requirements -Each catalog entry is one installable model. Both the setup wizard and the -hub's **Configure Backends… → Configure SGLang-Omni** screen use the same -checkbox tree (grouped by upstream org, like the audio.cpp one), with the -already-installed models pre-checked: checking a model installs it — running -its companion-package recipe, then pre-downloading its weights with the -venv's `hf` CLI (resumable, cancelable) — and unchecking one removes its -cached weights after a confirm (a managed server hosting that model is -stopped first). The whole diff runs as one task-view pass, removals before -downloads, and the tree re-opens afterwards reflecting the state on disk. -Models can also be left uninstalled — the first server start for one fetches -its weights implicitly, but a pre-download keeps the managed server's boot -inside the start timeout. +- Linux with an NVIDIA GPU +- Python 3.10-3.12 (`sglang-omni` requires `>=3.10,<3.13`) +- ~10 GB disk for the venv plus 1-10 GB per model of weights in `~/.cache/huggingface/hub` -| Catalog key | Model | Voice | Notes | -| --- | --- | --- | --- | -| `qwen3_tts_0_6b_customvoice` | Qwen3-TTS 0.6B CustomVoice | built-in speakers | lightest model | -| `qwen3_tts_0_6b_base` | Qwen3-TTS 0.6B Base | clone (reference required) | | -| `qwen3_tts_1_7b_base` | Qwen3-TTS 1.7B Base | clone (reference required) | higher quality | -| `qwen3_tts_1_7b_voicedesign` | Qwen3-TTS 1.7B VoiceDesign | `--instructions` | | -| `higgs_audio_v3_tts` | Higgs Audio v3 TTS | default voice or clone | launches with a vendored config: VRAM headroom for 24 GB cards + a raised generation cap (requests carry `max_new_tokens=3000` — upstream pins the thinker engine's context length at 4096, so prompt + generation must fit 4095 tokens, ~40 s of speech — and sub-requests cap at 80 words, with a pre-run popup offering the CHUNK_SIZE clamp) | -| `moss_tts` | MOSS-TTS v1.5 | clone (reference required) | | -| `moss_tts_local` | MOSS-TTS Local v1.5 | default voice or clone | 48 kHz | -| `voxtral_tts` | Voxtral TTS 4B | preset named voices | e.g. `default`, `casual_male` | -| `dots_tts_mf` | dots.tts (MeanFlow) | clone (reference required) | | -| `fish_s2_pro` | Fish Speech S2-Pro | default voice or clone | needs ~24 GB VRAM (known OOM on a single RTX 3090, upstream issue #359) | -| `zonos2` | ZONOS2 | clone (reference required) | 44.1 kHz; FP8 pipeline falls back to bf16 on GPUs below compute capability 8.9; requests carry `max_new_tokens=12288` because the engine's 1024-frame default caps one request at ~12 s of speech | +## Install -In the hub's **Generate Audiobooks** form the Model picker reads as a table, -like the audio.cpp one: each entry's label is padded to the widest one and its -capabilities are rendered as fixed columns so every capability word lines up -down its own column — `tts` in the first column (entries that voice plain -text with a preset or built-in default voice), `clone` in the second (the -entry clones a reference clip), `design` in the third (Qwen3-TTS VoiceDesign). -Clone entries that also narrate without a reference (Higgs Audio v3, MOSS-TTS -Local, Fish Speech S2-Pro) carry both `tts clone`. +1. Create the venv and install the server: -### Companion packages +```bash +uv venv --seed --python 3.12 app/envs/sglomni # or: python3.12 -m venv app/envs/sglomni +app/envs/sglomni/bin/python -m pip install --pre sglang-omni +``` -Some models need extra packages in the backend venv before their server -starts; installing a model through this tool runs its recipe automatically. +2. Install companion packages for the models you plan to host: -| Model family | Companion packages (as upstream instructs) | System packages | +| Models | Into `app/envs/sglomni` | System packages | | --- | --- | --- | -| Qwen3-TTS (all four) | `sox`, `einops`, `qwen-tts==0.1.1` — all with `--no-deps`: the qwen-tts demo pins Transformers 4, which would replace sglang-omni's pinned 5.x stack (sglang-omni shims the API differences) | the `sox` **binary** (e.g. `sudo pacman -S sox`, `sudo apt install sox`) | -| Fish Speech S2-Pro, ZONOS2 | `descript-audiotools==0.7.2`, `descript-audio-codec==1.0.0` (Descript DAC codec) | `ffmpeg` on the server's `PATH` | - -A missing system package never blocks a download — the wizard prints the -remediation and the weights install anyway — but the server will fail to -synthesize with that model until the package is present. - -## How voice cloning works - -Unlike `audio.cpp` (server-side voice presets) the reference clip travels -**with each request** as `ref_audio` + `ref_text`: - -- The transcript comes from `--transcription`, or a local Whisper - transcription of the clip (the qwen backend's flow). Without any - transcript the request is sent without `ref_text`, which lowers quality - for models that use it. -- On a locally-managed (or loopback `--api-url`) server the clip is sent as - its **local path** — the server reads the file directly. -- Against a remote server the clip is **inlined as a base64 data URL**, so - `--api-url` runs need no server-side file setup. - -Reference clips live in the project's `voices/` directory (10-20 seconds of -clean speech recommended). - -### Higgs Audio v3's request window - -The Higgs server pins its thinker engine's context length at 4096 tokens -(not configurable), and the scheduler rejects any request whose prompt — -including the reference-audio tokens — plus `max_new_tokens` exceeds that -window ("Request requires more tokens than the thinker KV cache can hold", -on every GPU: the KV pool side is never the binding constraint). One -request can therefore narrate at most ~40 s of speech, so: - -- Requests carry `max_new_tokens=3000` (the vendored config raises the - engine's 2048-frame default to match), and sub-requests cap at 80 words. - Before a run with Higgs, a popup offers to clamp CHUNK_SIZE for the run - ("Set Chunk to 80"), keep the configured size (audio may cut off - mid-chunk), or cancel; CLI runs answer on the console (`[S/t/c]`, and a - closed stdin clamps so unattended runs never truncate silently). -- When a request is still rejected (a long reference clip, for example), - the client refits `max_new_tokens` to the capacity the server reports - and resends once — the same request never retries as-is. -- Voice cloning with a *long* reference clip eats into the same window: - keep references around 10-20 seconds (as recommended above) for the - most usable generation headroom. - -## Server lifecycle - -The hub and the CLI start and stop the managed instance around each run -(like every backend): without `--api-url` the CLI boots the selected model's -server, converts, and stops it again; a server already answering at the -configured endpoint is used as-is and left running. Because one process -hosts one model, a run whose selected model differs from the hosted one -restarts a server this tool started — a foreign server hosting another -model refuses the run with an actionable message instead. +| All four Qwen3-TTS | `bin/python -m pip install --no-deps sox einops qwen-tts==0.1.1` | `sox` binary (e.g. `sudo apt install sox`) | +| Fish Speech S2-Pro, ZONOS2 | `bin/python -m pip install descript-audiotools==0.7.2 descript-audio-codec==1.0.0 protobuf==6.33.6` | `ffmpeg` on PATH | -The hub's **Start/Stop Backend Servers** menu asks which *downloaded* model -to load when starting the server fresh (with none downloaded the entry says -so and points at the Configure screen; a single model starts without -asking). A start also auto-installs a model's missing companion packages -first, so a boot never dies on their import even when the weights arrived -via the shared HuggingFace cache or an earlier install's pip run failed. - -The server boots a multi-stage pipeline (preprocessing → TTS generation → -vocoder) and may pull companion weights on first start, so its start -timeout is larger than the other backends' (20 minutes). Pre-downloading -models keeps cold boots well inside it. - -External servers: point the **Settings → SGLang-Omni remote URL** (or the -CLI's `--api-url`) at an `sgl-omni` instance. The hub discovers it via -`GET /health` (`{"status": "healthy", "stages": [...]}`) and lists its -model from `GET /v1/models`; models uploaded to that server via -`POST /v1/audio/voices` appear in its Voice menu. - -## Troubleshooting - -**The generate screen sits on "Status: starting", and the server log says -`Port 8100 is already in use ... Using port 37183 instead`.** Something -else already holds the configured port (often a stale `sgl-omni` from an -earlier attempt). The upstream launcher does not fail — it silently moves -the server to a random port, where no client ever looks for it. The app -detects this in the boot log, aborts the boot, and names both ports. -Find and stop the process holding the port -(`ss -tlnp 'sport = :8100'`), or move this server to a free port -(Settings → SGLang-Omni port), then start again. - -**The first request dies with `HTTP 500: CUDA out of memory. Tried to -allocate ~100 MiB` (and every retry fails identically) on a 24 GB card, -with the model far smaller than the card.** The model size is not the -problem: upstream's Qwen3-TTS, MOSS-TTS and Voxtral pipelines colocate -their AR engine and vocoder on GPU 0 with the engine's sglang -`mem_fraction_static` unset, so the static pool (weights + KV cache) is -auto-sized to nearly all free VRAM at boot. By the time a real request -runs, the engine's CUDA graphs (~1.5 GB) and the colocated vocoder are -resident too and only tens of MiB are left — the transient allocations -generation needs do not fit. The vendored configs for those models pin -the pool exactly for this — 0.70 for the small ones (~7 GB of headroom -on a 24 GB card); Higgs (`gpu_memory_fraction: 0.80`) and ZONOS2's -bf16 fallback (`0.70`) carry the same treatment. MOSS-TTS is the -exception: its ~17.1 GB of v1.5 bf16 weights alone exceed the 0.70 -budget on the ~20.7 GB the engine profiler sees at that point (its -colocated audio-tokenizer stage is already resident), so a 0.70 pin -aborts the boot outright with "Loaded weights leave no GPU memory for -the KV cache" (the floor is ~0.828 on a clean 24 GB card) instead of -fixing the request-time OOM — it runs `mem_fraction_static: 0.87`, -keeping a ~0.8 GB KV pool with ~2.7 GB of slack for the CUDA graphs -and the colocated vocoder. If you hit the failure -on a server started by hand or an older checkout, pin the budget on the -command line and let this tool use the running server as-is: +3. Download model weights: ```bash -app/envs/sglomni/bin/sgl-omni serve \ - --model-path Qwen/Qwen3-TTS-12Hz-0.6B-Base \ - --config app/backends/sglomni/configs/qwen3_tts_0_6b.yaml \ - --port 8100 --mem-fraction-static 0.70 +app/envs/sglomni/bin/hf download <hf-repo> # repos in the table below ``` -(MOSS-TTS Local needs none of this: its upstream config already budgets -its colocated stages explicitly. Fish Speech S2-Pro's OOM is a different -problem — the model itself needs ~24 GB, upstream issue #359.) +## Models -## Manual setup +| Key (TAG `--model`) | HF repo (`--model-path`) | Config | +| --- | --- | --- | +| `qwen3_tts_0_6b_customvoice` | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | `qwen3_tts_0_6b_customvoice.yaml` | +| `qwen3_tts_0_6b_base` | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | `qwen3_tts_0_6b.yaml` | +| `qwen3_tts_1_7b_base` | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | `qwen3_tts_1_7b.yaml` | +| `qwen3_tts_1_7b_voicedesign` | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | `qwen3_tts_1_7b_voicedesign.yaml` | +| `higgs_audio_v3_tts` | `bosonai/higgs-audio-v3-tts-4b` | `higgs_audio_v3_tts.yaml` | +| `moss_tts` | `OpenMOSS-Team/MOSS-TTS-v1.5` | `moss_tts.yaml` | +| `moss_tts_local` | `OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5` | `moss_tts_local.yaml` | +| `voxtral_tts` | `mistralai/Voxtral-4B-TTS-2603` | `voxtral_tts.yaml` | +| `dots_tts_mf` | `dots-studio/dots.tts-mf` | `dots_tts.yaml` | +| `fish_s2_pro` | `fishaudio/s2-pro` | `s2pro_tts.yaml` | +| `zonos2` | `Zyphra/zonos2` | none | + +Config files live in `app/backends/sglomni/configs/`. ZONOS2 runs without a config; on GPUs below compute capability 8.9 (RTX 30xx, A100) pass `--config app/backends/sglomni/configs/zonos2_bf16.yaml` instead. + +## Run the server ```bash -# 1. A compatible venv (any of: system python3.10-3.12, or uv-managed) -uv venv --seed --python 3.12 app/envs/sglomni - -# 2. The package (into that venv) -app/envs/sglomni/bin/python -m pip install --pre sglang-omni - -# 3. A model's companion packages (example: Qwen3-TTS) -app/envs/sglomni/bin/python -m pip install --no-deps sox einops -app/envs/sglomni/bin/python -m pip install --no-deps qwen-tts==0.1.1 - -# 4. Model weights (example) -app/envs/sglomni/bin/hf download bosonai/higgs-audio-v3-tts-4b - -# 5. Start a server manually (the managed flow does this for you) +# Example: Higgs Audio v3 app/envs/sglomni/bin/sgl-omni serve \ --model-path bosonai/higgs-audio-v3-tts-4b \ --config app/backends/sglomni/configs/higgs_audio_v3_tts.yaml \ --port 8100 -# 6. Convert -python audiobook.py --backend sglomni --model higgs_audio_v3_tts \ - --api-url http://127.0.0.1:8100 +# ZONOS2 (no config) +app/envs/sglomni/bin/sgl-omni serve --model-path Zyphra/zonos2 --port 8100 ``` -Models that take a vendored config file (all but ZONOS2) add -`--config app/backends/sglomni/configs/<key>.yaml` to the serve command — -the managed spec builds this from the catalog automatically. Higgs's config -also exists for a reason: the upstream pipeline budgets 98% of the card -across its colocated stages, which OOMs on a 24 GB GPU once anything else -uses VRAM, and its engine caps requests at ~27 s of speech. - -## CLI examples +## Use it ```bash -# Zero-shot narration with Higgs Audio v3 -python audiobook.py --backend sglomni --model higgs_audio_v3_tts - -# Voice cloning from a reference clip (transcript transcribed locally) -python audiobook.py --backend sglomni --model moss_tts_local --clone voices/narrator.wav - -# Built-in speakers (Qwen3-TTS CustomVoice) -python audiobook.py --backend sglomni --model qwen3_tts_0_6b_customvoice --voice Vivian - -# Voice design -python audiobook.py --backend sglomni --model qwen3_tts_1_7b_voicedesign \ - --instructions "A warm adult female narrator with a British accent" +python audiobook.py --backend sglomni --api-url http://127.0.0.1:8100 --model <key> ``` |