diff options
Diffstat (limited to 'app/docs/backend-sglomni.md')
| -rw-r--r-- | app/docs/backend-sglomni.md | 27 |
1 files changed, 27 insertions, 0 deletions
diff --git a/app/docs/backend-sglomni.md b/app/docs/backend-sglomni.md index 58e5ce5..9091ebd 100644 --- a/app/docs/backend-sglomni.md +++ b/app/docs/backend-sglomni.md @@ -170,6 +170,33 @@ Find and stop the process holding the port (`ss -tlnp 'sport = :8100'`), or move this server to a free port (Settings → SGLang-Omni port), then start again. +**The first request dies with `HTTP 500: CUDA out of memory. Tried to +allocate ~100 MiB` (and every retry fails identically) on a 24 GB card, +with the model far smaller than the card.** The model size is not the +problem: upstream's Qwen3-TTS, MOSS-TTS and Voxtral pipelines colocate +their AR engine and vocoder on GPU 0 with the engine's sglang +`mem_fraction_static` unset, so the static pool (weights + KV cache) is +auto-sized to nearly all free VRAM at boot. By the time a real request +runs, the engine's CUDA graphs (~1.5 GB) and the colocated vocoder are +resident too and only tens of MiB are left — the transient allocations +generation needs do not fit. The vendored configs for those models pin +the pool at `mem_fraction_static: 0.70` (~7 GB of headroom on a 24 GB +card) exactly for this; Higgs (`gpu_memory_fraction: 0.80`) and ZONOS2's +bf16 fallback (`0.70`) carry the same treatment. If you hit the failure +on a server started by hand or an older checkout, pin the budget on the +command line and let this tool use the running server as-is: + +```bash +app/envs/sglomni/bin/sgl-omni serve \ + --model-path Qwen/Qwen3-TTS-12Hz-0.6B-Base \ + --config app/backends/sglomni/configs/qwen3_tts_0_6b.yaml \ + --port 8100 --mem-fraction-static 0.70 +``` + +(MOSS-TTS Local needs none of this: its upstream config already budgets +its colocated stages explicitly. Fish Speech S2-Pro's OOM is a different +problem — the model itself needs ~24 GB, upstream issue #359.) + ## Manual setup ```bash |