feat: gplang-omni reserves 70% of gpu 0 memory, preventing oom errors on consumer cards

1 files changed, 27 insertions, 0 deletions

diff --git a/app/docs/backend-sglomni.md b/app/docs/backend-sglomni.md
index 58e5ce5..9091ebd 100644
--- a/app/docs/backend-sglomni.md
+++ b/app/docs/backend-sglomni.md
@@ -170,6 +170,33 @@ Find and stop the process holding the port
 (`ss -tlnp 'sport = :8100'`), or move this server to a free port
 (Settings → SGLang-Omni port), then start again.
 
+**The first request dies with `HTTP 500: CUDA out of memory. Tried to
+allocate ~100 MiB` (and every retry fails identically) on a 24 GB card,
+with the model far smaller than the card.** The model size is not the
+problem: upstream's Qwen3-TTS, MOSS-TTS and Voxtral pipelines colocate
+their AR engine and vocoder on GPU 0 with the engine's sglang
+`mem_fraction_static` unset, so the static pool (weights + KV cache) is
+auto-sized to nearly all free VRAM at boot. By the time a real request
+runs, the engine's CUDA graphs (~1.5 GB) and the colocated vocoder are
+resident too and only tens of MiB are left — the transient allocations
+generation needs do not fit. The vendored configs for those models pin
+the pool at `mem_fraction_static: 0.70` (~7 GB of headroom on a 24 GB
+card) exactly for this; Higgs (`gpu_memory_fraction: 0.80`) and ZONOS2's
+bf16 fallback (`0.70`) carry the same treatment. If you hit the failure
+on a server started by hand or an older checkout, pin the budget on the
+command line and let this tool use the running server as-is:
+
+```bash
+app/envs/sglomni/bin/sgl-omni serve \
+  --model-path Qwen/Qwen3-TTS-12Hz-0.6B-Base \
+  --config app/backends/sglomni/configs/qwen3_tts_0_6b.yaml \
+  --port 8100 --mem-fraction-static 0.70
+```
+
+(MOSS-TTS Local needs none of this: its upstream config already budgets
+its colocated stages explicitly. Fish Speech S2-Pro's OOM is a different
+problem — the model itself needs ~24 GB, upstream issue #359.)
+
 ## Manual setup
 
 ```bash