diff options
Diffstat (limited to 'app/docs/backend-sglomni.md')
| -rw-r--r-- | app/docs/backend-sglomni.md | 14 |
1 files changed, 11 insertions, 3 deletions
diff --git a/app/docs/backend-sglomni.md b/app/docs/backend-sglomni.md index 9091ebd..7af6a72 100644 --- a/app/docs/backend-sglomni.md +++ b/app/docs/backend-sglomni.md @@ -180,9 +180,17 @@ auto-sized to nearly all free VRAM at boot. By the time a real request runs, the engine's CUDA graphs (~1.5 GB) and the colocated vocoder are resident too and only tens of MiB are left — the transient allocations generation needs do not fit. The vendored configs for those models pin -the pool at `mem_fraction_static: 0.70` (~7 GB of headroom on a 24 GB -card) exactly for this; Higgs (`gpu_memory_fraction: 0.80`) and ZONOS2's -bf16 fallback (`0.70`) carry the same treatment. If you hit the failure +the pool exactly for this — 0.70 for the small ones (~7 GB of headroom +on a 24 GB card); Higgs (`gpu_memory_fraction: 0.80`) and ZONOS2's +bf16 fallback (`0.70`) carry the same treatment. MOSS-TTS is the +exception: its ~17.1 GB of v1.5 bf16 weights alone exceed the 0.70 +budget on the ~20.7 GB the engine profiler sees at that point (its +colocated audio-tokenizer stage is already resident), so a 0.70 pin +aborts the boot outright with "Loaded weights leave no GPU memory for +the KV cache" (the floor is ~0.828 on a clean 24 GB card) instead of +fixing the request-time OOM — it runs `mem_fraction_static: 0.87`, +keeping a ~0.8 GB KV pool with ~2.7 GB of slack for the CUDA graphs +and the colocated vocoder. If you hit the failure on a server started by hand or an older checkout, pin the budget on the command line and let this tool use the running server as-is: |