diff options
Diffstat (limited to 'app/backends')
| -rw-r--r-- | app/backends/sglomni/configs/moss_tts.yaml | 20 |
1 files changed, 15 insertions, 5 deletions
diff --git a/app/backends/sglomni/configs/moss_tts.yaml b/app/backends/sglomni/configs/moss_tts.yaml index 41bddd5..1de9a4c 100644 --- a/app/backends/sglomni/configs/moss_tts.yaml +++ b/app/backends/sglomni/configs/moss_tts.yaml @@ -11,14 +11,24 @@ # fix; note moss_tts_local does NOT need this — MossTTSLocalPipelineConfig # already budgets its colocated stages explicitly: 0.15/0.67/0.18). # -# 0.70 pins the static pool at ~70% of the card (~16.5 GB on a 24 GB GPU — -# a KV pool far larger than any narration request needs) and leaves ~7 GB -# for the vocoder, CUDA graphs, transient allocations, and other GPU -# processes. Precedent: dots_tts.yaml pins this same knob. +# Unlike those models, MOSS cannot take the 0.70 pin. The v1.5 AR weights +# alone are ~17.1 GB in bf16, and on a 24 GB card the engine's profiler only +# sees ~20.7 GB free at that point — the colocated preprocessing stage +# (MOSS-Audio-Tokenizer) is already resident. The KV budget is computed as +# free_after_weights minus pre_model_load_memory * (1 - mem_fraction_static), +# so 0.70 charges a negative KV budget and the boot aborts with "Loaded +# weights leave no GPU memory for the KV cache" (floor ~= 0.828 on a clean +# 24 GB card, and higher with any other GPU process on the card). 0.87 +# leaves a ~0.8 GB KV pool — far larger than any narration sub-request +# needs — with ~2.7 GB of slack for the CUDA graphs, the colocated vocoder, +# and transient allocations. If the first requests OOM anyway, drop to +# 0.86; on a guaranteed-headless card 0.88 buys a ~1 GB KV pool. The +# smaller pinned models (qwen3_tts, voxtral, zonos2 bf16 — weights at or +# under ~11.5 GB) keep 0.70. config_cls: MossTTSPipelineConfig model_path: OpenMOSS-Team/MOSS-TTS-v1.5 stages: tts_engine: engine: - mem_fraction_static: 0.70 + mem_fraction_static: 0.87 |