diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 60 |
1 files changed, 32 insertions, 28 deletions
@@ -56,6 +56,7 @@ You need to install one of the following backends (see below for installation/us | `--backend {gradio,faster,audiocpp}` | TTS server to use (default `gradio`). `faster` and `audiocpp` require their server running first — see the backend sections above. With `audiocpp` the server may host any audio.cpp TTS model family (see Option 4). | | `--voice <name>` | Voice to request from a server-side voice configuration (`--backend faster` or `audiocpp` only). Required for audio.cpp families without built-in speakers (everything except Qwen3-TTS). | | `--chunk` | Force client-side chunking into `CHUNK_SIZE`-word requests. Only matters for `--backend audiocpp`, which otherwise sends each chapter as one request and lets the server chunk long text itself (may double-chunk); the `gradio` and `faster` backends always chunk. | +| `--model <id>` | `--backend audiocpp` only: audio.cpp server model entry id to use for this run. Overrides `AUDIOCPP_MODEL_ID` in `converter/config.py`, so a server hosting several lazily-loaded models (one `server.json`, see Option 4) can be used without editing config — pick the model per run. Leave unset to use the config id, or to auto-select when the server hosts exactly one entry. | | `--debug` | Troubleshooting: dump each chunk's raw audio and sent text to `debug/` and log every request. | Other options including backend server URLs/ports are configured in `converter/config.py` @@ -164,7 +165,9 @@ python3 tools/model_manager_v2.py install qwen3_tts_1_7b_base_q8_0 python3 tools/model_manager_v2.py install qwen3_tts_1_7b_customvoice_q8_0 ``` -Create a `server.json` file. One server can host multiple models. Note that the `id:` field(s) must match `AUDIOCPP_MODEL_ID` and `AUDIOCPP_CLONE_MODEL_ID` in qwen3_ebook_converter's .`converter/config.py`. Optionally run `tools/make_audiocpp_server_json.py path/to/clone/wavs` to make `server.json` for you with automatic whisper transcription. +Create a `server.json` file. One server can host multiple models. Note that the `id:` field(s) must match `AUDIOCPP_MODEL_ID` and `AUDIOCPP_CLONE_MODEL_ID` in qwen3_ebook_converter's .`converter/config.py`. Optionally run `tools/make_audiocpp_server_json.py path/to/clone/wavs --audiocpp-dir /path/to/audio.cpp` to make `server.json` for you with automatic whisper transcription (see [Generating server.json](#generating-serverjson-with-make_audiocpp_server_json)). + +Cloning voices can be configured per model entry (`voice_presets`) or once at the server level (`voice_dir` + a `prompt_text` file), which every hosted model can clone from. The generator uses the server-level form: ```json { @@ -172,6 +175,7 @@ Create a `server.json` file. One server can host multiple models. Note that the "port": 8080, "backend": "cuda", "lazy_load": false, + "voice_dir": "/path/to/clone/wavs", "models": [ { "id": "qwen", @@ -185,22 +189,21 @@ Create a `server.json` file. One server can host multiple models. Note that the "family": "qwen3_tts", "path": "models/Qwen3-TTS-12Hz-1.7B-Base-GGUF", "task": "tts", - "mode": "offline", - "voice_presets": { - "narrator": { - "voice_ref": "/path/to/reference.wav", - "reference_text": "Transcript of the reference audio." - }, - "obama": { - "voice_ref": "/path/to/reference2.wav", - "reference_text": "Transcript of reference audio." - } - } + "mode": "offline" } ] } ``` +`voice_dir` points at a directory of `.wav` reference files plus a `prompt_text` file with one `<basename>|<transcript>` line per voice: + +``` +narrator|Transcript of the reference audio. +obama|Transcript of reference audio 2. +``` + +A request with `"voice": "narrator"` then clones `voice_dir/narrator.wav` using that transcript. (Per-entry `voice_presets` work too — see audio.cpp's server readme.) + Run the server. The `audiocpp_server` path will be slightly different depending on your platform and build options: ```bash @@ -232,7 +235,7 @@ Supported families (see [audio.cpp's model list](https://github.com/0xShug0/audi | `index_tts2` | IndexTTS-2 | zh, en | Top-tier cloning quality | | `index_tts2` (package `index_tts2_5_*`) | IndexTTS-2.5 | zh, en, ja, es, ar | Multilingual IndexTTS variant | -The converter also works with families not in this table (Fish Audio, Chatterbox, DotTTS, OmniVoice, ...) through its generic profile: clone-only, voice from `--voice`, language detected by the model itself. Anything you can host in `audiocpp_server` with `"task": "tts"` should work. +The converter also works with families not in this table (Fish Audio, Chatterbox, DotTTS, OmniVoice, ...) through its generic profile: clone-only, voice from `--voice`, language detected by the model itself. Anything you can host in `audiocpp_server` with `"task": "tts"` should work, and `make_audiocpp_server_json.py` reads the full catalog from your audio.cpp checkout, so every TTS family audio.cpp supports is offered — not just the ones listed above. ### Install and run @@ -245,7 +248,7 @@ python3 tools/model_manager_v2.py install higgs_audio_tts_4b_q8_0 # or: python3 tools/model_manager_v2.py install index_tts2_5_q8_0 ``` -Create a `server.json` hosting the model plus your cloning voices as `voice_presets`. Note that `id` must match `AUDIOCPP_MODEL_ID` in `converter/config.py` (set `AUDIOCPP_CLONE_MODEL_ID` to the same id — single-model servers use one entry for both): +Create a `server.json` hosting the model. Cloning voices go in a server-level `voice_dir` (a directory of `.wav` files plus a `prompt_text` file — see Option 3) so every hosted model can use them. Note that `id` must match `AUDIOCPP_MODEL_ID` in `converter/config.py` (set `AUDIOCPP_CLONE_MODEL_ID` to the same id — single-model servers use one entry for both): ```json { @@ -253,24 +256,21 @@ Create a `server.json` hosting the model plus your cloning voices as `voice_pres "port": 8080, "backend": "cuda", "lazy_load": false, + "voice_dir": "/path/to/clone/wavs", "models": [ { "id": "higgs", "family": "higgs_audio_tts", "path": "models/Higgs-Audio-v3-TTS-4B-GGUF", "task": "tts", - "mode": "offline", - "voice_presets": { - "narrator": { - "voice_ref": "/path/to/reference.wav", - "reference_text": "Transcript of the reference audio." - } - } + "mode": "offline" } ] } ``` +One `server.json` can host several families at once (add more entries to `models` and set `"lazy_load": true` so each loads only on first use). Then pick the entry per run with `--model <id>` (see below). + Start the server and convert: ```bash @@ -278,26 +278,30 @@ Start the server and convert: # In another terminal python audiobook.py --backend audiocpp --voice narrator +# or, on a multi-model server: +python audiobook.py --backend audiocpp --model higgs --voice narrator ``` VRAM note: the 4B Higgs Audio Q8_0 package needs roughly 2.5x the memory of the 1.7B Qwen3-TTS packages; VoxCPM2-2B and IndexTTS-2 sit in between. BF16/F16 packages roughly double the footprint again. ### Generating server.json with make_audiocpp_server_json -`tools/make_audiocpp_server_json.py` supports the families above directly, including automatic whisper transcription of your reference wavs and updating `converter/config.py` to point at the generated entry: +`tools/make_audiocpp_server_json.py` reads the model catalog (`model_specs/*.json`) from a local audio.cpp checkout and offers every TTS family it supports as a multi-select checklist, so one `server.json` can host several lazily-loaded models. It transcribes your reference wavs with whisper and writes a server-level `voice_dir` + `prompt_text` file automatically. ```bash -# Interactive: pick the family from a menu -python tools/make_audiocpp_server_json.py path/to/clone/wavs +# Interactive: point at your audio.cpp checkout and pick families from a checklist +python tools/make_audiocpp_server_json.py path/to/clone/wavs --audiocpp-dir /path/to/audio.cpp -# Fully specified: Higgs Audio with wavs transcribed into voice presets +# Non-interactive: host Higgs Audio + VoxCPM2 in one lazily-loaded server python tools/make_audiocpp_server_json.py path/to/clone/wavs \ - --family higgs_audio_tts --model-id higgs \ - --model-path models/Higgs-Audio-v3-TTS-4B-GGUF \ + --audiocpp-dir /path/to/audio.cpp \ + --families higgs_audio_tts,voxcpm2 \ --backend cuda --output server.json --force ``` -The tool offers to rewrite `AUDIOCPP_MODEL_ID`/`AUDIOCPP_CLONE_MODEL_ID` in `converter/config.py` to the new entry id so `audiobook.py` talks to it without manual editing (answer "y" at the prompt). As with the Qwen flow, transcripts matter a lot for cloning quality — fill in any empty `reference_text` fields by hand before starting the server. +The checkout can also be auto-detected (an `audio.cpp` directory next to/above your working directory, or the `AUDIOCPP_DIR` environment variable), so `--audiocpp-dir` is optional when you run from there. Pressing Enter at the checklist selects the default Qwen3-TTS flow (built-in speakers + cloning); otherwise enter comma-separated numbers for any combination of families. With more than one family the tool defaults to `"lazy_load": true` (models load on first use and stay in memory until the server exits — restart the server, or `POST /v1/tasks/unload_models`, before switching to a large model to free VRAM). + +For a single hosted entry the tool offers to rewrite `AUDIOCPP_MODEL_ID`/`AUDIOCPP_CLONE_MODEL_ID` in `converter/config.py` to the new id so `audiobook.py` talks to it without manual editing. With several entries it instead prints the available ids — pick one per run with `--model` (or set `AUDIOCPP_MODEL_ID`). Transcripts matter a lot for cloning quality — fill in any empty lines in `prompt_text` by hand before starting the server. ### Language handling |