feat: experimental support for non-qwen models

1 files changed, 93 insertions, 6 deletions

diff --git a/README.md b/README.md
index b9d3057..9515272 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
 
 Convert TXT, PDF, and EPUB files into audiobooks using the Qwen3-TTS voice model.
 
-This builds upon [WhiskeyCoder/Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) adding more output files, support for faster backends, metadata, generated cover art, transcription/speed/language options, better text cleanup, and clearer instructions. The converter supports three TTS backends, selected with `--backend`: the original Qwen3-TTS Gradio demos (`gradio`, default), [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) (`faster`), and [audio.cpp](https://github.com/0xShug0/audio.cpp) (`audiocpp`) — all serving the same Qwen3-TTS 1.7B model.
+This builds upon [WhiskeyCoder/Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) adding more output files, support for faster backends, metadata, generated cover art, transcription/speed/language options, better text cleanup, and clearer instructions. The converter supports three TTS backends, selected with `--backend`: the original Qwen3-TTS Gradio demos (`gradio`, default), [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) (`faster`), and [audio.cpp](https://github.com/0xShug0/audio.cpp) (`audiocpp`) — the last of which can serve either the same Qwen3-TTS 1.7B model (Option 3) or any of audio.cpp's larger non-Qwen TTS families like Higgs Audio v3 4B, VoxCPM2, and IndexTTS-2/2.5 (Option 4).
 
 ## Overview
 
@@ -10,7 +10,7 @@ The converter sends text extracted from your books to a locally running Qwen3-TT
 
 - Input: `.txt`, `.pdf`, or `.epub`
 - Output: `.m4b`, `.mp3`, `.ogg`, or `.flac`
-- Supports [qwen-tts](https://pypi.org/project/qwen-tts/), [faster-qwen-tts](https://github.com/andimarafioti/faster-qwen3-tts), and [audio.cpp](https://github.com/0xShug0/audio.cpp) backend servers
+- Supports [qwen-tts](https://pypi.org/project/qwen-tts/), [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts), and [audio.cpp](https://github.com/0xShug0/audio.cpp) backend servers (audio.cpp can host any of its TTS model families, not just Qwen3-TTS)
 - Output a single file or one per chapter
 - Automatic metadata (title/artist/album tags, chapter track numbers) and a generated cover
 - Clone voices from .wav reference files or use the built-in speaker in the CustomVoice model.
@@ -39,7 +39,8 @@ You will also need to install one of the following backends (see below for insta
 | -------------------------------------------------------------------- | ------------------------------------------------- |
 | [Qwen-TTS](https://pypi.org/project/qwen-tts/)                       | Gradio server released by Qwen                    |
 | [Faster-Qwen-TTS](https://github.com/andimarafioti/faster-qwen3-tts) | Server with 2-8x faster inference for NVidia GPUs |
-| [audio.cpp](https://github.com/0xShug0/audio.cpp)                    | Newer C++ TTS backend that supports Qwen-TTS      |
+| [audio.cpp](https://github.com/0xShug0/audio.cpp) (Qwen)             | Newer C++ TTS backend that supports Qwen-TTS      |
+| [audio.cpp](https://github.com/0xShug0/audio.cpp) (other families)   | Same backend hosting larger/higher-quality models |
 
 ## Options
 
@@ -52,8 +53,8 @@ You will also need to install one of the following backends (see below for insta
 | `--speed <n>`                 | Playback speed, pitch-preserving (`1.0` = normal). A normal-speed copy is also output.           |
 | `--single-file`               | Merge all chapters into a single file (default: one file per chapter). `m4b` is always one file. |
 | `--language <lang>`           | Output language for the synthesized speech. Can add an accent even if the text is English.       |
-| `--backend {gradio,faster,audiocpp}` | TTS server to use (default `gradio`). `faster` and `audiocpp` require their server running first — see the backend sections above. |
-| `--voice <name>` | Voice to request from a server-side voice configuration (`--backend faster` or `audiocpp` only). |
+| `--backend {gradio,faster,audiocpp}` | TTS server to use (default `gradio`). `faster` and `audiocpp` require their server running first — see the backend sections above. With `audiocpp` the server may host any audio.cpp TTS model family (see Option 4). |
+| `--voice <name>` | Voice to request from a server-side voice configuration (`--backend faster` or `audiocpp` only). Required for audio.cpp families without built-in speakers (everything except Qwen3-TTS). |
 | `--chunk`                     | Force client-side chunking into `CHUNK_SIZE`-word requests. Only matters for `--backend audiocpp`, which otherwise sends each chapter as one request and lets the server chunk long text itself (may double-chunk); the `gradio` and `faster` backends always chunk.    |
 | `--debug`                     | Troubleshooting: dump each chunk's raw audio and sent text to `debug/` and log every request.    |
 
@@ -146,7 +147,7 @@ Then from another terminal, run audiobook.py with `--backend faster`
 python audiobook.py --backend faster [--voice NAME]
 ```
 
-## Backend Option 3: audio.cpp
+## Backend Option 3: audio.cpp with Qwen3-TTS
 
 Build `audiocpp_server` for your platform and backend `(cuda, vulkan, hip, cpu)`. Check [audio.cpp's readme](https://github.com/0xShug0/audio.cpp) for details. I'm using one of the helper scripts:
 
@@ -216,6 +217,92 @@ python audiobook.py --backend audiocpp
 python audiobook.py --backend audiocpp --voice narrator
 ```
 
+## Backend Option 4: audio.cpp with non-Qwen models
+
+The same `audiocpp_server` can host most of audio.cpp's other TTS model families, including models that are larger or higher quality than Qwen3-TTS 1.7B. The converter detects the model family from the server at startup and adapts its requests automatically (language codes, style instructions, etc.), so no other converter settings change: point `AUDIOCPP_MODEL_ID` at the entry you want, start the server, and convert with `--backend audiocpp --voice <name>`.
+
+One difference from Qwen3-TTS: **all of these families are clone-only** — they have no built-in speakers, so a reference voice must be configured on the server and selected with `--voice`. Running without `--voice` fails fast with a hint instead of synthesizing a random voice.
+
+Supported families (see [audio.cpp's model list](https://github.com/0xShug0/audio.cpp#supported-models) for the full catalog):
+
+| Family                                      | Model                 | Languages                | Notes                                            |
+| ------------------------------------------- | --------------------- | ------------------------ | ------------------------------------------------ |
+| `higgs_audio_tts`                           | Higgs Audio v3 TTS 4B | 100+                     | Largest TTS in audio.cpp; expressive, inline emotion/style control |
+| `voxcpm2`                                   | VoxCPM2-2B            | 29 listed                | 48 kHz output (others are 24 kHz); cloning + "ultimate clone" (audio + transcript) |
+| `index_tts2`                                | IndexTTS-2            | zh, en                   | Top-tier cloning quality                         |
+| `index_tts2` (package `index_tts2_5_*`)     | IndexTTS-2.5          | zh, en, ja, es, ar       | Multilingual IndexTTS variant                    |
+
+The converter also works with families not in this table (Fish Audio, Chatterbox, DotTTS, OmniVoice, ...) through its generic profile: clone-only, voice from `--voice`, language detected by the model itself. Anything you can host in `audiocpp_server` with `"task": "tts"` should work.
+
+### Install and run
+
+Build `audiocpp_server` exactly as in Option 3 (same binary), then download a model package with audio.cpp's model manager from the audio.cpp checkout:
+
+```bash
+python3 tools/model_manager_v2.py install higgs_audio_tts_4b_q8_0
+# or: python3 tools/model_manager_v2.py install voxcpm2_q8_0
+# or: python3 tools/model_manager_v2.py install index_tts2_q8_0
+# or: python3 tools/model_manager_v2.py install index_tts2_5_q8_0
+```
+
+Create a `server.json` hosting the model plus your cloning voices as `voice_presets`. Note that `id` must match `AUDIOCPP_MODEL_ID` in `converter/config.py` (set `AUDIOCPP_CLONE_MODEL_ID` to the same id — single-model servers use one entry for both):
+
+```json
+{
+  "host": "127.0.0.1",
+  "port": 8080,
+  "backend": "cuda",
+  "lazy_load": false,
+  "models": [
+    {
+      "id": "higgs",
+      "family": "higgs_audio_tts",
+      "path": "models/Higgs-Audio-v3-TTS-4B-GGUF",
+      "task": "tts",
+      "mode": "offline",
+      "voice_presets": {
+        "narrator": {
+          "voice_ref": "/path/to/reference.wav",
+          "reference_text": "Transcript of the reference audio."
+        }
+      }
+    }
+  ]
+}
+```
+
+Start the server and convert:
+
+```bash
+./build/linux-cuda-release/bin/audiocpp_server --config server.json
+
+# In another terminal
+python audiobook.py --backend audiocpp --voice narrator
+```
+
+VRAM note: the 4B Higgs Audio Q8_0 package needs roughly 2.5x the memory of the 1.7B Qwen3-TTS packages; VoxCPM2-2B and IndexTTS-2 sit in between. BF16/F16 packages roughly double the footprint again.
+
+### Generating server.json with make_audiocpp_server_json
+
+`tools/make_audiocpp_server_json.py` supports the families above directly, including automatic whisper transcription of your reference wavs and updating `converter/config.py` to point at the generated entry:
+
+```bash
+# Interactive: pick the family from a menu
+python tools/make_audiocpp_server_json.py path/to/clone/wavs
+
+# Fully specified: Higgs Audio with wavs transcribed into voice presets
+python tools/make_audiocpp_server_json.py path/to/clone/wavs \
+    --family higgs_audio_tts --model-id higgs \
+    --model-path models/Higgs-Audio-v3-TTS-4B-GGUF \
+    --backend cuda --output server.json --force
+```
+
+The tool offers to rewrite `AUDIOCPP_MODEL_ID`/`AUDIOCPP_CLONE_MODEL_ID` in `converter/config.py` to the new entry id so `audiobook.py` talks to it without manual editing (answer "y" at the prompt). As with the Qwen flow, transcripts matter a lot for cloning quality — fill in any empty `reference_text` fields by hand before starting the server.
+
+### Language handling
+
+`--language` works with these families too, adapted per family: IndexTTS sends a language code (`English` → `en`), while Higgs Audio and VoxCPM2 detect the language from the text themselves and omit the field. `--language Auto` never sends a language.
+
 ## Optional: FlashAttention for qwen-tts-demo server
 
 FlashAttention provides a small speed boost on the `qwen-tts-demo` backend. It is **not** relevant with the `faster` or `audiocpp` backends, and switching to either of those will provide a bigger speed boost.