diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 99 |
1 files changed, 93 insertions, 6 deletions
@@ -2,7 +2,7 @@ Convert TXT, PDF, and EPUB files into audiobooks using the Qwen3-TTS voice model. -This builds upon [WhiskeyCoder/Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) adding more output files, support for faster backends, metadata, generated cover art, transcription/speed/language options, better text cleanup, and clearer instructions. The converter supports three TTS backends, selected with `--backend`: the original Qwen3-TTS Gradio demos (`gradio`, default), [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) (`faster`), and [audio.cpp](https://github.com/0xShug0/audio.cpp) (`audiocpp`) — all serving the same Qwen3-TTS 1.7B model. +This builds upon [WhiskeyCoder/Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) adding more output files, support for faster backends, metadata, generated cover art, transcription/speed/language options, better text cleanup, and clearer instructions. The converter supports three TTS backends, selected with `--backend`: the original Qwen3-TTS Gradio demos (`gradio`, default), [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) (`faster`), and [audio.cpp](https://github.com/0xShug0/audio.cpp) (`audiocpp`) — the last of which can serve either the same Qwen3-TTS 1.7B model (Option 3) or any of audio.cpp's larger non-Qwen TTS families like Higgs Audio v3 4B, VoxCPM2, and IndexTTS-2/2.5 (Option 4). ## Overview @@ -10,7 +10,7 @@ The converter sends text extracted from your books to a locally running Qwen3-TT - Input: `.txt`, `.pdf`, or `.epub` - Output: `.m4b`, `.mp3`, `.ogg`, or `.flac` -- Supports [qwen-tts](https://pypi.org/project/qwen-tts/), [faster-qwen-tts](https://github.com/andimarafioti/faster-qwen3-tts), and [audio.cpp](https://github.com/0xShug0/audio.cpp) backend servers +- Supports [qwen-tts](https://pypi.org/project/qwen-tts/), [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts), and [audio.cpp](https://github.com/0xShug0/audio.cpp) backend servers (audio.cpp can host any of its TTS model families, not just Qwen3-TTS) - Output a single file or one per chapter - Automatic metadata (title/artist/album tags, chapter track numbers) and a generated cover - Clone voices from .wav reference files or use the built-in speaker in the CustomVoice model. @@ -39,7 +39,8 @@ You will also need to install one of the following backends (see below for insta | -------------------------------------------------------------------- | ------------------------------------------------- | | [Qwen-TTS](https://pypi.org/project/qwen-tts/) | Gradio server released by Qwen | | [Faster-Qwen-TTS](https://github.com/andimarafioti/faster-qwen3-tts) | Server with 2-8x faster inference for NVidia GPUs | -| [audio.cpp](https://github.com/0xShug0/audio.cpp) | Newer C++ TTS backend that supports Qwen-TTS | +| [audio.cpp](https://github.com/0xShug0/audio.cpp) (Qwen) | Newer C++ TTS backend that supports Qwen-TTS | +| [audio.cpp](https://github.com/0xShug0/audio.cpp) (other families) | Same backend hosting larger/higher-quality models | ## Options @@ -52,8 +53,8 @@ You will also need to install one of the following backends (see below for insta | `--speed <n>` | Playback speed, pitch-preserving (`1.0` = normal). A normal-speed copy is also output. | | `--single-file` | Merge all chapters into a single file (default: one file per chapter). `m4b` is always one file. | | `--language <lang>` | Output language for the synthesized speech. Can add an accent even if the text is English. | -| `--backend {gradio,faster,audiocpp}` | TTS server to use (default `gradio`). `faster` and `audiocpp` require their server running first — see the backend sections above. | -| `--voice <name>` | Voice to request from a server-side voice configuration (`--backend faster` or `audiocpp` only). | +| `--backend {gradio,faster,audiocpp}` | TTS server to use (default `gradio`). `faster` and `audiocpp` require their server running first — see the backend sections above. With `audiocpp` the server may host any audio.cpp TTS model family (see Option 4). | +| `--voice <name>` | Voice to request from a server-side voice configuration (`--backend faster` or `audiocpp` only). Required for audio.cpp families without built-in speakers (everything except Qwen3-TTS). | | `--chunk` | Force client-side chunking into `CHUNK_SIZE`-word requests. Only matters for `--backend audiocpp`, which otherwise sends each chapter as one request and lets the server chunk long text itself (may double-chunk); the `gradio` and `faster` backends always chunk. | | `--debug` | Troubleshooting: dump each chunk's raw audio and sent text to `debug/` and log every request. | @@ -146,7 +147,7 @@ Then from another terminal, run audiobook.py with `--backend faster` python audiobook.py --backend faster [--voice NAME] ``` -## Backend Option 3: audio.cpp +## Backend Option 3: audio.cpp with Qwen3-TTS Build `audiocpp_server` for your platform and backend `(cuda, vulkan, hip, cpu)`. Check [audio.cpp's readme](https://github.com/0xShug0/audio.cpp) for details. I'm using one of the helper scripts: @@ -216,6 +217,92 @@ python audiobook.py --backend audiocpp python audiobook.py --backend audiocpp --voice narrator ``` +## Backend Option 4: audio.cpp with non-Qwen models + +The same `audiocpp_server` can host most of audio.cpp's other TTS model families, including models that are larger or higher quality than Qwen3-TTS 1.7B. The converter detects the model family from the server at startup and adapts its requests automatically (language codes, style instructions, etc.), so no other converter settings change: point `AUDIOCPP_MODEL_ID` at the entry you want, start the server, and convert with `--backend audiocpp --voice <name>`. + +One difference from Qwen3-TTS: **all of these families are clone-only** — they have no built-in speakers, so a reference voice must be configured on the server and selected with `--voice`. Running without `--voice` fails fast with a hint instead of synthesizing a random voice. + +Supported families (see [audio.cpp's model list](https://github.com/0xShug0/audio.cpp#supported-models) for the full catalog): + +| Family | Model | Languages | Notes | +| ------------------------------------------- | --------------------- | ------------------------ | ------------------------------------------------ | +| `higgs_audio_tts` | Higgs Audio v3 TTS 4B | 100+ | Largest TTS in audio.cpp; expressive, inline emotion/style control | +| `voxcpm2` | VoxCPM2-2B | 29 listed | 48 kHz output (others are 24 kHz); cloning + "ultimate clone" (audio + transcript) | +| `index_tts2` | IndexTTS-2 | zh, en | Top-tier cloning quality | +| `index_tts2` (package `index_tts2_5_*`) | IndexTTS-2.5 | zh, en, ja, es, ar | Multilingual IndexTTS variant | + +The converter also works with families not in this table (Fish Audio, Chatterbox, DotTTS, OmniVoice, ...) through its generic profile: clone-only, voice from `--voice`, language detected by the model itself. Anything you can host in `audiocpp_server` with `"task": "tts"` should work. + +### Install and run + +Build `audiocpp_server` exactly as in Option 3 (same binary), then download a model package with audio.cpp's model manager from the audio.cpp checkout: + +```bash +python3 tools/model_manager_v2.py install higgs_audio_tts_4b_q8_0 +# or: python3 tools/model_manager_v2.py install voxcpm2_q8_0 +# or: python3 tools/model_manager_v2.py install index_tts2_q8_0 +# or: python3 tools/model_manager_v2.py install index_tts2_5_q8_0 +``` + +Create a `server.json` hosting the model plus your cloning voices as `voice_presets`. Note that `id` must match `AUDIOCPP_MODEL_ID` in `converter/config.py` (set `AUDIOCPP_CLONE_MODEL_ID` to the same id — single-model servers use one entry for both): + +```json +{ + "host": "127.0.0.1", + "port": 8080, + "backend": "cuda", + "lazy_load": false, + "models": [ + { + "id": "higgs", + "family": "higgs_audio_tts", + "path": "models/Higgs-Audio-v3-TTS-4B-GGUF", + "task": "tts", + "mode": "offline", + "voice_presets": { + "narrator": { + "voice_ref": "/path/to/reference.wav", + "reference_text": "Transcript of the reference audio." + } + } + } + ] +} +``` + +Start the server and convert: + +```bash +./build/linux-cuda-release/bin/audiocpp_server --config server.json + +# In another terminal +python audiobook.py --backend audiocpp --voice narrator +``` + +VRAM note: the 4B Higgs Audio Q8_0 package needs roughly 2.5x the memory of the 1.7B Qwen3-TTS packages; VoxCPM2-2B and IndexTTS-2 sit in between. BF16/F16 packages roughly double the footprint again. + +### Generating server.json with make_audiocpp_server_json + +`tools/make_audiocpp_server_json.py` supports the families above directly, including automatic whisper transcription of your reference wavs and updating `converter/config.py` to point at the generated entry: + +```bash +# Interactive: pick the family from a menu +python tools/make_audiocpp_server_json.py path/to/clone/wavs + +# Fully specified: Higgs Audio with wavs transcribed into voice presets +python tools/make_audiocpp_server_json.py path/to/clone/wavs \ + --family higgs_audio_tts --model-id higgs \ + --model-path models/Higgs-Audio-v3-TTS-4B-GGUF \ + --backend cuda --output server.json --force +``` + +The tool offers to rewrite `AUDIOCPP_MODEL_ID`/`AUDIOCPP_CLONE_MODEL_ID` in `converter/config.py` to the new entry id so `audiobook.py` talks to it without manual editing (answer "y" at the prompt). As with the Qwen flow, transcripts matter a lot for cloning quality — fill in any empty `reference_text` fields by hand before starting the server. + +### Language handling + +`--language` works with these families too, adapted per family: IndexTTS sends a language code (`English` → `en`), while Higgs Audio and VoxCPM2 detect the language from the text themselves and omit the field. `--language Auto` never sends a language. + ## Optional: FlashAttention for qwen-tts-demo server FlashAttention provides a small speed boost on the `qwen-tts-demo` backend. It is **not** relevant with the `faster` or `audiocpp` backends, and switching to either of those will provide a bigger speed boost. |