diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 195 |
1 files changed, 125 insertions, 70 deletions
@@ -2,7 +2,7 @@ Convert TXT, PDF, and EPUB files into audiobooks using the Qwen3-TTS voice model. -This builds upon [WhiskeyCoder/Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) adding more output files, support for a faster backend, metadata, generated cover art, transcription/speed/language options, better text cleanup, and clearer instructions. It also expects the qwen-tts server to be on different ports per model, so two server processes can run at once. +This builds upon [WhiskeyCoder/Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) adding more output files, support for faster backends, metadata, generated cover art, transcription/speed/language options, better text cleanup, and clearer instructions. The converter supports three TTS backends, selected with `--backend`: the original Qwen3-TTS Gradio demos (`gradio`, default), [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) (`faster`), and [audio.cpp](https://github.com/0xShug0/audio.cpp) (`audiocpp`) — all serving the same Qwen3-TTS 1.7B model. ## Overview @@ -10,11 +10,10 @@ The converter sends text extracted from your books to a locally running Qwen3-TT - Input: `.txt`, `.pdf`, or `.epub` - Output: `.m4b`, `.mp3`, `.ogg`, or `.flac` +- Supports [qwen-tts](https://pypi.org/project/qwen-tts/), [faster-qwen-tts](https://github.com/andimarafioti/faster-qwen3-tts), and [audio.cpp](https://github.com/0xShug0/audio.cpp) backend servers - Output a single file or one per chapter - Automatic metadata (title/artist/album tags, chapter track numbers) and a generated cover -- Two voice modes: - - Custom voice: pre-built speakers - - Voice clone: clone a voice from a `.wav` reference audio file +- Clone voices from .wav reference files or use the built-in speaker in the CustomVoice model. ## Prerequisites @@ -24,40 +23,56 @@ The converter sends text extracted from your books to a locally running Qwen3-TT ## Installation -Install ffmpeg and your python environment of choice, e.g. conda. - -```bash -sudo pacman -S conda ffmpeg #Arch Linux -sudo apt-get install ffmpeg #Debian, conda must be installed separately -``` - -### Install Qwen3-TTS (Server) - ```bash conda create -n qwen3-tts python=3.12 -y conda activate qwen3-tts -pip install -U qwen-tts -``` - -### Install the conversion script - -```bash git clone https://git.historia.vg/git/qwen3-audiobook-converter cd qwen3-audiobook-converter pip install -r requirements.txt ``` -## Running the Qwen Gradio server and audiobook script +Put your book files (epub, etc.) in the `input/` directory. The output goes to `output/`. -The audiobook script talks to a Qwen3-TTS Gradio server that is run using `qwen-tts-demo`. Add `--no-flash-attn` if FlashAttention isn't installed (see below). The script expects the custom voice model and base model to be on different ports depending on which you're using. The Qwen model(s) will automatically download. +You will also need to install one of the following backends (see below for installation/usage) -Put your book files (epub, etc.) in the `input/` folder. Then run the script. The output goes to `output/`. +| Backend | Description | +| -------------------------------------------------------------------- | ------------------------------------------------- | +| [Qwen-TTS](https://pypi.org/project/qwen-tts/) | Gradio server released by Qwen | +| [Faster-Qwen-TTS](https://github.com/andimarafioti/faster-qwen3-tts) | Server with 2-8x faster inference for NVidia GPUs | +| [audio.cpp](https://github.com/0xShug0/audio.cpp) | Newer C++ TTS backend that supports Qwen-TTS | -### Voice clone +## Options + +| Flag | Description | +| ----------------------------- | ------------------------------------------------------------------------------------------------ | +| `--format {mp3,m4b,ogg,flac}` | Output format (default `m4b`). `m4b` uses AAC audio and has built-in chapters. | +| `--clone <path>` | Reference audio (`wav`) for voice cloning. | +| `--transcription "..."` | Override whisper auto-transcription with manual audio transcript. | +| `--no-transcription` | Skip auto-transcription of the reference audio. | +| `--speed <n>` | Playback speed, pitch-preserving (`1.0` = normal). A normal-speed copy is also output. | +| `--single-file` | Merge all chapters into a single file (default: one file per chapter). `m4b` is always one file. | +| `--language <lang>` | Output language for the synthesized speech. Can add an accent even if the text is English. | +| `--backend {gradio,faster,audiocpp}` | TTS server to use (default `gradio`). `faster` and `audiocpp` require their server running first — see the backend sections above. | +| `--voice <name>` | Voice to request from a server-side voice configuration (`--backend faster` or `audiocpp` only). | +| `--debug` | Troubleshooting: dump each chunk's raw audio and sent text to `debug/` and log every request. | + +Other options including backend server URLs/ports are configured in `converter/config.py` + +## Backend Option 1: Qwen3-TTS + +Install qwen-tts with pip: ```bash conda activate qwen3-tts -qwen-tts-demo Qwen/Qwen3-TTS-12Hz-1.7B-Base --ip 127.0.0.1 --port 7861 --no-flash-attn +pip install -U qwen-tts +``` + +Run the backend with `qwen-tts-demo`. Add `--no-flash-attn` if FlashAttention isn't installed (see below). Note that the Base model and CustomVoice model run on different ports. + +### Voice clone + +```bash +qwen-tts-demo Qwen/Qwen3-TTS-12Hz-1.7B-Base --ip 127.0.0.1 --port 7861 [--no-flash-attn] ``` Then in another terminal: @@ -75,7 +90,7 @@ Whisper (`faster_whisper` or `whisper`) is used automatically to transcribe the ```bash conda activate qwen3-tts -qwen-tts-demo Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --ip 127.0.0.1 --port 7860 --no-flash-attn +qwen-tts-demo Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --ip 127.0.0.1 --port 7860 [--no-flash-attn] ``` ```bash @@ -85,22 +100,22 @@ python audiobook.py Change the voice settings in `converter/config.py`. -## Optional: Using the `--faster` backend - -Instead of the Qwen Gradio demos, `--faster` talks to the OpenAI-compatible server from [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts), which uses CUDA graph capture for roughly 5-10x faster inference with the same models. **It requires an NVIDIA GPU**. +## Backend Option 2: faster-qwen-tts -**The `--faster` backend always uses voice cloning**. The reference voice and language are configured on the **server**, not through the converter. The server does not transcribe reference audio itself, so transcripts must come from you — either by hand, or with the `tools/make_voices.py` helper (see below). +`--backend faster` talks to the OpenAI-compatible server from [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts), which uses CUDA graph capture for roughly 5-10x faster inference with the same models. **It requires an NVIDIA GPU**. -### Install -Install into the **same `qwen3-tts` conda environment** used for the Gradio server. +Install into the **same `qwen3-tts` conda environment** used for qwen-tts. ```bash conda activate qwen3-tts +pip install -U qwen-tts pip install "faster-qwen3-tts[demo]" ``` -### Run the server +### Voice Clone + +**This backend always uses voice cloning**. The reference voice and language are configured on the **server**, not through the converter. The server does not transcribe reference audio itself, so do it manually or use the `tools/make_faster_voices_json.py` helper (see below). The pip package does not include the server script, so clone the repository: @@ -109,17 +124,7 @@ git clone https://github.com/andimarafioti/faster-qwen3-tts cd faster-qwen3-tts ``` -Single voice (the voice is named `default`): - -```bash -python examples/openai_server.py \ - --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \ - --ref-audio /absolute/path/to/reference.wav \ - --ref-text "Transcript of the reference audio." \ - --language English --port 8000 -``` - -Multiple voices — create a `voices.json` mapping names to reference configurations: +Create a `voices.json` mapping names to reference configurations (.wav to clone, transcript, language). Optionally run `python ./tools/make_faster_voices_json.py path/to/clone/wavs` to automatically create a `voices.json` using whisper to automatically transcribe the test audio. ```json { @@ -128,44 +133,93 @@ Multiple voices — create a `voices.json` mapping names to reference configurat } ``` +Run the server + ```bash python examples/openai_server.py --voices voices.json --port 8000 ``` -### Generating voices.json (optional) +Then from another terminal, run audiobook.py with `--backend faster` + +```bash +python audiobook.py --backend faster [--voice NAME] +``` + +## Backend Option 3: audio.cpp -The `tools/make_voices.py` helper builds a `voices.json` for the server: it transcribes every `.wav` in a directory with using whisper (which is in the qwen3-tts environment). By default it puts voices.json into the input directory. Check the help with `-h` for more options. +Build `audiocpp_server` for your platform and backend `(cuda, vulkan, hip, cpu)`. Check [audio.cpp's readme](https://github.com/0xShug0/audio.cpp) for details. I'm using one of the helper scripts: ```bash -python tools/make_voices.py path/to/wavs +git clone https://github.com/0xShug0/audio.cpp +cd audio.cpp +scripts/build_linux.sh --backend cuda --target audiocpp_server ``` -### Run the converter +Download the Qwen3-TTS GGUF packages (Base for cloning, CustomVoice for built-in speakers) with the python model manager script. This will download these to `./models` ```bash -python audiobook.py --faster [--faster-voice NAME] +python3 tools/model_manager_v2.py install qwen3_tts_1_7b_base_q8_0 +python3 tools/model_manager_v2.py install qwen3_tts_1_7b_customvoice_q8_0 ``` -## Options +Create a `server.json` file. One server can host multiple models. Note that the `id:` field(s) must match `AUDIOCPP_MODEL_ID` and `AUDIOCPP_CLONE_MODEL_ID` in qwen3_ebook_converter's .`converter/config.py`. Optionally run `tools/make_audiocpp_server_json.py path/to/clone/wavs` to make `server.json` for you with automatic whisper transcription. -| Flag | Description | -| ----------------------------- | ------------------------------------------------------------------------------------------------ | -| `--format {mp3,m4b,ogg,flac}` | Output format (default `m4b`). `m4b` uses AAC audio and has built-in chapters. | -| `--clone <path>` | Reference audio (`wav`) for voice cloning. | -| `--transcription "..."` | Override whisper auto-transcription with manual audio transcript. | -| `--no-transcription` | Skip auto-transcription of the reference audio. | -| `--speed <n>` | Playback speed, pitch-preserving (`1.0` = normal). A normal-speed copy is also output. | -| `--single-file` | Merge all chapters into a single file (default: one file per chapter). `m4b` is always one file. | -| `--language <lang>` | Output language for the synthesized speech. Can add an accent even if the text is English. | -| `--faster` | Use a faster-qwen3-tts OpenAI-compatible server (up to 5x faster in certain cases). | -| `--faster-voice <name>` | Chooses a voice from voices.json when using `--faster` with multiple voices. | -| `--debug` | Troubleshooting: dump each chunk's raw audio and sent text to `debug/` and log every request. | +```json +{ + "host": "127.0.0.1", + "port": 8080, + "backend": "cuda", + "lazy_load": false, + "models": [ + { + "id": "qwen", + "family": "qwen3_tts", + "path": "models/Qwen3-TTS-12Hz-1.7B-CustomVoice-GGUF", + "task": "tts", + "mode": "offline" + }, + { + "id": "qwen3-clone", + "family": "qwen3_tts", + "path": "models/Qwen3-TTS-12Hz-1.7B-Base-GGUF", + "task": "tts", + "mode": "offline", + "voice_presets": { + "narrator": { + "voice_ref": "/path/to/reference.wav", + "reference_text": "Transcript of the reference audio." + }, + "obama": { + "voice_ref": "/path/to/reference2.wav", + "reference_text": "Transcript of reference audio." + } + } + } + ] +} +``` + +Run the server. The `audiocpp_server` path will be slightly different depending on your platform and build options: -Other options and defaults are configured in `converter/config.py` +```bash +./build/linux-cuda-release/bin/audiocpp_server --config server.json +``` + +Then in a different terminal, run `audiobook.py` + +```bash +# Built-in speaker +python audiobook.py --backend audiocpp + +# Voice cloning +python audiobook.py --backend audiocpp --voice narrator +``` ## Optional: FlashAttention for qwen-tts-demo server -This is **not** used with the `--faster` backend. The Gradio server tries to use FlashAttention 2 by default, but requires `--no-flash-attn` without it. On supported GPUs FlashAttention can give a modest speedup. +FlashAttention provides a small speed boost on the `qwen-tts-demo` backend. It is **not** relevant with the `faster` or `audiocpp` backends, and switching to either of those will provide a bigger speed boost. + +`qwen-tts-demo` server tries to use FlashAttention 2 by default and requires `--no-flash-attn` without it. You have two options to install FlashAttention in your python environment: 1. Build from source (takes absolutely forever). If you run out of memory, lower MAX_JOBS until you don't. @@ -175,23 +229,23 @@ pip install ninja packaging psutil MAX_JOBS=4 pip install --no-build-isolation flash-attn ``` -2. Or pip install a prebuilt wheel matching your torch / CUDA / Python / CXX11-ABI combination: +2. pip install a prebuilt wheel matching your torch / CUDA / Python / CXX11-ABI combination: ```bash +conda activate qwen3-tts python -c "import torch; print(torch.__version__, torch.version.cuda, torch._C._GLIBCXX_USE_CXX11_ABI)" ``` -Official wheels: https://github.com/Dao-AILab/flash-attention/releases (pick `cp312` + matching `cuX` + `torchX.Y` + `cxx11abiTRUE/FALSE`). - -Third-party wheels: https://mjunya.com/flash-attention-prebuild-wheels/ (hosted at https://github.com/mjun0812/flash-attention-prebuild-wheels). +- [Official wheels](https://github.com/Dao-AILab/flash-attention/releases) - Pick `cp312` + matching `cuX` + `torchX.Y` + `cxx11abiTRUE/FALSE` +- [Third-party wheels](https://mjunya.com/flash-attention-prebuild-wheels/) ## Tips -Transcription affects the output a lot. Whisper is okay, but does not give perfect transcription. A manual transcription passed via `--transcription` is better. +Transcription affects the output a lot. Whisper does not always give perfect transcription. Manual transcription is better. If you're cloning one language and outputting another language, `--no-transcription` will remove the accent. Alternatively, setting the "wrong" output `--language` can add an accent. -Even tiny amounts of pause between phrases in the sample audio can have a big impact. Try increasing or decreasing them. +Even tiny amounts of pause between phrases in the sample audio can have a big impact. Try increasing or decreasing them or find a sample with different cadence. ## License @@ -199,5 +253,6 @@ MIT ## Credits -- [Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) by WhiskeyCoder. +- [Qwen3-Audiobook-Converter](https://github.com/WhiskeyCoder/Qwen3-Audiobook-Converter) by WhiskeyCoder. This project was originally built upon this. - [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) voice model. +- [audio.cpp](https://github.com/0xShug0/audio.cpp) inference engine for the `audiocpp` backend. |