How I Built Self-Hosted Chinese TTS

Every HSKStory text is developed and closely edited before it reaches the audio pipeline. The self-hosted Chinese TTS system turns that approved text into two consistent Mandarin voices; it does not make the editorial decisions.

Every published story includes Mandarin audio reviewed for pronunciation, pacing, and synchronization. This guide explains the current production path behind that promise.

The Current System at a Glance

The production setup below was checked against the current code and operating documents on July 29, 2026.

The job is deliberately a pipeline rather than one model call:

1. Register both voices for the pinned model on a private worker.
2. Generate every chapter in both voices from the current story.md.
3. Check completeness and audio quality, then apply the loudness post-pass.
4. Generate and validate word timings with MFA.
5. Promote the story only when both voices have matching audio and timestamp chapter sets.

Why I Still Self-Host

Self-hosting gives me control over the parts that matter for a graded reader: the exact model pin, voice references, text chunking, retry behavior, native speaking rate, and every file that reaches quality control. It also lets me prepare a whole library as an offline batch instead of making learner playback depend on a live synthesis request.

That control is useful because speech quality is not one setting. A voice can sound natural in a short sample but speed up in dialogue, truncate a long passage, pronounce a name incorrectly, or acquire codec noise after export. Owning the generation path makes each of those failures inspectable.

The tradeoff is operational work. GPU marketplace prices and availability change, instances can fail, and a completed generation process does not stop rental billing. Outputs must be downloaded and verified, then every rented worker must be destroyed and confirmed absent.

Building Voices from Native-Speaker References

The voice system uses short reference recordings from native Mandarin speakers. Each recording is paired with its exact transcript. On a new worker, the registration step converts those inputs into a speaker cache under the public voice IDs luna and kai.

That cache is derived state tied to CosyVoice2-0.5B; it is regenerated for a new worker and cannot be treated as a model-independent voice file. Changing the model means registering and validating both voices again.

Reference quality affects more than timbre. It can also influence pace, expressiveness, and stability across a chapter. That is why a successful registration is only the start: both voices still go through complete-story generation and the same listening and measurement gates.

After registration, CosyVoice synthesizes the editors' approved story text in the two product voices. The native-speaker recordings are reference inputs to that system, while the finished story audio is self-hosted text-to-speech.

Generating Long Multi-Chapter Stories

The worker receives each story's current Markdown source and generates all its chapters in both voices. For large batches, stories are divided among workers by Chinese-text length rather than file count so one worker does not receive all the longest stories.

The current generator, checked July 29, 2026, prepares each chapter like this:

1. Parse the chapter title and body, remove Markdown formatting, and normalize forms that need a stable spoken equivalent.
2. Split long text at paragraph breaks, line breaks, and sentence-ending punctuation, aiming for a current default target of 200 characters. A single sentence without a supported breakpoint can remain longer.
3. Synthesize each chunk offline at speed 1.0.
4. Join the chunks with a 300 ms pause and export one MP3 for the chapter.
5. Write a manifest recording chunk boundaries, durations, text hashes, model, voice, and speed.

CosyVoice can occasionally stop a chunk too early. The generator compares the result with a conservative duration estimate, retries an implausibly short chunk, and fails the chapter if the retries still look truncated. A failed chapter stays failed rather than becoming a deceptively valid MP3.

Each story-and-voice job writes an explicit success or failure result. The staging output also contains manifests and diagnostic files, so only validated chapter-XX.mp3 files are promoted into the story library.

Pacing, Pronunciation, and Loudness QA

Learner audio needs layered checks because no single metric proves that a recording is correct.

Completeness and listening. Both voices must have the same non-empty chapter set. I listen to the beginning, middle, and end of each changed story in both voices, checking for truncation, silence, artifacts, garbling, the wrong voice, unstable pacing, and non-Mandarin coloring.

Pace. The chunk manifest lets the checker compare Chinese characters per second across a chapter. As checked July 29, 2026, it warns at a 1.3 max/min ratio and hard-fails at 1.5; with exact Chinese-character counts it also hard-fails when a reliable chunk exceeds 6.0 characters per second. A warning still needs listening because an expressive pause can increase the raw ratio without making the spoken words too fast. Genuinely fast chunks are regenerated rather than stretched, which avoids introducing robotic artifacts.

Pronunciation and text agreement. Representative excerpts are transcribed and compared with the edited Chinese source. High-error results receive manual review for incorrect pronunciation, dialect coloring, text mismatch, or garbling. This complements direct listening; it does not replace it.

Loudness and codec behavior. The current post-pass uses fixed linear gain and an oversampled true-peak limiter rather than time-varying normalization. The July 29 configuration targets about -19.5 LUFS and uses a -2.5 dBTP limiter ceiling so the final low-bitrate MP3 remains inside the Story Contract's true-peak gate after re-encoding. Each final file also passes a simulated Bluetooth SBC round trip, which catches high-frequency rasp that can be hidden during ordinary wired playback.

Only the final loudness-normalized MP3 continues to timestamp alignment.

Word-Level Timing and Synchronization

CosyVoice returns synthesized speech but no native word timings. HSKStory therefore treats alignment as a separate production stage and uses MFA as the sole accepted timestamp workflow.

The prepare step converts the final MP3 to a 16 kHz mono WAV and builds an alignment transcript from the chapter title plus the exact timed word groups in the story's pinyin data. It also records hashes for the audio, metadata, pinyin, story source, and alignment script.

The generate step uses the pinned MFA 3.3.9 environment with Mandarin acoustic, dictionary, and pronunciation models. MFA produces a TextGrid; our converter maps its intervals back onto the story's word groups and writes the timestamp JSON used for synchronized highlighting.

Before import, the timestamp structure is validated against the current pinyin data. The timestamp file and a provenance record are then installed as one rollback unit. That provenance binds the exact timestamp bytes to the exact aligned MP3, preventing timings from silently surviving after an audio change.

The complete prepare, generate, and import sequence runs separately for luna and kai. Final checks require audio and timestamp chapter parity for both voices, followed by a synchronization review in the reader.

What the Earlier Qwen and Dylan System Taught Me

The earlier self-hosted system used Qwen3-TTS with a voice called Dylan. That pipeline is historical: it is not the current generation path, and its Qwen timestamp aligner has been retired.

It still produced durable lessons that shaped the replacement:

• Long prose needs structure-aware chunking; splitting at an arbitrary character cuts sentences and creates audible seams.
• A detached batch needs explicit per-item results, resumable operations, and worker teardown that is verified outside the SSH session.
• Audio must be tied to the exact story source, because a text correction makes the old recording and its timings stale.
• A model's preferred voice or setup is not a permanent product architecture.
• Manual listening and word alignment must remain independent of the synthesis engine.

Those lessons survived. The model, product voices, QA gates, and timing implementation changed.

Current Limits and Operational Tradeoffs

Self-hosted Chinese TTS moves responsibility rather than removing it:

• The pinned model and registered voices make production repeatable, but a model upgrade requires full voice registration and validation again.
• Generation is stochastic. Retry and reroll tools reduce truncation and pace outliers, but manual listening still decides whether a take is acceptable.
• The current loudness post-pass re-encodes the raw MP3. Its measurements and final-file gate matter more than assuming an export setting is harmless.
• Forced alignment estimates where the spoken words occurred; it cannot prove that the pronunciation itself was correct.
• Rented compute keeps the system flexible, but pricing, host quality, transfer, monitoring, and verified teardown remain operational work.

The TTS pipeline is one part of How I Make Graded Chinese Stories. The word groups used for alignment come from the separate guide to adding accurate pinyin to Chinese text.