Abstract:
Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks.
However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese.
Our preliminary experiments on Chinese speech synthesis reveal the issue of ``tone shift", where
a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue,
we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision
to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that
the discrete units acquired through the TonUnit resolve the ``tone shift" issue in synthesized Chinese speech
and yield favorable results in English synthesis. Moreover, the experimental results suggest that
finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data.