CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

CapTalk paper abstract

CapTalk module framework

The figure below is the framework image from the paper. It shows the hierarchical variational timbre conditioning module and the unified caption-conditioned autoregressive generation pipeline for both single-utterance and dialogue settings.

Figure 1. CapTalk framework. Left: hierarchical variational timbre conditioning. Bottom/right: unified caption-conditioned autoregressive generation for single-utterance and dialogue speech.

Data construction, attribute extraction, and evaluation

Figure 2 is worth keeping on the demo page because it clarifies where the single-utterance and dialogue data come from, how CoT attributes are extracted, and how the evaluation protocol is organized.

Figure 2. Data construction from public and internal speech data, CoT-style attribute extraction for dialogue, and the complete single-utterance and dialogue evaluation pipeline.

Table 1. Results on the InstructTTSEval-ZH Benchmark

Table 2. Human evaluation for single-utterance voice design

All metrics rated on a 1–5 scale. MOS denotes human-rated naturalness.

We present two complete dialogue sessions, not sliced utterances. In each case, the target speaker is synthesized, while the other speaker remains the original recorded voice from the session. Across these examples, the synthesized target speaker maintains stable timbre for nearly three minutes of continuous dialogue.

Two cases are shown here. Each case keeps one target speaker fixed and compares three different dialogue contexts, making the main dialogue claim directly audible: stable target-speaker timbre across turns, while expression stays adaptive to the current local context.

This subsection keeps only the objective controllability part and uses the final dialogue model trained on 32,171 sessions / 6270.68 hours. Same target speaker, same dialogue context, same target text, and same random seed; only one CoT attribute changes at a time. The page focuses on the three most reliable and directly audible dimensions: pitch, energy, and speed.

Table 3. Automatic evaluation results for dialogue voice design.

This is the most important dialogue comparison on the page. Use two representative cases where the gain from CoT is easy to hear in contextual appropriateness, delivery, or role intent.

Table 4. Context coherence comparison in dialogue evaluation.

Keep this as one curated case. The target speaker in this snippet is the first speaker in the dialogue, and the key question is whether the system preserves that speaker's identity while staying coherent with the surrounding turns.

Prompt compatibility note: CapTalk uses the target-speaker natural-language caption directly. Fish Speech S2 Pro does not natively support long natural-language speaker descriptions in dialogue; it works better with short bracketed control tags. This means the Fish side is an interface-adapted comparison with matched caption meaning, but it is not a perfectly symmetric prompt setting.

What this section is designed to show

As described in Section 3.3 of the paper, the hierarchical variational timbre conditioning module is inspired by the factorization idea of FHVAE. It uses an utterance-level speaker encoder and a segment-level pooled latent, with KL regularization between the segment posterior and an utterance-conditioned prior, so that stable speaker and voice attributes are preserved while segment-specific affective variation is attenuated. Its purpose is to let a designed timbre be reused across utterances and dialogue turns while keeping expression adaptive to the current context.

At the same time, caption-conditioned voice design is inherently a one-to-many task. A caption is a high-level, underspecified description that naturally corresponds to a distribution of valid voices rather than a single unique one — many concrete timbres can satisfy the same caption equally well. Furthermore, the stochastic sampling process of the model means that each generation draws a different instance from this distribution. In the standard caption-only generation setting, each run independently predicts a new êspk from the caption-conditioned distribution, so repeated samples under the same caption may sound like different but still caption-consistent voices. This sample-level variability is a property of caption-conditioned sampling, not evidence that the module fails to preserve timbre.

To directly test the timbre-reuse claim of the module, the appendix reports a dedicated timbre reuse analysis that contrasts two inference modes: Fixed êspk, where a designed voice is predicted once and then reused across utterances, and Resampled êspk, where the speaker embedding is re-predicted for every utterance. The resulting cross-utterance SIM is 0.92 in the fixed mode and 0.42 in the resampled mode, showing that once a designed voice is fixed, the module preserves it consistently across utterances.

The listening panel below is therefore meant to complement that analysis from the perceptual side: it lets listeners hear the one-to-many nature of caption-only generation directly, while the appendix numbers quantify the separate timbre-reuse path enabled by fixing the designed speaker embedding.