Skip to content

Add Qwen3-TTS voice design, consented cloning, and optional fine-tuning #5381

Description

@atomantic

Part of #5377. Blocked by #5380.

Goal

Add one thoroughly integrated richer local voice backend for original voice design, consented rapid cloning, instruction-controlled synthesis, streaming, and later optional fine-tuning.

Proposed approach

  • Add Qwen3-TTS in an isolated, health-probed Python runtime behind the shared voice-engine capability contract.
  • Make model downloads, voice design, cloning, and fine-tuning explicit user actions with visible provider/model/resource requirements.
  • Add Voice lab flows for same-script design previews, consent and rights confirmation for uploaded/recorded material, source transcription, comparison, promotion, and rollback.
  • Support streaming and separately approved studio and interactive routes; benchmark character similarity and first-audio latency before enabling interactive use.
  • Reuse the existing training UX pattern for dataset readiness, progress, samples, checkpoints, cancellation, and explicit checkpoint promotion.
  • Keep raw recordings, performer data, and artifacts machine-local.

Acceptance criteria

  • Voice design and consented instant cloning create candidate profiles without changing the approved character voice.
  • Promotion is explicit and records engine/model revision, seed/instructions, source hashes, rights confirmation, and benchmark revision.
  • Interactive routing is enabled only after a host-specific latency/readiness benchmark passes.
  • Fine-tuning is optional, cancellable, checkpointed, and never assumes the last checkpoint is best.
  • Unsupported controls are capability-gated rather than silently approximated.
  • No model download, training, or batch generation runs at boot or without direct user consent.

Out of scope

  • Adding multiple competing new TTS backends in the first slice.
  • Cloning a voice without documented consent or license posture.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:mediaarea:voiceVoice stack: STT/TTS pipeline, proactive speech, voice tools, call bridgedependenciesProposed from a dependency-freedom auditeffort:xhighDispatch reasoning effort: extra highmodel:heavyDispatch capability: heavyplanTracked by /do:replanplan-featureFeature plan filed by the plan-feature brainstorm

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions