用 Qwen3-TTS 做自定义声音

With my kids I'm currently building a little dungeon crawler and we're trying to learn a bit about game development along the way. However given that we live in the modern times of increasing capabilities we wanted to get a narrator and voice over. But we also wanted it to not sound like everyone's standard AI voices so I looked into what options there are on the table. After some investigation I narrowed down to using Qwen3-TTS which uses voice embeddings for custom voices. With a bit of creativity you can also do more with those custom voices than appears initially. In case this is of interest to you, this post just covers what we learned and with some examples. You can probably point your agent to it and it can do something with it.

我正和孩子们做一个小的地牢爬行游戏，顺便学点游戏开发。但既然活在能力日新月异的现代，我们想给它加个旁白配音——又不想听起来像人人都有的标准 AI 声音，于是我调查了桌上的选项。一番筛选后锁定 Qwen3-TTS：它用声音嵌入（voice embeddings）做自定义声音，而稍加创意，这些自定义声音能玩出的花样比表面看起来多。如果你有兴趣，这篇就是我们学到的东西加示例——你大概可以把你的 agent 指过来让它自己折腾。

Qwen3 is not the only TTS model in town, and it wasn't even my first choice. We initially used Kokoro. However Qwen3 supports voice cloning and we wanted to have it feel a bit more unique. In practice it's a bit trickier though. Qwen3 ships three kinds of models: 1.7B-Base (and 0.6B) clones a voice from a few seconds of reference audio. 1.7B-CustomVoice comes with nine built-in speakers and instructions. 1.7B-VoiceDesign invents a voice from a text description. The base model clones either in context or from the speaker embedding alone but there is no documented way to clone a voice and direct it with instructions. That means unlike the custom voice and voice design models you cannot say "speak sad" or "whisper" with a cloned base model. At present, the official options for a custom voice are fine-tuning Base on a single speaker or to make a clip with VoiceDesign and clone it with Base. That is not all that bad, but it means instructions are gone.

Qwen3 不是城里唯一的 TTS 模型，甚至不是我第一选择——我们最初用 Kokoro。但 Qwen3 支持声音克隆，而我们想要更独特一点。实操比想象的绕一点。Qwen3 出三种模型：**1.7B-Base（及 0.6B）**从几秒参考音频克隆声音；**1.7B-CustomVoice** 自带九个内置说话人与指令；**1.7B-VoiceDesign** 从文字描述发明一个声音。Base 模型可以上下文内克隆或只用说话人嵌入克隆，但**没有文档化的办法既克隆声音又用指令引导它**——也就是说，与 CustomVoice 和 VoiceDesign 不同，你不能对克隆的 Base 模型说「说得悲伤」或「耳语」。目前官方的自定义声音路线是：在单说话人数据上微调 Base，或用 VoiceDesign 造一段再拿 Base 克隆——不算糟，但指令没了。

For this blog post I took 28 seconds of me from my talk at PyAI Conf, cut between two pauses so that it starts and ends on a full sentence, and transcribed it with Whisper. The transcript is really only needed for baking but to compare against an already existing voice it's useful to have it say the same thing as the original recording. Note that a conference talk recording is not ideal but it works anyways. Which is also a good reminder that you really don't need a particularly clean recording (or a very long one) to get a halfway convincing version of a person. Particularly when then presented to you on a low quality phone call, it won't be very hard to trick your grandmother.

这篇的素材：我从 PyAI Conf 演讲里取了 28 秒自己的声音，在两个停顿之间下刀（首尾都是完整句），用 Whisper 转了文字。转录其实只在「烘焙」时需要，但要让克隆声音与已有声音对比，让它说原录音同样的话更直观。会议录音并不理想但照样能用——这也提醒我们：**要做出一个半信半疑真假的人声，其实不需要多干净（或多长）的录音**。尤其再经低质量电话一放，骗过你祖母不难。

The first thing you can do is to blend that recording with another voice. For instance you can use VoiceDesign to make up a voice from a description. For this blog post, the designed voice was created like this, with Qwen's own Python package qwen-tts (pip install qwen-tts, I used 0.1.1), which the rest of the code in this post uses too: import torch; from qwen_tts import Qwen3TTSModel; kw = dict(device_map="mps", dtype=torch.bfloat16, attn_implementation="sdpa"); design = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign", **kw); wavs, sr = design.generate_voice_design(text=transcript, language="English", instruct="A man in his forties with a warm, natural, medium-low voice, speaking clearly and calmly at an easy conversational pace, like someone reading aloud to a friend.")

第一件能做的事：把这段录音与另一个声音混合。比如用 VoiceDesign 从描述造一个声音。本文的设计声音是这样造的（Qwen 自家 Python 包 qwen-tts，pip install qwen-tts，我用 0.1.1，下文代码同）：VoiceDesign 模型（Qwen3-TTS-12Hz-1.7B-VoiceDesign，device_map="mps"、dtype=torch.bfloat16，Apple 芯）加载，generate_voice_design 传入转录文本、language="English"、instruct=「四十来岁男人，温暖自然的中低音，清晰从容、不紧不慢的会话语速，像读给朋友听」。

What Base takes from a recording when cloning from the embedding alone is a speaker embedding which is basically 2048 numbers out of a speaker encoder that was trained. The same voice gets similar numbers and different voices get different ones. Which btw, is also a convenient way to do speaker detection from recordings if you want to! It's not perfect, but it's quite helpful for a single audio track recording to tell two speakers apart if you do some basic slicing and dicing.

只用嵌入克隆时，Base 从录音里取的东西是**说话人嵌入（speaker embedding）**——本质是训练过的说话人编码器输出的 2048 个数。同一声音得到相近的数、不同声音得到相远的。顺带一提，这也是做录音说话人检测的便捷方式：不完美，但对单音轨录音，稍微切一切分一分，区分两个说话人相当好用。

Because it is just a vector, you can do arithmetic on it. For instance a weighted average of two embeddings is a speaker that is somewhere between two voices: base = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base", **kw); def fingerprint(path): prompt = base.create_voice_clone_prompt(ref_audio=path, x_vector_only_mode=True); return prompt[0].ref_spk_embedding; def blend(a, b, t): a, b = a.float(), b.float(); mixed = (1 - t) * a + t * b; return (mixed / mixed.norm() * ((1 - t) * a.norm() + t * b.norm())).to(torch.bfloat16); armin, designed = fingerprint("voices/armin.flac"), fingerprint("voices/designed.flac"). Here is the same sentence in eleven blends.

因为它只是一个向量，你可以对它做算术。比如两个嵌入的加权平均就是一个介于两个声音之间的说话人：Base 模型（Qwen3-TTS-12Hz-1.7B-Base，dtype=torch.bfloat16）的 create_voice_clone_prompt(ref_audio=path, x_vector_only_mode=True) 拿到 prompt[0] 的 ref_spk_embedding（即 fingerprint 函数）；blend(a, b, t) 做 (1-t)·a + t·b 的加权混合并保范数归一（mixed/mixed.norm() 乘回两侧范数的加权和）。同一句话在十一个混合比例下的效果见原文音频。

Note that it stops sounding like me quite quickly, but it retains some of my features. And it for sure ends up with creating a unique voice regardless. I found some better successes when ensuring that the two voices have similar spectograms (eg: similar "recording setups"). In this recording you can hear that one voice picked up by the lapel mic was a bit hollow compared to the designed voice.

注意它很快就不再像我了，但保留了一些我的特征，而且无论如何都会产出一个独一无二的声音。我发现**两个声音频谱相近（比如「录音环境」相似）时混合更成功**——这段录音里能听出领夹麦收的那个声比设计的声音空一点。

If you like a voice that comes out, you can bake it back into a new recording that you can then use as reference: import soundfile as sf; mixed = blend(armin, designed, 0.5); audio, rate = say(open("voices/transcript.txt").read(), mixed); sf.write("voices/final.flac", audio, rate); final = fingerprint("voices/final.flac"). The baked voice does not come back exactly as the blend but it makes handling it easier if you want to. It's just one voice reference file.

喜欢某个混出来的声音，可以把它**烘焙（bake）**回一段新录音当参考用：blend 取 0.5，say() 合成转录文本，sf.write 存成 final.flac，再对它 fingerprint。烘焙回来的嵌入与原混合不完全一致，但管理起来更容易——就是一个声音参考文件。

So on the question of using those embeddings for speaker recognition: how good does that work? I took an episode of State of Agentic Coding, the podcast Ben and I record, cut it into 2-second windows one second apart and compared each with an embedding of me from the conference. Somewhat surprising, but maybe not too surprising using my conference recording clip as the reference this does not work at all with Qwen3. If instead I take the 20 seconds where I introduce myself at the start of the episode as the reference, it gets 98% of the rest right. On the other hand using a model that was trained for speaker verification instead does manage to pick me up from the conference recording clip. For instance ECAPA, finds me from the conference clip in 97% of the windows.

那这些嵌入做说话人识别效果如何？我拿一期 Ben 和我录的播客 State of Agentic Coding，切成 2 秒窗、间隔 1 秒，逐窗与「会议录音的我」的嵌入比对。有点意外但也许不意外：用会议录音片段做参考，Qwen3 上**完全不管用**；改用播客开头 20 秒我自介绍的片段做参考，其余部分 98% 正确。而换成专为说话人验证训练的模型（如 ECAPA），用会议片段也能在 97% 的窗口里找到我。

But the embeddings are very useful for the voice cloning! But how to give them instructions? Base and CustomVoice manipulate the speaker the same way: one vector inserted into the talker's input sequence. In CustomVoice that vector is a row of the embedding table. In Base it is the x-vector from the speaker encoder, which only Base has. The generation code takes either (voice_clone_prompt is a regular argument of generate), and the transformer weights of the two models are nearly identical. So take the embedding with Base, then hand it to CustomVoice instead of one of its speakers: cv = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", **kw); def say(text, xvec, instruct=None, language="English"): ids = cv._tokenize_texts([cv._build_assistant_text(text)]); told = [cv._tokenize_texts([cv._build_instruct_text(instruct)])[0] if instruct else None]; voice = {"ref_code": [None], "ref_spk_embedding": [xvec], "x_vector_only_mode": [True], "icl_mode": [False]}; codes, _ = cv.model.generate(input_ids=ids, instruct_ids=told, voice_clone_prompt=voice, languages=[language], non_streaming_mode=True, **cv._merge_generate_kwargs(temperature=0.7, subtalker_temperature=0.7)); wavs, rate = cv.model.speech_tokenizer.decode([{"audio_codes": codes[0]}]); return wavs[0], rate # 取 codes[0] 与 wavs[0]; wav, rate = say("I really thought this would work.", armin, "Speak in a sad, sorrowful tone, voice low and heavy, on the verge of tears.")

嵌入对声音克隆非常有用！但怎么给它指令？**Base 与 CustomVoice 操纵说话人的方式相同：一个向量插进说话人的输入序列。**CustomVoice 里那个向量是嵌入表的一行；Base 里是说话人编码器的 x-vector（只有 Base 有）。生成代码两者都收（voice_clone_prompt 是 generate 的常规参数），而两个模型的 transformer 权重几乎一致——所以：用 Base 提取嵌入，然后把它交给 CustomVoice 顶替某个内置说话人。say(text, xvec, instruct) 函数：CustomVoice 模型（Qwen3-TTS-12Hz-1.7B-CustomVoice）的 generate（temperature=0.7）里 voice_clone_prompt 传 {"ref_spk_embedding": [xvec], "x_vector_only_mode": [True], "icl_mode": [False]}、instruct_ids 传指令文本，克隆声音就能吃指令——示例：「用悲伤、哀恸的语气说，声音低沉厚重，几近落泪」地说「我真以为这能行」。

Instructions work on cloned voices, but less reliably than on the built-in speakers, and it depends on the emotion, but honestly, it came out better than I anticipated. Definitely good enough to have some true custom voices for a voice over that don't necessarily sound like something, someone else has in their game.

指令在克隆声音上有效，但不如内置说话人可靠、且因情感而异——不过说实话，比我预期的好。绝对够用：给你的配音弄出真正自定义的声音，不必听着像别人游戏里的东西。

If a voice is a point in a 2048-dimensional space and blending works we can maybe do some other funny math on it. The obvious one is to try to see if some math can be used to manipulate gender. To find it I had VoiceDesign create 14 different voices for men and women each. When you then take the average woman minus the average man you can then move any voice along that direction: women = torch.stack([fingerprint(f"design/f{i}.wav") for i in range(14)]).float(); men = torch.stack([fingerprint(f"design/m{i}.wav") for i in range(14)]).float(); middle = (women.mean(0) + men.mean(0)) / 2; axis = women.mean(0) - men.mean(0); axis = axis / axis.norm(); def move(xvec, t): x = xvec.float(); return (x + (t - (x - middle) @ axis) * axis).to(torch.bfloat16). Here is the result for my clone.

如果声音是 2048 维空间中的一个点、而且混合有效，那也许还能做点别的趣味数学。最显然的一个：看看能不能用数学操纵性别。为找这个方向，我让 VoiceDesign 各造了 14 个女声与男声；平均女人减平均男人得到方向轴（axis，归一化），middle 取两均值的中点（代码以 torch.bfloat16 精度运算）；move(xvec, t) 把任意声音移到轴上位置 t（先算当前投影 (x-middle)@axis，再沿 axis 补到目标 t），其余维度不动。我克隆声音的效果见原文音频。

Anyways, that's that. If you want to play around with Qwen3 TTS I hope this was of some help, and maybe you got some ideas for your own project.

总之就这样。想玩玩 Qwen3-TTS 的话，希望这篇帮上了忙，也许还给你的项目添了点子。