LibriLight-CV3 ckpt22000 朗读

LibriLight-CV3 checkpoint22000 未合并

用上述模型直接朗读两组文本:之前"训练数据 text 直接合成"测试用过的 5 段文本(每段独立一个 turn),以及 /proj/mmllm/prosody/texts 的 3 段 Shai 客服对话(每行一个 turn,上下文串联)。无指令、无 caption:每个 turn 按该 run 的训练格式输入 文本[SEP_1],模型自由续写 CosyVoice3 音频 token,直到采样出 [SEP_2]。对话的前后 turn 之间只用 [SEP_2] 相连。这个 run 的训练格式里没有角色名,所以 user / agent 不送进模型,只用来选解码音色。

checkpoint/proj/spans/lyf/checkpoint/capcv3-lib128-bs128-20261022-r2/checkpoint-22000 加载方式Qwen3-8B-Base + 原始 LoRA(未与 Instruct 合并、未缩放);W&B run capcv3-lib128-bs128-20261022-r2,step 22000 采样评测规则:文本 0.1 / 音频 0.7(质量守恒)、top_k 150、top_p 0.9;每 turn 最多 1000 音频 token;seed = turn 序号(与 spkname128 ckpt-22010 朗读同一批文本时的设置相同) 解码音色CosyVoice3 flow+HiFT,参考音频只决定音色,语言模型看不到。对话里 agent 句子用 EARS p001 真人男声(愉悦版录音),user 句子和 5 段训练数据文本用新闻播报女声 转写核对WhisperX large-v2 转写每个 turn,WER 对比输入文本(小写、去标点、连字符拆开)

汇总

文本设置turn正常结束音频 s音频 tokenWER 均值WER 中位WER≤0.2
训练数据文本整段一个 turn(独立,不串联)55/545.411360.0210.05/5
Dialog 1:急诊室保险覆盖查询每行一个 turn(上下文串联,只用 [SEP_2] 相连);user/agent 不进 prompt,只决定解码音色1713/17208.552130.2280.013/17
Dialog 2:会员卡补发每行一个 turn(上下文串联,只用 [SEP_2] 相连);user/agent 不进 prompt,只决定解码音色2020/2065.216310.2160.03816/20
Dialog 3:配偶团体计划生效确认每行一个 turn(上下文串联,只用 [SEP_2] 相连);user/agent 不进 prompt,只决定解码音色129/12158.239560.3060.08/12

WER 均值按单句上限 1.0 截断。对话的整段连播里 turn 之间插入了 0.3 s 静音,仅用于播放。

观察:对话里的一词句(Yes. / No. / OK.)只生成了 2–7 个音频 token(0.08–0.28 s),短于一个完整的词,转写结果不可靠。dialog1 第 10–13 句和 dialog3 第 9–11 句都紧跟在这样的短句之后:各句读出整句(dialog1 第 10 句)或只读出开头一两个词后,就一直重复同一个静音 code(0、1 或 28),直到 1000 token 上限,并连续影响后面 2–3 句。

训练数据文本(之前"训练数据 text 直接合成"测试的 5 段)

整段一个 turn(独立,不串联)

5 段,各自独立生成(seed 0,互不进入上下文)

Dialog 1:急诊室保险覆盖查询(17 句)

user: A family member is critically ill and in the emergency room, I need to verify the coverage. agent: I can't imagine how hard this must be for you. agent: I will get the information right away. agent: In order to verify who you are, please provide your date of birth. user: Sure. My date of birth is January fourth, nineteen ninety-one. agent: Thank you. I can confirm your plan has ER coverage with a co-pay of one hundred dollars. agent: There is no deductible or out of pocket for emergency room visits. agent: Would you like me to send you a summary of your coverage through text? user: Yes. agent: Thank you. I will text you the summary of your coverage to the number you are calling from ending with six six five four. user: OK. agent: Please let me know if I can provide any other assistance. user: No, I'm good. agent: I am sorry about your situation. agent: I hope your family member gets better soon. agent: Please reach out with any further concerns. agent: Thank you for calling Baymax Health.

每行一个 turn(上下文串联,只用 [SEP_2] 相连);user/agent 不进 prompt,只决定解码音色

17 个 turn · 拼接 213.32 s · 生成耗时 226.16 s
整段连播(17 句按顺序拼接,句间 0.3 s 静音):
逐句(每句独立音频):

Dialog 2:会员卡补发(20 句)

user: I have requested a member ID card multiple times, but have not yet received it. agent: I am sorry you feel frustrated. agent: I will help you get a member ID card right away. agent: In order to verify who you are, I need you to provide your date of birth. user: Sure. My date of birth is January fourth, nineteen ninety-one. agent: Thank you. I will text you a copy of your member ID card to the number you are calling from, ending with six six five four. agent: This ensures you will immediately get your ID. user: OK. agent: I have just sent you the ID card through text. agent: Please confirm if you have received it. user: I got it. agent: Would you also like me to send a physical copy of your ID card to your home address with zip code one one zero zero one? user: Yes. agent: I have sent the ID card. agent: It will arrive at your address within seven business days. agent: Is there anything else I can help you with? user: No. agent: I am sorry it took us so long to respond to your request. agent: Please reach out with any further concerns. agent: Thank you for calling Baymax Health.

每行一个 turn(上下文串联,只用 [SEP_2] 相连);user/agent 不进 prompt,只决定解码音色

20 个 turn · 拼接 70.94 s · 生成耗时 81.86 s
整段连播(20 句按顺序拼接,句间 0.3 s 静音):
逐句(每句独立音频):

Dialog 3:配偶团体计划生效确认(12 句)

user: I recently got married and got the benefits through my spouse group plan. I want to check if my plan is active. agent: Congratulations! agent: I would be glad to help you! agent: In order to verify who you are, please provide your date of birth. user: Sure. My date of birth is January fourth, nineteen ninety one. agent: Thank you! I can confirm your coverage is effective as of June first, twenty twenty-five, and you are enrolled in the PPO plus plan. agent: Would you like me to provide you a summary of your in-network benefits? user: No. agent: OK, is there anything else I can help you with? user: No, I'm good. agent: Thank you for calling Baymax Health! agent: Have a great day!

每行一个 turn(上下文串联,只用 [SEP_2] 相连);user/agent 不进 prompt,只决定解码音色

12 个 turn · 拼接 161.54 s · 生成耗时 170.34 s
整段连播(12 句按顺序拼接,句间 0.3 s 静音):
逐句(每句独立音频):

每句一个独立音频(mp3 48 kbps,由该句的 WAV 转码);播放任一音频时其他音频自动暂停。原始 24 kHz WAV、result.json 和 prompt/response JSONL 在实验目录 outputs/ 下。