final_asr ckpt10000 对话朗读

final_asr (asr32) checkpoint10000 Instruction 合并(1.0)

用上述模型直接朗读 /proj/mmllm/prosody/texts 的 3 段客服对话(shai_dialog1–3)。无指令、无 caption、无 system prompt:每个 turn 以 <|im_start|>角色\n文本[SEP_1] 输入,模型自由续写 CosyVoice3 音频 token 直到采样出 [SEP_2];同一文本的 turn 按预训练格式串联在上下文中。对话的角色名直接用原文件里的 user / agent。

checkpoint/proj/spans/lyf/checkpoint/asr32-20261020T154649Z/checkpoint-10000 加载方式Instruction 合并 1.0(i+delta):W = W_Qwen3-8B + 1.0 × (W_ckpt10000 − W_Qwen3-8B-Base),新增语音 / 特殊 token 行取 checkpoint 训练值;直接加载已验证的合并模型 采样评测规则:文本 0.1 / 音频 0.7(质量守恒)、top_k 150、top_p 0.9;每 turn 最多 1000 音频 token;seed = turn 序号 解码音色CosyVoice3 flow+HiFT,参考音频只决定音色,语言模型看不到。agent(Baymax Health 客服)句子用 EARS p001 真人男声(愉悦版录音)解码;user 句子用新闻播报女声 转写核对WhisperX large-v2 转写每个 turn,WER 对比输入文本(小写、去标点、连字符拆开)

汇总

文本变体turn正常结束音频 s音频 tokenWER 均值WER 中位WER≤0.2
Dialog 1:急诊室保险覆盖查询每行一个 turn(上下文串联),角色名按原文件:user→user、agent→agent174/17539.9134970.5790.6674/17
Dialog 2:会员卡补发每行一个 turn(上下文串联),角色名按原文件:user→user、agent→agent200/20800.8200200.9291.00/20
Dialog 3:配偶团体计划生效确认每行一个 turn(上下文串联),角色名按原文件:user→user、agent→agent1210/12124.131020.1470.010/12

WER 均值按单句上限 1.0 截断;拼接音频里 turn 之间插入了 0.3 s 静音,仅用于播放。达到 1000 token 上限的 turn 没有生成 [SEP_2],接下一句时在上下文里人为补上 [SEP_2]<|im_end|>(原始输出不改)。这些 turn 的音频一般是先读出句子开头几个词,然后同一个 code 重复几百次(静音、嗡声或拖长的元音)直到上限;每句的“最长同码重复”给出重复次数和 code。

Dialog 1:急诊室保险覆盖查询(17 句)

user: A family member is critically ill and in the emergency room, I need to verify the coverage. agent: I can't imagine how hard this must be for you. agent: I will get the information right away. agent: In order to verify who you are, please provide your date of birth. user: Sure. My date of birth is January fourth, nineteen ninety-one. agent: Thank you. I can confirm your plan has ER coverage with a co-pay of one hundred dollars. agent: There is no deductible or out of pocket for emergency room visits. agent: Would you like me to send you a summary of your coverage through text? user: Yes. agent: Thank you. I will text you the summary of your coverage to the number you are calling from ending with six six five four. user: OK. agent: Please let me know if I can provide any other assistance. user: No, I'm good. agent: I am sorry about your situation. agent: I hope your family member gets better soon. agent: Please reach out with any further concerns. agent: Thank you for calling Baymax Health.

每行一个 turn(上下文串联),角色名按原文件:user→user、agent→agent

17 个 turn · 拼接 544.98 s · 生成耗时 317.62 s
整段连播(17 句按顺序拼接,句间 0.3 s 静音):
逐句(每句独立音频):

Dialog 2:会员卡补发(20 句)

user: I have requested a member ID card multiple times, but have not yet received it. agent: I am sorry you feel frustrated. agent: I will help you get a member ID card right away. agent: In order to verify who you are, I need you to provide your date of birth. user: Sure. My date of birth is January fourth, nineteen ninety-one. agent: Thank you. I will text you a copy of your member ID card to the number you are calling from, ending with six six five four. agent: This ensures you will immediately get your ID. user: OK. agent: I have just sent you the ID card through text. agent: Please confirm if you have received it. user: I got it. agent: Would you also like me to send a physical copy of your ID card to your home address with zip code one one zero zero one? user: Yes. agent: I have sent the ID card. agent: It will arrive at your address within seven business days. agent: Is there anything else I can help you with? user: No. agent: I am sorry it took us so long to respond to your request. agent: Please reach out with any further concerns. agent: Thank you for calling Baymax Health.

每行一个 turn(上下文串联),角色名按原文件:user→user、agent→agent

20 个 turn · 拼接 806.8 s · 生成耗时 470.57 s
整段连播(20 句按顺序拼接,句间 0.3 s 静音):
逐句(每句独立音频):

Dialog 3:配偶团体计划生效确认(12 句)

user: I recently got married and got the benefits through my spouse group plan. I want to check if my plan is active. agent: Congratulations! agent: I would be glad to help you! agent: In order to verify who you are, please provide your date of birth. user: Sure. My date of birth is January fourth, nineteen ninety one. agent: Thank you! I can confirm your coverage is effective as of June first, twenty twenty-five, and you are enrolled in the PPO plus plan. agent: Would you like me to provide you a summary of your in-network benefits? user: No. agent: OK, is there anything else I can help you with? user: No, I'm good. agent: Thank you for calling Baymax Health! agent: Have a great day!

每行一个 turn(上下文串联),角色名按原文件:user→user、agent→agent

12 个 turn · 拼接 127.68 s · 生成耗时 78.85 s
整段连播(12 句按顺序拼接,句间 0.3 s 静音):
逐句(每句独立音频):

每句一个独立音频(mp3 48 kbps,由该句重新解码后的 WAV 转码),变体顶部的播放器是整段拼接;播放任一音频时其他音频自动暂停。原始 24 kHz WAV、result.json 和 prompt/response JSONL 在实验目录 outputs/ 下。