updated pipeline
This commit is contained in:
Plachta
@@ -0,0 +1,43 @@
|
|||||||
|
本仓库的pipeline支持多种声音样本上传方式，您只需根据您所持有的样本选择任意一种或其中几种即可。
|
||||||
|
|
||||||
|
1.`.zip`文件打包的，按角色名排列的短音频，该压缩文件结构应如下所示：
|
||||||
|
```
|
||||||
|
Your-zip-file.zip
|
||||||
|
├───Character_name_1
|
||||||
|
├ ├───xxx.wav
|
||||||
|
├ ├───...
|
||||||
|
├ ├───yyy.mp3
|
||||||
|
├ └───zzz.wav
|
||||||
|
├───Character_name_2
|
||||||
|
├ ├───xxx.wav
|
||||||
|
├ ├───...
|
||||||
|
├ ├───yyy.mp3
|
||||||
|
├ └───zzz.wav
|
||||||
|
├───...
|
||||||
|
├
|
||||||
|
└───Character_name_n
|
||||||
|
├───xxx.wav
|
||||||
|
├───...
|
||||||
|
├───yyy.mp3
|
||||||
|
└───zzz.wav
|
||||||
|
```
|
||||||
|
注意音频的格式和名称都不重要，只要它们是音频文件。
|
||||||
|
质量要求：2秒以上，10秒以内，尽量不要有背景噪音。
|
||||||
|
数量要求：一个角色至少10条，最好每个角色20条以上。
|
||||||
|
2. 以角色名命名的长音频文件，音频内只能有单说话人，背景音会被自动去除。命名格式为：`{CharacterName}_{random_number}.wav`
|
||||||
|
(例如：`Diana_234135.wav`, `MinatoAqua_234252.wav`)，必须是`.wav`文件。
|
||||||
|
|
||||||
|
|
||||||
|
3. 以角色名命名的长视频文件，视频内只能有单说话人，背景音会被自动去除。命名格式为：`{CharacterName}_{random_number}.mp4`
|
||||||
|
(例如：`Taffy_332452.mp4`, `Dingzhen_957315.mp4`)，必须是`.mp4`文件。
|
||||||
|
注意：命名中，`CharacterName`必须是英文字符，`random_number`是为了区分同一个角色的多个文件，必须要添加，该数字可以为0~999999之间的任意整数。
|
||||||
|
|
||||||
|
4. 包含多行`{CharacterName}|{video_url}`的`.txt`文件，格式应如下所示：
|
||||||
|
```
|
||||||
|
Char1|https://xyz.com/video1/
|
||||||
|
Char2|https://xyz.com/video2/
|
||||||
|
Char2|https://xyz.com/video3/
|
||||||
|
Char3|https://xyz.com/video4/
|
||||||
|
```
|
||||||
|
视频内只能有单说话人，背景音会被自动去除。目前仅支持来自bilibili的视频，其它网站视频的url还没测试过。
|
||||||
|
若对格式有疑问，可以在[这里](https://drive.google.com/file/d/132l97zjanpoPY4daLgqXoM7HKXPRbS84/view?usp=sharing)找到所有格式对应的数据样本。
|
||||||
+21
-38
@@ -1,17 +1,22 @@
|
|||||||
English Documentation Please Click [here](https://github.com/Plachtaa/VITS-fast-fine-tuning/blob/main/README_EN.md)
|
English Documentation Please Click [here](https://github.com/Plachtaa/VITS-fast-fine-tuning/blob/main/README_EN.md)
|
||||||
# VITS 快速微调
|
# VITS 快速微调
|
||||||
这个代码库会指导你如何将自定义角色，甚至你自己的声线加入一个现有的VITS模型中，在1小时内的微调使模型具备如下功能：
|
这个代码库会指导你如何将自定义角色（甚至你自己），加入预训练的VITS模型中，在1小时内的微调使模型具备如下功能：
|
||||||
1. 在 你 & 你加入的角色 & 预设角色 之间进行任意声线转换
|
1. 在 模型所包含的任意两个角色 之间进行声线转换
|
||||||
2. 以 你的声线 & 你加入的角色声线 & 预设角色声线 进行中日英三语 文本到语音合成。
|
2. 以 你加入的角色声线 进行中日英三语 文本到语音合成。
|
||||||
|
|
||||||
本项目使用的底模涵盖常见二次元男/女配音声线（来自原神数据集）以及现实世界常见男/女声线（来自VCTK数据集），支持中日英三语，保证能够在微调时快速适应新的声线。
|
本项目使用的底模涵盖常见二次元男/女配音声线（来自原神数据集）以及现实世界常见男/女声线（来自VCTK数据集），支持中日英三语，保证能够在微调时快速适应新的声线。
|
||||||
|
|
||||||
欢迎体验微调所使用的底模！
|
欢迎体验微调所使用的底模！
|
||||||
[](https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer)
|
|
||||||
|
中日英：[](https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer)
|
||||||
|
|
||||||
|
中日：[](https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai)
|
||||||
|
|
||||||
### 目前支持的任务:
|
### 目前支持的任务:
|
||||||
- [x] 转换用户声线到 [这些角色](https://github.com/SongtingLiu/VITS_voice_conversion/blob/main/configs/finetune_speaker.json)
|
- [x] 从 10条以上的短音频 克隆角色声音
|
||||||
- [x] 自定义角色的中日英三语TTS！
|
- [x] 从 3分钟以上的长音频 克隆角色声音
|
||||||
|
- [x] 从 3分钟以上的视频（只包含单说话人） 克隆角色声音
|
||||||
|
- [x] 通过输入 bilibili视频链接（只包含单说话人） 克隆角色声音
|
||||||
|
|
||||||
### 目前支持声线转换和中日英三语TTS的角色
|
### 目前支持声线转换和中日英三语TTS的角色
|
||||||
- [x] 赛马娘 （仅已实装角色）（预训练时使用的角色）
|
- [x] 赛马娘 （仅已实装角色）（预训练时使用的角色）
|
||||||
@@ -26,36 +31,10 @@ English Documentation Please Click [here](https://github.com/Plachtaa/VITS-fast-
|
|||||||
建议使用 [Google Colab](https://colab.research.google.com/drive/1omMhfYKrAAQ7a6zOCsyqpla-wU-QyfZn?usp=sharing)
|
建议使用 [Google Colab](https://colab.research.google.com/drive/1omMhfYKrAAQ7a6zOCsyqpla-wU-QyfZn?usp=sharing)
|
||||||
进行微调任务，因为VITS在多语言情况下的某些环境依赖相当难以配置。
|
进行微调任务，因为VITS在多语言情况下的某些环境依赖相当难以配置。
|
||||||
### 在Google Colab里，我需要花多长时间？
|
### 在Google Colab里，我需要花多长时间？
|
||||||
1. 安装依赖 (2 min)
|
1. 安装依赖 (3 min)
|
||||||
2. 录入你自己的声音，阅读内容会在UI中提供，每句不超过20个字。 (5~10 min)
|
2. 选择预训练模型，详细区别参见Colab笔记本页面。
|
||||||
3. 上传你希望加入的其它角色声音，用一个`.zip`文件打包
|
3. 上传你希望加入的其它角色声音，详细上传方式见[DATA.MD]()
|
||||||
文件结构应该如下所示:
|
4. 进行微调，根据选择的微调方式和样本数量不同，花费时长可能在20分钟到2小时不等。
|
||||||
```
|
|
||||||
Your-zip-file.zip
|
|
||||||
├───Character_name_1
|
|
||||||
├ ├───xxx.wav
|
|
||||||
├ ├───...
|
|
||||||
├ ├───yyy.mp3
|
|
||||||
├ └───zzz.wav
|
|
||||||
├───Character_name_2
|
|
||||||
├ ├───xxx.wav
|
|
||||||
├ ├───...
|
|
||||||
├ ├───yyy.mp3
|
|
||||||
├ └───zzz.wav
|
|
||||||
├───...
|
|
||||||
├
|
|
||||||
└───Character_name_n
|
|
||||||
├───xxx.wav
|
|
||||||
├───...
|
|
||||||
├───yyy.mp3
|
|
||||||
└───zzz.wav
|
|
||||||
```
|
|
||||||
注意音频的格式和名称都不重要，只要它们是音频文件。
|
|
||||||
质量要求：2秒以上，20秒以内，尽量不要有背景噪音。
|
|
||||||
数量要求：一个角色至少10条，最好每个角色20条以上。
|
|
||||||
你可以选择进行步骤2或3，或二者一起，取决于你的需求。
|
|
||||||
|
|
||||||
4. 进行微调 (30 min)
|
|
||||||
|
|
||||||
微调结束后可以直接下载微调好的模型，日后在本地运行（不需要GPU）
|
微调结束后可以直接下载微调好的模型，日后在本地运行（不需要GPU）
|
||||||
|
|
||||||
@@ -64,7 +43,7 @@ Your-zip-file.zip
|
|||||||
1. 下载最新的Release包（在Github页面的右侧）
|
1. 下载最新的Release包（在Github页面的右侧）
|
||||||
2. 把下载的模型和config文件放在 `inference`文件夹下, 确保模型的文件名为 `G_latest.pth` ，config文件名为 `finetune_speaker.json`
|
2. 把下载的模型和config文件放在 `inference`文件夹下, 确保模型的文件名为 `G_latest.pth` ，config文件名为 `finetune_speaker.json`
|
||||||
3. 一切准备就绪后，文件结构应该如下所示:
|
3. 一切准备就绪后，文件结构应该如下所示:
|
||||||
```shell
|
```
|
||||||
inference
|
inference
|
||||||
├───inference.exe
|
├───inference.exe
|
||||||
├───...
|
├───...
|
||||||
@@ -73,3 +52,7 @@ inference
|
|||||||
```
|
```
|
||||||
4. 运行 `inference.exe`, 浏览器会自动弹出窗口, 注意其所在路径不能有中文字符或者空格.
|
4. 运行 `inference.exe`, 浏览器会自动弹出窗口, 注意其所在路径不能有中文字符或者空格.
|
||||||
|
|
||||||
|
## 在MoeGoe使用
|
||||||
|
0. MoeGoe以及类似其它VITS推理UI使用的config格式略有不同，需要下载的文件为模型`G_latest.pth`和配置文件`moegoe_config.json`
|
||||||
|
1. 按照[MoeGoe](https://github.com/CjangCjengh/MoeGoe)页面的提示配置路径即可使用。
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,19 @@
|
|||||||
|
import text
|
||||||
|
|
||||||
|
with open("custom_character_anno.txt", 'r', encoding='utf-8') as f:
|
||||||
|
speaker_annos = f.readlines()
|
||||||
|
# clean annotation
|
||||||
|
cleaned_speaker_annos = []
|
||||||
|
for i, line in enumerate(speaker_annos):
|
||||||
|
path, sid, txt = line.split("|")
|
||||||
|
if len(txt) > 100:
|
||||||
|
continue
|
||||||
|
cleaned_text = text._clean_text(txt, ["cjke_cleaners2"])
|
||||||
|
cleaned_text += "\n" if not cleaned_text.endswith("\n") else ""
|
||||||
|
cleaned_speaker_annos.append(path + "|" + sid + "|" + cleaned_text)
|
||||||
|
# write into annotation
|
||||||
|
with open("custom_character_anno.txt", 'w', encoding='utf-8') as f:
|
||||||
|
for line in speaker_annos:
|
||||||
|
f.write(line)
|
||||||
|
|
||||||
|
|
||||||
@@ -4,9 +4,12 @@
|
|||||||
"eval_interval": 1000,
|
"eval_interval": 1000,
|
||||||
"seed": 1234,
|
"seed": 1234,
|
||||||
"epochs": 10000,
|
"epochs": 10000,
|
||||||
"learning_rate": 2e-4,
|
"learning_rate": 0.0002,
|
||||||
"betas": [0.8, 0.99],
|
"betas": [
|
||||||
"eps": 1e-9,
|
0.8,
|
||||||
|
0.99
|
||||||
|
],
|
||||||
|
"eps": 1e-09,
|
||||||
"batch_size": 12,
|
"batch_size": 12,
|
||||||
"fp16_run": true,
|
"fp16_run": true,
|
||||||
"lr_decay": 0.999875,
|
"lr_decay": 0.999875,
|
||||||
@@ -17,9 +20,11 @@
|
|||||||
"c_kl": 1.0
|
"c_kl": 1.0
|
||||||
},
|
},
|
||||||
"data": {
|
"data": {
|
||||||
"training_files":"final_annotation_train.txt",
|
"training_files": "final_annotation_train.txt",
|
||||||
"validation_files":"final_annotation_val.txt",
|
"validation_files": "final_annotation_val.txt",
|
||||||
"text_cleaners":["cjke_cleaners2"],
|
"text_cleaners": [
|
||||||
|
"cjke_cleaners2"
|
||||||
|
],
|
||||||
"max_wav_value": 32768.0,
|
"max_wav_value": 32768.0,
|
||||||
"sampling_rate": 22050,
|
"sampling_rate": 22050,
|
||||||
"filter_length": 1024,
|
"filter_length": 1024,
|
||||||
@@ -29,7 +34,7 @@
|
|||||||
"mel_fmin": 0.0,
|
"mel_fmin": 0.0,
|
||||||
"mel_fmax": null,
|
"mel_fmax": null,
|
||||||
"add_blank": true,
|
"add_blank": true,
|
||||||
"n_speakers": 1001,
|
"n_speakers": 1003,
|
||||||
"cleaned_text": true
|
"cleaned_text": true
|
||||||
},
|
},
|
||||||
"model": {
|
"model": {
|
||||||
@@ -41,164 +46,266 @@
|
|||||||
"kernel_size": 3,
|
"kernel_size": 3,
|
||||||
"p_dropout": 0.1,
|
"p_dropout": 0.1,
|
||||||
"resblock": "1",
|
"resblock": "1",
|
||||||
"resblock_kernel_sizes": [3,7,11],
|
"resblock_kernel_sizes": [
|
||||||
"resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]],
|
3,
|
||||||
"upsample_rates": [8,8,2,2],
|
7,
|
||||||
|
11
|
||||||
|
],
|
||||||
|
"resblock_dilation_sizes": [
|
||||||
|
[
|
||||||
|
1,
|
||||||
|
3,
|
||||||
|
5
|
||||||
|
],
|
||||||
|
[
|
||||||
|
1,
|
||||||
|
3,
|
||||||
|
5
|
||||||
|
],
|
||||||
|
[
|
||||||
|
1,
|
||||||
|
3,
|
||||||
|
5
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"upsample_rates": [
|
||||||
|
8,
|
||||||
|
8,
|
||||||
|
2,
|
||||||
|
2
|
||||||
|
],
|
||||||
"upsample_initial_channel": 512,
|
"upsample_initial_channel": 512,
|
||||||
"upsample_kernel_sizes": [16,16,4,4],
|
"upsample_kernel_sizes": [
|
||||||
|
16,
|
||||||
|
16,
|
||||||
|
4,
|
||||||
|
4
|
||||||
|
],
|
||||||
"n_layers_q": 3,
|
"n_layers_q": 3,
|
||||||
"use_spectral_norm": false,
|
"use_spectral_norm": false,
|
||||||
"gin_channels": 256
|
"gin_channels": 256
|
||||||
},
|
},
|
||||||
"symbols": ["_", ",", ".", "!", "?", "-", "~", "\u2026", "N", "Q", "a", "b", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "s", "t", "u", "v", "w", "x", "y", "z", "\u0251", "\u00e6", "\u0283", "\u0291", "\u00e7", "\u026f", "\u026a", "\u0254", "\u025b", "\u0279", "\u00f0", "\u0259", "\u026b", "\u0265", "\u0278", "\u028a", "\u027e", "\u0292", "\u03b8", "\u03b2", "\u014b", "\u0266", "\u207c", "\u02b0", "`", "^", "#", "*", "=", "\u02c8", "\u02cc", "\u2192", "\u2193", "\u2191", " "],
|
"symbols": [
|
||||||
"speakers": {"特别周 Special Week (Umamusume Pretty Derby)": 0,
|
"_",
|
||||||
"无声铃鹿 Silence Suzuka (Umamusume Pretty Derby)": 1,
|
",",
|
||||||
"东海帝王 Tokai Teio (Umamusume Pretty Derby)": 2,
|
".",
|
||||||
"丸善斯基 Maruzensky (Umamusume Pretty Derby)": 3,
|
"!",
|
||||||
"富士奇迹 Fuji Kiseki (Umamusume Pretty Derby)": 4,
|
"?",
|
||||||
"小栗帽 Oguri Cap (Umamusume Pretty Derby)": 5,
|
"-",
|
||||||
"黄金船 Gold Ship (Umamusume Pretty Derby)": 6,
|
"~",
|
||||||
"伏特加 Vodka (Umamusume Pretty Derby)": 7,
|
"\u2026",
|
||||||
"大和赤骥 Daiwa Scarlet (Umamusume Pretty Derby)": 8,
|
"N",
|
||||||
"大树快车 Taiki Shuttle (Umamusume Pretty Derby)": 9,
|
"Q",
|
||||||
"草上飞 Grass Wonder (Umamusume Pretty Derby)": 10,
|
"a",
|
||||||
"菱亚马逊 Hishi Amazon (Umamusume Pretty Derby)": 11,
|
"b",
|
||||||
"目白麦昆 Mejiro Mcqueen (Umamusume Pretty Derby)": 12,
|
"d",
|
||||||
"神鹰 El Condor Pasa (Umamusume Pretty Derby)": 13,
|
"e",
|
||||||
"好歌剧 T.M. Opera O (Umamusume Pretty Derby)": 14,
|
"f",
|
||||||
"成田白仁 Narita Brian (Umamusume Pretty Derby)": 15,
|
"g",
|
||||||
"鲁道夫象征 Symboli Rudolf (Umamusume Pretty Derby)": 16,
|
"h",
|
||||||
"气槽 Air Groove (Umamusume Pretty Derby)": 17,
|
"i",
|
||||||
"爱丽数码 Agnes Digital (Umamusume Pretty Derby)": 18,
|
"j",
|
||||||
"青云天空 Seiun Sky (Umamusume Pretty Derby)": 19,
|
"k",
|
||||||
"玉藻十字 Tamamo Cross (Umamusume Pretty Derby)": 20,
|
"l",
|
||||||
"美妙姿势 Fine Motion (Umamusume Pretty Derby)": 21,
|
"m",
|
||||||
"琵琶晨光 Biwa Hayahide (Umamusume Pretty Derby)": 22,
|
"n",
|
||||||
"重炮 Mayano Topgun (Umamusume Pretty Derby)": 23,
|
"o",
|
||||||
"曼城茶座 Manhattan Cafe (Umamusume Pretty Derby)": 24,
|
"p",
|
||||||
"美普波旁 Mihono Bourbon (Umamusume Pretty Derby)": 25,
|
"s",
|
||||||
"目白雷恩 Mejiro Ryan (Umamusume Pretty Derby)": 26,
|
"t",
|
||||||
"雪之美人 Yukino Bijin (Umamusume Pretty Derby)": 28,
|
"u",
|
||||||
"米浴 Rice Shower (Umamusume Pretty Derby)": 29,
|
"v",
|
||||||
"艾尼斯风神 Ines Fujin (Umamusume Pretty Derby)": 30,
|
"w",
|
||||||
"爱丽速子 Agnes Tachyon (Umamusume Pretty Derby)": 31,
|
"x",
|
||||||
"爱慕织姬 Admire Vega (Umamusume Pretty Derby)": 32,
|
"y",
|
||||||
"稻荷一 Inari One (Umamusume Pretty Derby)": 33,
|
"z",
|
||||||
"胜利奖券 Winning Ticket (Umamusume Pretty Derby)": 34,
|
"\u0251",
|
||||||
"空中神宫 Air Shakur (Umamusume Pretty Derby)": 35,
|
"\u00e6",
|
||||||
"荣进闪耀 Eishin Flash (Umamusume Pretty Derby)": 36,
|
"\u0283",
|
||||||
"真机伶 Curren Chan (Umamusume Pretty Derby)": 37,
|
"\u0291",
|
||||||
"川上公主 Kawakami Princess (Umamusume Pretty Derby)": 38,
|
"\u00e7",
|
||||||
"黄金城市 Gold City (Umamusume Pretty Derby)": 39,
|
"\u026f",
|
||||||
"樱花进王 Sakura Bakushin O (Umamusume Pretty Derby)": 40,
|
"\u026a",
|
||||||
"采珠 Seeking the Pearl (Umamusume Pretty Derby)": 41,
|
"\u0254",
|
||||||
"新光风 Shinko Windy (Umamusume Pretty Derby)": 42,
|
"\u025b",
|
||||||
"东商变革 Sweep Tosho (Umamusume Pretty Derby)": 43,
|
"\u0279",
|
||||||
"超级小溪 Super Creek (Umamusume Pretty Derby)": 44,
|
"\u00f0",
|
||||||
"醒目飞鹰 Smart Falcon (Umamusume Pretty Derby)": 45,
|
"\u0259",
|
||||||
"荒漠英雄 Zenno Rob Roy (Umamusume Pretty Derby)": 46,
|
"\u026b",
|
||||||
"东瀛佐敦 Tosen Jordan (Umamusume Pretty Derby)": 47,
|
"\u0265",
|
||||||
"中山庆典 Nakayama Festa (Umamusume Pretty Derby)": 48,
|
"\u0278",
|
||||||
"成田大进 Narita Taishin (Umamusume Pretty Derby)": 49,
|
"\u028a",
|
||||||
"西野花 Nishino Flower (Umamusume Pretty Derby)": 50,
|
"\u027e",
|
||||||
"春乌拉拉 Haru Urara (Umamusume Pretty Derby)": 51,
|
"\u0292",
|
||||||
"青竹回忆 Bamboo Memory (Umamusume Pretty Derby)": 52,
|
"\u03b8",
|
||||||
"待兼福来 Matikane Fukukitaru (Umamusume Pretty Derby)": 55,
|
"\u03b2",
|
||||||
"名将怒涛 Meisho Doto (Umamusume Pretty Derby)": 57,
|
"\u014b",
|
||||||
"目白多伯 Mejiro Dober (Umamusume Pretty Derby)": 58,
|
"\u0266",
|
||||||
"优秀素质 Nice Nature (Umamusume Pretty Derby)": 59,
|
"\u207c",
|
||||||
"帝王光环 King Halo (Umamusume Pretty Derby)": 60,
|
"\u02b0",
|
||||||
"待兼诗歌剧 Matikane Tannhauser (Umamusume Pretty Derby)": 61,
|
"`",
|
||||||
"生野狄杜斯 Ikuno Dictus (Umamusume Pretty Derby)": 62,
|
"^",
|
||||||
"目白善信 Mejiro Palmer (Umamusume Pretty Derby)": 63,
|
"#",
|
||||||
"大拓太阳神 Daitaku Helios (Umamusume Pretty Derby)": 64,
|
"*",
|
||||||
"双涡轮 Twin Turbo (Umamusume Pretty Derby)": 65,
|
"=",
|
||||||
"里见光钻 Satono Diamond (Umamusume Pretty Derby)": 66,
|
"\u02c8",
|
||||||
"北部玄驹 Kitasan Black (Umamusume Pretty Derby)": 67,
|
"\u02cc",
|
||||||
"樱花千代王 Sakura Chiyono O (Umamusume Pretty Derby)": 68,
|
"\u2192",
|
||||||
"天狼星象征 Sirius Symboli (Umamusume Pretty Derby)": 69,
|
"\u2193",
|
||||||
"目白阿尔丹 Mejiro Ardan (Umamusume Pretty Derby)": 70,
|
"\u2191",
|
||||||
"八重无敌 Yaeno Muteki (Umamusume Pretty Derby)": 71,
|
" "
|
||||||
"鹤丸刚志 Tsurumaru Tsuyoshi (Umamusume Pretty Derby)": 72,
|
],
|
||||||
"目白光明 Mejiro Bright (Umamusume Pretty Derby)": 73,
|
"speakers": {
|
||||||
"樱花桂冠 Sakura Laurel (Umamusume Pretty Derby)": 74,
|
"\u7279\u522b\u5468 Special Week (Umamusume Pretty Derby)": 0,
|
||||||
"成田路 Narita Top Road (Umamusume Pretty Derby)": 75,
|
"\u65e0\u58f0\u94c3\u9e7f Silence Suzuka (Umamusume Pretty Derby)": 1,
|
||||||
"也文摄辉 Yamanin Zephyr (Umamusume Pretty Derby)": 76,
|
"\u4e1c\u6d77\u5e1d\u738b Tokai Teio (Umamusume Pretty Derby)": 2,
|
||||||
"真弓快车 Aston Machan (Umamusume Pretty Derby)": 80,
|
"\u4e38\u5584\u65af\u57fa Maruzensky (Umamusume Pretty Derby)": 3,
|
||||||
"骏川手纲 Hayakawa Tazuna (Umamusume Pretty Derby)": 81,
|
"\u5bcc\u58eb\u5947\u8ff9 Fuji Kiseki (Umamusume Pretty Derby)": 4,
|
||||||
"小林历奇 Kopano Rickey (Umamusume Pretty Derby)": 83,
|
"\u5c0f\u6817\u5e3d Oguri Cap (Umamusume Pretty Derby)": 5,
|
||||||
"奇锐骏 Wonder Acute (Umamusume Pretty Derby)": 85,
|
"\u9ec4\u91d1\u8239 Gold Ship (Umamusume Pretty Derby)": 6,
|
||||||
"秋川理事长 President Akikawa (Umamusume Pretty Derby)": 86,
|
"\u4f0f\u7279\u52a0 Vodka (Umamusume Pretty Derby)": 7,
|
||||||
"綾地 寧々 Ayachi Nene (Sanoba Witch)": 87,
|
"\u5927\u548c\u8d64\u9aa5 Daiwa Scarlet (Umamusume Pretty Derby)": 8,
|
||||||
"因幡 めぐる Inaba Meguru (Sanoba Witch)": 88,
|
"\u5927\u6811\u5feb\u8f66 Taiki Shuttle (Umamusume Pretty Derby)": 9,
|
||||||
"椎葉 紬 Shiiba Tsumugi (Sanoba Witch)": 89,
|
"\u8349\u4e0a\u98de Grass Wonder (Umamusume Pretty Derby)": 10,
|
||||||
"仮屋 和奏 Kariya Wakama (Sanoba Witch)": 90,
|
"\u83f1\u4e9a\u9a6c\u900a Hishi Amazon (Umamusume Pretty Derby)": 11,
|
||||||
"戸隠 憧子 Togakushi Touko (Sanoba Witch)": 91,
|
"\u76ee\u767d\u9ea6\u6606 Mejiro Mcqueen (Umamusume Pretty Derby)": 12,
|
||||||
"九条裟罗 Kujou Sara (Genshin Impact)": 92,
|
"\u795e\u9e70 El Condor Pasa (Umamusume Pretty Derby)": 13,
|
||||||
"芭芭拉 Barbara (Genshin Impact)": 93,
|
"\u597d\u6b4c\u5267 T.M. Opera O (Umamusume Pretty Derby)": 14,
|
||||||
"派蒙 Paimon (Genshin Impact)": 94,
|
"\u6210\u7530\u767d\u4ec1 Narita Brian (Umamusume Pretty Derby)": 15,
|
||||||
"荒泷一斗 Arataki Itto (Genshin Impact)": 96,
|
"\u9c81\u9053\u592b\u8c61\u5f81 Symboli Rudolf (Umamusume Pretty Derby)": 16,
|
||||||
"早柚 Sayu (Genshin Impact)": 97,
|
"\u6c14\u69fd Air Groove (Umamusume Pretty Derby)": 17,
|
||||||
"香菱 Xiangling (Genshin Impact)": 98,
|
"\u7231\u4e3d\u6570\u7801 Agnes Digital (Umamusume Pretty Derby)": 18,
|
||||||
"神里绫华 Kamisato Ayaka (Genshin Impact)": 99,
|
"\u9752\u4e91\u5929\u7a7a Seiun Sky (Umamusume Pretty Derby)": 19,
|
||||||
"重云 Chongyun (Genshin Impact)": 100,
|
"\u7389\u85fb\u5341\u5b57 Tamamo Cross (Umamusume Pretty Derby)": 20,
|
||||||
"流浪者 Wanderer (Genshin Impact)": 102,
|
"\u7f8e\u5999\u59ff\u52bf Fine Motion (Umamusume Pretty Derby)": 21,
|
||||||
"优菈 Eula (Genshin Impact)": 103,
|
"\u7435\u7436\u6668\u5149 Biwa Hayahide (Umamusume Pretty Derby)": 22,
|
||||||
"凝光 Ningguang (Genshin Impact)": 105,
|
"\u91cd\u70ae Mayano Topgun (Umamusume Pretty Derby)": 23,
|
||||||
"钟离 Zhongli (Genshin Impact)": 106,
|
"\u66fc\u57ce\u8336\u5ea7 Manhattan Cafe (Umamusume Pretty Derby)": 24,
|
||||||
"雷电将军 Raiden Shogun (Genshin Impact)": 107,
|
"\u7f8e\u666e\u6ce2\u65c1 Mihono Bourbon (Umamusume Pretty Derby)": 25,
|
||||||
"枫原万叶 Kaedehara Kazuha (Genshin Impact)": 108,
|
"\u76ee\u767d\u96f7\u6069 Mejiro Ryan (Umamusume Pretty Derby)": 26,
|
||||||
"赛诺 Cyno (Genshin Impact)": 109,
|
"\u96ea\u4e4b\u7f8e\u4eba Yukino Bijin (Umamusume Pretty Derby)": 28,
|
||||||
"诺艾尔 Noelle (Genshin Impact)": 112,
|
"\u7c73\u6d74 Rice Shower (Umamusume Pretty Derby)": 29,
|
||||||
"八重神子 Yae Miko (Genshin Impact)": 113,
|
"\u827e\u5c3c\u65af\u98ce\u795e Ines Fujin (Umamusume Pretty Derby)": 30,
|
||||||
"凯亚 Kaeya (Genshin Impact)": 114,
|
"\u7231\u4e3d\u901f\u5b50 Agnes Tachyon (Umamusume Pretty Derby)": 31,
|
||||||
"魈 Xiao (Genshin Impact)": 115,
|
"\u7231\u6155\u7ec7\u59ec Admire Vega (Umamusume Pretty Derby)": 32,
|
||||||
"托马 Thoma (Genshin Impact)": 116,
|
"\u7a3b\u8377\u4e00 Inari One (Umamusume Pretty Derby)": 33,
|
||||||
"可莉 Klee (Genshin Impact)": 117,
|
"\u80dc\u5229\u5956\u5238 Winning Ticket (Umamusume Pretty Derby)": 34,
|
||||||
"迪卢克 Diluc (Genshin Impact)": 120,
|
"\u7a7a\u4e2d\u795e\u5bab Air Shakur (Umamusume Pretty Derby)": 35,
|
||||||
"夜兰 Yelan (Genshin Impact)": 121,
|
"\u8363\u8fdb\u95ea\u8000 Eishin Flash (Umamusume Pretty Derby)": 36,
|
||||||
"鹿野院平藏 Shikanoin Heizou (Genshin Impact)": 123,
|
"\u771f\u673a\u4f36 Curren Chan (Umamusume Pretty Derby)": 37,
|
||||||
"辛焱 Xinyan (Genshin Impact)": 124,
|
"\u5ddd\u4e0a\u516c\u4e3b Kawakami Princess (Umamusume Pretty Derby)": 38,
|
||||||
"丽莎 Lisa (Genshin Impact)": 125,
|
"\u9ec4\u91d1\u57ce\u5e02 Gold City (Umamusume Pretty Derby)": 39,
|
||||||
"云堇 Yun Jin (Genshin Impact)": 126,
|
"\u6a31\u82b1\u8fdb\u738b Sakura Bakushin O (Umamusume Pretty Derby)": 40,
|
||||||
"坎蒂丝 Candace (Genshin Impact)": 127,
|
"\u91c7\u73e0 Seeking the Pearl (Umamusume Pretty Derby)": 41,
|
||||||
"罗莎莉亚 Rosaria (Genshin Impact)": 128,
|
"\u65b0\u5149\u98ce Shinko Windy (Umamusume Pretty Derby)": 42,
|
||||||
"北斗 Beidou (Genshin Impact)": 129,
|
"\u4e1c\u5546\u53d8\u9769 Sweep Tosho (Umamusume Pretty Derby)": 43,
|
||||||
"珊瑚宫心海 Sangonomiya Kokomi (Genshin Impact)": 132,
|
"\u8d85\u7ea7\u5c0f\u6eaa Super Creek (Umamusume Pretty Derby)": 44,
|
||||||
"烟绯 Yanfei (Genshin Impact)": 133,
|
"\u9192\u76ee\u98de\u9e70 Smart Falcon (Umamusume Pretty Derby)": 45,
|
||||||
"久岐忍 Kuki Shinobu (Genshin Impact)": 136,
|
"\u8352\u6f20\u82f1\u96c4 Zenno Rob Roy (Umamusume Pretty Derby)": 46,
|
||||||
"宵宫 Yoimiya (Genshin Impact)": 139,
|
"\u4e1c\u701b\u4f50\u6566 Tosen Jordan (Umamusume Pretty Derby)": 47,
|
||||||
"安柏 Amber (Genshin Impact)": 143,
|
"\u4e2d\u5c71\u5e86\u5178 Nakayama Festa (Umamusume Pretty Derby)": 48,
|
||||||
"迪奥娜 Diona (Genshin Impact)": 144,
|
"\u6210\u7530\u5927\u8fdb Narita Taishin (Umamusume Pretty Derby)": 49,
|
||||||
"班尼特 Bennett (Genshin Impact)": 146,
|
"\u897f\u91ce\u82b1 Nishino Flower (Umamusume Pretty Derby)": 50,
|
||||||
"雷泽 Razor (Genshin Impact)": 147,
|
"\u6625\u4e4c\u62c9\u62c9 Haru Urara (Umamusume Pretty Derby)": 51,
|
||||||
"阿贝多 Albedo (Genshin Impact)": 151,
|
"\u9752\u7af9\u56de\u5fc6 Bamboo Memory (Umamusume Pretty Derby)": 52,
|
||||||
"温迪 Venti (Genshin Impact)": 152,
|
"\u5f85\u517c\u798f\u6765 Matikane Fukukitaru (Umamusume Pretty Derby)": 55,
|
||||||
"空 Player Male (Genshin Impact)": 153,
|
"\u540d\u5c06\u6012\u6d9b Meisho Doto (Umamusume Pretty Derby)": 57,
|
||||||
"神里绫人 Kamisato Ayato (Genshin Impact)": 154,
|
"\u76ee\u767d\u591a\u4f2f Mejiro Dober (Umamusume Pretty Derby)": 58,
|
||||||
"琴 Jean (Genshin Impact)": 155,
|
"\u4f18\u79c0\u7d20\u8d28 Nice Nature (Umamusume Pretty Derby)": 59,
|
||||||
"艾尔海森 Alhaitham (Genshin Impact)": 156,
|
"\u5e1d\u738b\u5149\u73af King Halo (Umamusume Pretty Derby)": 60,
|
||||||
"莫娜 Mona (Genshin Impact)": 157,
|
"\u5f85\u517c\u8bd7\u6b4c\u5267 Matikane Tannhauser (Umamusume Pretty Derby)": 61,
|
||||||
"妮露 Nilou (Genshin Impact)": 159,
|
"\u751f\u91ce\u72c4\u675c\u65af Ikuno Dictus (Umamusume Pretty Derby)": 62,
|
||||||
"胡桃 Hu Tao (Genshin Impact)": 160,
|
"\u76ee\u767d\u5584\u4fe1 Mejiro Palmer (Umamusume Pretty Derby)": 63,
|
||||||
"甘雨 Ganyu (Genshin Impact)": 161,
|
"\u5927\u62d3\u592a\u9633\u795e Daitaku Helios (Umamusume Pretty Derby)": 64,
|
||||||
"纳西妲 Nahida (Genshin Impact)": 162,
|
"\u53cc\u6da1\u8f6e Twin Turbo (Umamusume Pretty Derby)": 65,
|
||||||
"刻晴 Keqing (Genshin Impact)": 165,
|
"\u91cc\u89c1\u5149\u94bb Satono Diamond (Umamusume Pretty Derby)": 66,
|
||||||
"荧 Player Female (Genshin Impact)": 169,
|
"\u5317\u90e8\u7384\u9a79 Kitasan Black (Umamusume Pretty Derby)": 67,
|
||||||
"埃洛伊 Aloy (Genshin Impact)": 179,
|
"\u6a31\u82b1\u5343\u4ee3\u738b Sakura Chiyono O (Umamusume Pretty Derby)": 68,
|
||||||
"柯莱 Collei (Genshin Impact)": 182,
|
"\u5929\u72fc\u661f\u8c61\u5f81 Sirius Symboli (Umamusume Pretty Derby)": 69,
|
||||||
"多莉 Dori (Genshin Impact)": 184,
|
"\u76ee\u767d\u963f\u5c14\u4e39 Mejiro Ardan (Umamusume Pretty Derby)": 70,
|
||||||
"提纳里 Tighnari (Genshin Impact)": 186,
|
"\u516b\u91cd\u65e0\u654c Yaeno Muteki (Umamusume Pretty Derby)": 71,
|
||||||
"砂糖 Sucrose (Genshin Impact)": 188,
|
"\u9e64\u4e38\u521a\u5fd7 Tsurumaru Tsuyoshi (Umamusume Pretty Derby)": 72,
|
||||||
"行秋 Xingqiu (Genshin Impact)": 190,
|
"\u76ee\u767d\u5149\u660e Mejiro Bright (Umamusume Pretty Derby)": 73,
|
||||||
"奥兹 Oz (Genshin Impact)": 193,
|
"\u6a31\u82b1\u6842\u51a0 Sakura Laurel (Umamusume Pretty Derby)": 74,
|
||||||
"五郎 Gorou (Genshin Impact)": 198,
|
"\u6210\u7530\u8def Narita Top Road (Umamusume Pretty Derby)": 75,
|
||||||
"达达利亚 Tartalia (Genshin Impact)": 202,
|
"\u4e5f\u6587\u6444\u8f89 Yamanin Zephyr (Umamusume Pretty Derby)": 76,
|
||||||
"七七 Qiqi (Genshin Impact)": 207,
|
"\u771f\u5f13\u5feb\u8f66 Aston Machan (Umamusume Pretty Derby)": 80,
|
||||||
"申鹤 Shenhe (Genshin Impact)": 217,
|
"\u9a8f\u5ddd\u624b\u7eb2 Hayakawa Tazuna (Umamusume Pretty Derby)": 81,
|
||||||
"莱依拉 Layla (Genshin Impact)": 228,
|
"\u5c0f\u6797\u5386\u5947 Kopano Rickey (Umamusume Pretty Derby)": 83,
|
||||||
"菲谢尔 Fishl (Genshin Impact)": 230,
|
"\u5947\u9510\u9a8f Wonder Acute (Umamusume Pretty Derby)": 85,
|
||||||
"User": 999
|
"\u79cb\u5ddd\u7406\u4e8b\u957f President Akikawa (Umamusume Pretty Derby)": 86,
|
||||||
|
"\u7dbe\u5730 \u5be7\u3005 Ayachi Nene (Sanoba Witch)": 87,
|
||||||
|
"\u56e0\u5e61 \u3081\u3050\u308b Inaba Meguru (Sanoba Witch)": 88,
|
||||||
|
"\u690e\u8449 \u7d2c Shiiba Tsumugi (Sanoba Witch)": 89,
|
||||||
|
"\u4eee\u5c4b \u548c\u594f Kariya Wakama (Sanoba Witch)": 90,
|
||||||
|
"\u6238\u96a0 \u61a7\u5b50 Togakushi Touko (Sanoba Witch)": 91,
|
||||||
|
"\u4e5d\u6761\u88df\u7f57 Kujou Sara (Genshin Impact)": 92,
|
||||||
|
"\u82ad\u82ad\u62c9 Barbara (Genshin Impact)": 93,
|
||||||
|
"\u6d3e\u8499 Paimon (Genshin Impact)": 94,
|
||||||
|
"\u8352\u6cf7\u4e00\u6597 Arataki Itto (Genshin Impact)": 96,
|
||||||
|
"\u65e9\u67da Sayu (Genshin Impact)": 97,
|
||||||
|
"\u9999\u83f1 Xiangling (Genshin Impact)": 98,
|
||||||
|
"\u795e\u91cc\u7eeb\u534e Kamisato Ayaka (Genshin Impact)": 99,
|
||||||
|
"\u91cd\u4e91 Chongyun (Genshin Impact)": 100,
|
||||||
|
"\u6d41\u6d6a\u8005 Wanderer (Genshin Impact)": 102,
|
||||||
|
"\u4f18\u83c8 Eula (Genshin Impact)": 103,
|
||||||
|
"\u51dd\u5149 Ningguang (Genshin Impact)": 105,
|
||||||
|
"\u949f\u79bb Zhongli (Genshin Impact)": 106,
|
||||||
|
"\u96f7\u7535\u5c06\u519b Raiden Shogun (Genshin Impact)": 107,
|
||||||
|
"\u67ab\u539f\u4e07\u53f6 Kaedehara Kazuha (Genshin Impact)": 108,
|
||||||
|
"\u8d5b\u8bfa Cyno (Genshin Impact)": 109,
|
||||||
|
"\u8bfa\u827e\u5c14 Noelle (Genshin Impact)": 112,
|
||||||
|
"\u516b\u91cd\u795e\u5b50 Yae Miko (Genshin Impact)": 113,
|
||||||
|
"\u51ef\u4e9a Kaeya (Genshin Impact)": 114,
|
||||||
|
"\u9b48 Xiao (Genshin Impact)": 115,
|
||||||
|
"\u6258\u9a6c Thoma (Genshin Impact)": 116,
|
||||||
|
"\u53ef\u8389 Klee (Genshin Impact)": 117,
|
||||||
|
"\u8fea\u5362\u514b Diluc (Genshin Impact)": 120,
|
||||||
|
"\u591c\u5170 Yelan (Genshin Impact)": 121,
|
||||||
|
"\u9e7f\u91ce\u9662\u5e73\u85cf Shikanoin Heizou (Genshin Impact)": 123,
|
||||||
|
"\u8f9b\u7131 Xinyan (Genshin Impact)": 124,
|
||||||
|
"\u4e3d\u838e Lisa (Genshin Impact)": 125,
|
||||||
|
"\u4e91\u5807 Yun Jin (Genshin Impact)": 126,
|
||||||
|
"\u574e\u8482\u4e1d Candace (Genshin Impact)": 127,
|
||||||
|
"\u7f57\u838e\u8389\u4e9a Rosaria (Genshin Impact)": 128,
|
||||||
|
"\u5317\u6597 Beidou (Genshin Impact)": 129,
|
||||||
|
"\u73ca\u745a\u5bab\u5fc3\u6d77 Sangonomiya Kokomi (Genshin Impact)": 132,
|
||||||
|
"\u70df\u7eef Yanfei (Genshin Impact)": 133,
|
||||||
|
"\u4e45\u5c90\u5fcd Kuki Shinobu (Genshin Impact)": 136,
|
||||||
|
"\u5bb5\u5bab Yoimiya (Genshin Impact)": 139,
|
||||||
|
"\u5b89\u67cf Amber (Genshin Impact)": 143,
|
||||||
|
"\u8fea\u5965\u5a1c Diona (Genshin Impact)": 144,
|
||||||
|
"\u73ed\u5c3c\u7279 Bennett (Genshin Impact)": 146,
|
||||||
|
"\u96f7\u6cfd Razor (Genshin Impact)": 147,
|
||||||
|
"\u963f\u8d1d\u591a Albedo (Genshin Impact)": 151,
|
||||||
|
"\u6e29\u8fea Venti (Genshin Impact)": 152,
|
||||||
|
"\u7a7a Player Male (Genshin Impact)": 153,
|
||||||
|
"\u795e\u91cc\u7eeb\u4eba Kamisato Ayato (Genshin Impact)": 154,
|
||||||
|
"\u7434 Jean (Genshin Impact)": 155,
|
||||||
|
"\u827e\u5c14\u6d77\u68ee Alhaitham (Genshin Impact)": 156,
|
||||||
|
"\u83ab\u5a1c Mona (Genshin Impact)": 157,
|
||||||
|
"\u59ae\u9732 Nilou (Genshin Impact)": 159,
|
||||||
|
"\u80e1\u6843 Hu Tao (Genshin Impact)": 160,
|
||||||
|
"\u7518\u96e8 Ganyu (Genshin Impact)": 161,
|
||||||
|
"\u7eb3\u897f\u59b2 Nahida (Genshin Impact)": 162,
|
||||||
|
"\u523b\u6674 Keqing (Genshin Impact)": 165,
|
||||||
|
"\u8367 Player Female (Genshin Impact)": 169,
|
||||||
|
"\u57c3\u6d1b\u4f0a Aloy (Genshin Impact)": 179,
|
||||||
|
"\u67ef\u83b1 Collei (Genshin Impact)": 182,
|
||||||
|
"\u591a\u8389 Dori (Genshin Impact)": 184,
|
||||||
|
"\u63d0\u7eb3\u91cc Tighnari (Genshin Impact)": 186,
|
||||||
|
"\u7802\u7cd6 Sucrose (Genshin Impact)": 188,
|
||||||
|
"\u884c\u79cb Xingqiu (Genshin Impact)": 190,
|
||||||
|
"\u5965\u5179 Oz (Genshin Impact)": 193,
|
||||||
|
"\u4e94\u90ce Gorou (Genshin Impact)": 198,
|
||||||
|
"\u8fbe\u8fbe\u5229\u4e9a Tartalia (Genshin Impact)": 202,
|
||||||
|
"\u4e03\u4e03 Qiqi (Genshin Impact)": 207,
|
||||||
|
"\u7533\u9e64 Shenhe (Genshin Impact)": 217,
|
||||||
|
"\u83b1\u4f9d\u62c9 Layla (Genshin Impact)": 228,
|
||||||
|
"\u83f2\u8c22\u5c14 Fishl (Genshin Impact)": 230,
|
||||||
|
"User": 999,
|
||||||
|
"rosalia": 1000,
|
||||||
|
"taffy": 1001,
|
||||||
|
"YaeSakura": 1002
|
||||||
}
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
+3
-142
@@ -10,146 +10,6 @@ import commons
|
|||||||
from mel_processing import spectrogram_torch
|
from mel_processing import spectrogram_torch
|
||||||
from utils import load_wav_to_torch, load_filepaths_and_text
|
from utils import load_wav_to_torch, load_filepaths_and_text
|
||||||
from text import text_to_sequence, cleaned_text_to_sequence
|
from text import text_to_sequence, cleaned_text_to_sequence
|
||||||
|
|
||||||
|
|
||||||
class TextAudioLoader(torch.utils.data.Dataset):
|
|
||||||
"""
|
|
||||||
1) loads audio, text pairs
|
|
||||||
2) normalizes text and converts them to sequences of integers
|
|
||||||
3) computes spectrograms from audio files.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, audiopaths_and_text, hparams):
|
|
||||||
self.audiopaths_and_text = load_filepaths_and_text(audiopaths_and_text)
|
|
||||||
self.text_cleaners = hparams.text_cleaners
|
|
||||||
self.max_wav_value = hparams.max_wav_value
|
|
||||||
self.sampling_rate = hparams.sampling_rate
|
|
||||||
self.filter_length = hparams.filter_length
|
|
||||||
self.hop_length = hparams.hop_length
|
|
||||||
self.win_length = hparams.win_length
|
|
||||||
self.sampling_rate = hparams.sampling_rate
|
|
||||||
|
|
||||||
self.cleaned_text = getattr(hparams, "cleaned_text", False)
|
|
||||||
|
|
||||||
self.add_blank = hparams.add_blank
|
|
||||||
self.min_text_len = getattr(hparams, "min_text_len", 1)
|
|
||||||
self.max_text_len = getattr(hparams, "max_text_len", 190)
|
|
||||||
|
|
||||||
random.seed(1234)
|
|
||||||
random.shuffle(self.audiopaths_and_text)
|
|
||||||
self._filter()
|
|
||||||
|
|
||||||
def _filter(self):
|
|
||||||
"""
|
|
||||||
Filter text & store spec lengths
|
|
||||||
"""
|
|
||||||
# Store spectrogram lengths for Bucketing
|
|
||||||
# wav_length ~= file_size / (wav_channels * Bytes per dim) = file_size / (1 * 2)
|
|
||||||
# spec_length = wav_length // hop_length
|
|
||||||
|
|
||||||
audiopaths_and_text_new = []
|
|
||||||
lengths = []
|
|
||||||
for audiopath, text in self.audiopaths_and_text:
|
|
||||||
if self.min_text_len <= len(text) and len(text) <= self.max_text_len:
|
|
||||||
audiopaths_and_text_new.append([audiopath, text])
|
|
||||||
lengths.append(os.path.getsize(audiopath) // (2 * self.hop_length))
|
|
||||||
self.audiopaths_and_text = audiopaths_and_text_new
|
|
||||||
self.lengths = lengths
|
|
||||||
|
|
||||||
def get_audio_text_pair(self, audiopath_and_text):
|
|
||||||
# separate filename and text
|
|
||||||
audiopath, text = audiopath_and_text[0], audiopath_and_text[1]
|
|
||||||
text = self.get_text(text)
|
|
||||||
spec, wav = self.get_audio(audiopath)
|
|
||||||
return (text, spec, wav)
|
|
||||||
|
|
||||||
def get_audio(self, filename):
|
|
||||||
audio, sampling_rate = load_wav_to_torch(filename)
|
|
||||||
if sampling_rate != self.sampling_rate:
|
|
||||||
raise ValueError("{} {} SR doesn't match target {} SR".format(
|
|
||||||
sampling_rate, self.sampling_rate))
|
|
||||||
audio_norm = audio / self.max_wav_value
|
|
||||||
audio_norm = audio_norm.unsqueeze(0)
|
|
||||||
spec_filename = filename.replace(".wav", ".spec.pt")
|
|
||||||
if os.path.exists(spec_filename):
|
|
||||||
spec = torch.load(spec_filename)
|
|
||||||
else:
|
|
||||||
spec = spectrogram_torch(audio_norm, self.filter_length,
|
|
||||||
self.sampling_rate, self.hop_length, self.win_length,
|
|
||||||
center=False)
|
|
||||||
spec = torch.squeeze(spec, 0)
|
|
||||||
torch.save(spec, spec_filename)
|
|
||||||
return spec, audio_norm
|
|
||||||
|
|
||||||
def get_text(self, text):
|
|
||||||
if self.cleaned_text:
|
|
||||||
text_norm = cleaned_text_to_sequence(text)
|
|
||||||
else:
|
|
||||||
text_norm = text_to_sequence(text, self.text_cleaners)
|
|
||||||
if self.add_blank:
|
|
||||||
text_norm = commons.intersperse(text_norm, 0)
|
|
||||||
text_norm = torch.LongTensor(text_norm)
|
|
||||||
return text_norm
|
|
||||||
|
|
||||||
def __getitem__(self, index):
|
|
||||||
return self.get_audio_text_pair(self.audiopaths_and_text[index])
|
|
||||||
|
|
||||||
def __len__(self):
|
|
||||||
return len(self.audiopaths_and_text)
|
|
||||||
|
|
||||||
|
|
||||||
class TextAudioCollate():
|
|
||||||
""" Zero-pads model inputs and targets
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, return_ids=False):
|
|
||||||
self.return_ids = return_ids
|
|
||||||
|
|
||||||
def __call__(self, batch):
|
|
||||||
"""Collate's training batch from normalized text and aduio
|
|
||||||
PARAMS
|
|
||||||
------
|
|
||||||
batch: [text_normalized, spec_normalized, wav_normalized]
|
|
||||||
"""
|
|
||||||
# Right zero-pad all one-hot text sequences to max input length
|
|
||||||
_, ids_sorted_decreasing = torch.sort(
|
|
||||||
torch.LongTensor([x[1].size(1) for x in batch]),
|
|
||||||
dim=0, descending=True)
|
|
||||||
|
|
||||||
max_text_len = max([len(x[0]) for x in batch])
|
|
||||||
max_spec_len = max([x[1].size(1) for x in batch])
|
|
||||||
max_wav_len = max([x[2].size(1) for x in batch])
|
|
||||||
|
|
||||||
text_lengths = torch.LongTensor(len(batch))
|
|
||||||
spec_lengths = torch.LongTensor(len(batch))
|
|
||||||
wav_lengths = torch.LongTensor(len(batch))
|
|
||||||
|
|
||||||
text_padded = torch.LongTensor(len(batch), max_text_len)
|
|
||||||
spec_padded = torch.FloatTensor(len(batch), batch[0][1].size(0), max_spec_len)
|
|
||||||
wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len)
|
|
||||||
text_padded.zero_()
|
|
||||||
spec_padded.zero_()
|
|
||||||
wav_padded.zero_()
|
|
||||||
for i in range(len(ids_sorted_decreasing)):
|
|
||||||
row = batch[ids_sorted_decreasing[i]]
|
|
||||||
|
|
||||||
text = row[0]
|
|
||||||
text_padded[i, :text.size(0)] = text
|
|
||||||
text_lengths[i] = text.size(0)
|
|
||||||
|
|
||||||
spec = row[1]
|
|
||||||
spec_padded[i, :, :spec.size(1)] = spec
|
|
||||||
spec_lengths[i] = spec.size(1)
|
|
||||||
|
|
||||||
wav = row[2]
|
|
||||||
wav_padded[i, :, :wav.size(1)] = wav
|
|
||||||
wav_lengths[i] = wav.size(1)
|
|
||||||
|
|
||||||
if self.return_ids:
|
|
||||||
return text_padded, text_lengths, spec_padded, spec_lengths, wav_padded, wav_lengths, ids_sorted_decreasing
|
|
||||||
return text_padded, text_lengths, spec_padded, spec_lengths, wav_padded, wav_lengths
|
|
||||||
|
|
||||||
|
|
||||||
"""Multi speaker version"""
|
"""Multi speaker version"""
|
||||||
|
|
||||||
|
|
||||||
@@ -160,7 +20,7 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
|||||||
3) computes spectrograms from audio files.
|
3) computes spectrograms from audio files.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, audiopaths_sid_text, hparams):
|
def __init__(self, audiopaths_sid_text, hparams, symbols):
|
||||||
self.audiopaths_sid_text = load_filepaths_and_text(audiopaths_sid_text)
|
self.audiopaths_sid_text = load_filepaths_and_text(audiopaths_sid_text)
|
||||||
self.text_cleaners = hparams.text_cleaners
|
self.text_cleaners = hparams.text_cleaners
|
||||||
self.max_wav_value = hparams.max_wav_value
|
self.max_wav_value = hparams.max_wav_value
|
||||||
@@ -175,6 +35,7 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
|||||||
self.add_blank = hparams.add_blank
|
self.add_blank = hparams.add_blank
|
||||||
self.min_text_len = getattr(hparams, "min_text_len", 1)
|
self.min_text_len = getattr(hparams, "min_text_len", 1)
|
||||||
self.max_text_len = getattr(hparams, "max_text_len", 190)
|
self.max_text_len = getattr(hparams, "max_text_len", 190)
|
||||||
|
self.symbols = symbols
|
||||||
|
|
||||||
random.seed(1234)
|
random.seed(1234)
|
||||||
random.shuffle(self.audiopaths_sid_text)
|
random.shuffle(self.audiopaths_sid_text)
|
||||||
@@ -232,7 +93,7 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
|||||||
|
|
||||||
def get_text(self, text):
|
def get_text(self, text):
|
||||||
if self.cleaned_text:
|
if self.cleaned_text:
|
||||||
text_norm = cleaned_text_to_sequence(text)
|
text_norm = cleaned_text_to_sequence(text, self.symbols)
|
||||||
else:
|
else:
|
||||||
text_norm = text_to_sequence(text, self.text_cleaners)
|
text_norm = text_to_sequence(text, self.text_cleaners)
|
||||||
if self.add_blank:
|
if self.add_blank:
|
||||||
|
|||||||
@@ -0,0 +1,18 @@
|
|||||||
|
import os
|
||||||
|
import torchaudio
|
||||||
|
raw_audio_dir = "./raw_audio/"
|
||||||
|
denoise_audio_dir = "./denoised_audio/"
|
||||||
|
filelist = list(os.walk(raw_audio_dir))[0][2]
|
||||||
|
|
||||||
|
for file in filelist:
|
||||||
|
if file.endswith(".wav"):
|
||||||
|
os.system(f"demucs --two-stems=vocals {raw_audio_dir}{file}")
|
||||||
|
for file in filelist:
|
||||||
|
file = file.replace(".wav", "")
|
||||||
|
wav, sr = torchaudio.load(f"./separated/htdemucs/{file}/vocals.wav", frame_offset=0, num_frames=-1, normalize=True,
|
||||||
|
channels_first=True)
|
||||||
|
# merge two channels into one
|
||||||
|
wav = wav.mean(dim=0).unsqueeze(0)
|
||||||
|
if sr != 22050:
|
||||||
|
wav = torchaudio.transforms.Resample(orig_freq=sr, new_freq=22050)(wav)
|
||||||
|
torchaudio.save(denoise_audio_dir + file + ".wav", wav, 22050, channels_first=True)
|
||||||
+3
-2
@@ -1,3 +1,4 @@
|
|||||||
from google.colab import files
|
from google.colab import files
|
||||||
files.download("./OUTPUT_MODEL/G_latest.pth")
|
files.download("./G_latest.pth")
|
||||||
files.download("./finetune_speaker.json")
|
files.download("./finetune_speaker.json")
|
||||||
|
files.download("./moegoe_config.json")
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
from google.colab import files
|
||||||
|
import shutil
|
||||||
|
import os
|
||||||
|
basepath = os.getcwd()
|
||||||
|
uploaded = files.upload() # 上传文件
|
||||||
|
for filename in uploaded.keys():
|
||||||
|
assert(filename.endswith(".txt")), "speaker-videolink info could only be .txt file!"
|
||||||
|
shutil.move(os.path.join(basepath, filename), os.path.join("./speaker_links.txt"))
|
||||||
|
|
||||||
|
with open("./speaker_links.txt", 'r', encoding='utf-8') as f:
|
||||||
|
lines = f.readlines()
|
||||||
|
speakers = []
|
||||||
|
for line in lines:
|
||||||
|
line = line.replace("\n", "").replace(" ", "")
|
||||||
|
if line == "":
|
||||||
|
continue
|
||||||
|
speaker, link = line.split("|")
|
||||||
|
if speaker not in speakers:
|
||||||
|
speakers.append(speaker)
|
||||||
|
# download link
|
||||||
|
import random
|
||||||
|
filename = speaker + "_" + str(random.randint(0, 1000000))
|
||||||
|
os.system(f"youtube-dl -f 0 {link} -o ./video_data/{filename}.mp4")
|
||||||
@@ -0,0 +1,320 @@
|
|||||||
|
import os
|
||||||
|
import json
|
||||||
|
import argparse
|
||||||
|
import itertools
|
||||||
|
import math
|
||||||
|
import torch
|
||||||
|
from torch import nn, optim
|
||||||
|
from torch.nn import functional as F
|
||||||
|
from torch.utils.data import DataLoader
|
||||||
|
from torch.utils.tensorboard import SummaryWriter
|
||||||
|
import torch.multiprocessing as mp
|
||||||
|
import torch.distributed as dist
|
||||||
|
from torch.nn.parallel import DistributedDataParallel as DDP
|
||||||
|
from torch.cuda.amp import autocast, GradScaler
|
||||||
|
from tqdm import tqdm
|
||||||
|
|
||||||
|
import librosa
|
||||||
|
import logging
|
||||||
|
|
||||||
|
logging.getLogger('numba').setLevel(logging.WARNING)
|
||||||
|
|
||||||
|
import commons
|
||||||
|
import utils
|
||||||
|
from data_utils import (
|
||||||
|
TextAudioSpeakerLoader,
|
||||||
|
TextAudioSpeakerCollate,
|
||||||
|
DistributedBucketSampler
|
||||||
|
)
|
||||||
|
from models import (
|
||||||
|
SynthesizerTrn,
|
||||||
|
MultiPeriodDiscriminator,
|
||||||
|
)
|
||||||
|
from losses import (
|
||||||
|
generator_loss,
|
||||||
|
discriminator_loss,
|
||||||
|
feature_loss,
|
||||||
|
kl_loss
|
||||||
|
)
|
||||||
|
from mel_processing import mel_spectrogram_torch, spec_to_mel_torch
|
||||||
|
|
||||||
|
|
||||||
|
torch.backends.cudnn.benchmark = True
|
||||||
|
global_step = 0
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Assume Single Node Multi GPUs Training Only"""
|
||||||
|
assert torch.cuda.is_available(), "CPU training is not allowed."
|
||||||
|
|
||||||
|
n_gpus = torch.cuda.device_count()
|
||||||
|
os.environ['MASTER_ADDR'] = 'localhost'
|
||||||
|
os.environ['MASTER_PORT'] = '8000'
|
||||||
|
|
||||||
|
hps = utils.get_hparams()
|
||||||
|
mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
|
||||||
|
|
||||||
|
|
||||||
|
def run(rank, n_gpus, hps):
|
||||||
|
global global_step
|
||||||
|
symbols = hps['symbols']
|
||||||
|
if rank == 0:
|
||||||
|
logger = utils.get_logger(hps.model_dir)
|
||||||
|
logger.info(hps)
|
||||||
|
utils.check_git_hash(hps.model_dir)
|
||||||
|
writer = SummaryWriter(log_dir=hps.model_dir)
|
||||||
|
writer_eval = SummaryWriter(log_dir=os.path.join(hps.model_dir, "eval"))
|
||||||
|
|
||||||
|
dist.init_process_group(backend='nccl', init_method='env://', world_size=n_gpus, rank=rank)
|
||||||
|
torch.manual_seed(hps.train.seed)
|
||||||
|
torch.cuda.set_device(rank)
|
||||||
|
|
||||||
|
train_dataset = TextAudioSpeakerLoader(hps.data.training_files, hps.data, symbols)
|
||||||
|
train_sampler = DistributedBucketSampler(
|
||||||
|
train_dataset,
|
||||||
|
hps.train.batch_size,
|
||||||
|
[32,300,400,500,600,700,800,900,1000],
|
||||||
|
num_replicas=n_gpus,
|
||||||
|
rank=rank,
|
||||||
|
shuffle=True)
|
||||||
|
collate_fn = TextAudioSpeakerCollate()
|
||||||
|
train_loader = DataLoader(train_dataset, num_workers=2, shuffle=False, pin_memory=True,
|
||||||
|
collate_fn=collate_fn, batch_sampler=train_sampler)
|
||||||
|
# train_loader = DataLoader(train_dataset, batch_size=hps.train.batch_size, num_workers=2, shuffle=False, pin_memory=True,
|
||||||
|
# collate_fn=collate_fn)
|
||||||
|
if rank == 0:
|
||||||
|
eval_dataset = TextAudioSpeakerLoader(hps.data.validation_files, hps.data, symbols)
|
||||||
|
eval_loader = DataLoader(eval_dataset, num_workers=0, shuffle=False,
|
||||||
|
batch_size=hps.train.batch_size, pin_memory=True,
|
||||||
|
drop_last=False, collate_fn=collate_fn)
|
||||||
|
|
||||||
|
net_g = SynthesizerTrn(
|
||||||
|
len(symbols),
|
||||||
|
hps.data.filter_length // 2 + 1,
|
||||||
|
hps.train.segment_size // hps.data.hop_length,
|
||||||
|
n_speakers=hps.data.n_speakers,
|
||||||
|
**hps.model).cuda(rank)
|
||||||
|
net_d = MultiPeriodDiscriminator(hps.model.use_spectral_norm).cuda(rank)
|
||||||
|
|
||||||
|
# load existing model
|
||||||
|
_, _, _, _ = utils.load_checkpoint("./pretrained_models/G_0.pth", net_g, None, drop_speaker_emb=hps.drop_speaker_embed)
|
||||||
|
_, _, _, _ = utils.load_checkpoint("./pretrained_models/D_0.pth", net_d, None)
|
||||||
|
epoch_str = 1
|
||||||
|
global_step = 0
|
||||||
|
# freeze all other layers except speaker embedding
|
||||||
|
for p in net_g.parameters():
|
||||||
|
p.requires_grad = True
|
||||||
|
for p in net_d.parameters():
|
||||||
|
p.requires_grad = True
|
||||||
|
# for p in net_d.parameters():
|
||||||
|
# p.requires_grad = False
|
||||||
|
# net_g.emb_g.weight.requires_grad = True
|
||||||
|
optim_g = torch.optim.AdamW(
|
||||||
|
net_g.parameters(),
|
||||||
|
hps.train.learning_rate,
|
||||||
|
betas=hps.train.betas,
|
||||||
|
eps=hps.train.eps)
|
||||||
|
optim_d = torch.optim.AdamW(
|
||||||
|
net_d.parameters(),
|
||||||
|
hps.train.learning_rate,
|
||||||
|
betas=hps.train.betas,
|
||||||
|
eps=hps.train.eps)
|
||||||
|
# optim_d = None
|
||||||
|
net_g = DDP(net_g, device_ids=[rank])
|
||||||
|
net_d = DDP(net_d, device_ids=[rank])
|
||||||
|
|
||||||
|
scheduler_g = torch.optim.lr_scheduler.ExponentialLR(optim_g, gamma=hps.train.lr_decay)
|
||||||
|
scheduler_d = torch.optim.lr_scheduler.ExponentialLR(optim_d, gamma=hps.train.lr_decay)
|
||||||
|
|
||||||
|
scaler = GradScaler(enabled=hps.train.fp16_run)
|
||||||
|
|
||||||
|
for epoch in range(epoch_str, hps.train.epochs + 1):
|
||||||
|
if rank==0:
|
||||||
|
train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval])
|
||||||
|
else:
|
||||||
|
train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, None], None, None)
|
||||||
|
scheduler_g.step()
|
||||||
|
scheduler_d.step()
|
||||||
|
|
||||||
|
|
||||||
|
def train_and_evaluate(rank, epoch, hps, nets, optims, schedulers, scaler, loaders, logger, writers):
|
||||||
|
net_g, net_d = nets
|
||||||
|
optim_g, optim_d = optims
|
||||||
|
scheduler_g, scheduler_d = schedulers
|
||||||
|
train_loader, eval_loader = loaders
|
||||||
|
if writers is not None:
|
||||||
|
writer, writer_eval = writers
|
||||||
|
|
||||||
|
# train_loader.batch_sampler.set_epoch(epoch)
|
||||||
|
global global_step
|
||||||
|
|
||||||
|
net_g.train()
|
||||||
|
net_d.train()
|
||||||
|
for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers) in enumerate(tqdm(train_loader)):
|
||||||
|
x, x_lengths = x.cuda(rank, non_blocking=True), x_lengths.cuda(rank, non_blocking=True)
|
||||||
|
spec, spec_lengths = spec.cuda(rank, non_blocking=True), spec_lengths.cuda(rank, non_blocking=True)
|
||||||
|
y, y_lengths = y.cuda(rank, non_blocking=True), y_lengths.cuda(rank, non_blocking=True)
|
||||||
|
speakers = speakers.cuda(rank, non_blocking=True)
|
||||||
|
|
||||||
|
with autocast(enabled=hps.train.fp16_run):
|
||||||
|
y_hat, l_length, attn, ids_slice, x_mask, z_mask,\
|
||||||
|
(z, z_p, m_p, logs_p, m_q, logs_q) = net_g(x, x_lengths, spec, spec_lengths, speakers)
|
||||||
|
|
||||||
|
mel = spec_to_mel_torch(
|
||||||
|
spec,
|
||||||
|
hps.data.filter_length,
|
||||||
|
hps.data.n_mel_channels,
|
||||||
|
hps.data.sampling_rate,
|
||||||
|
hps.data.mel_fmin,
|
||||||
|
hps.data.mel_fmax)
|
||||||
|
y_mel = commons.slice_segments(mel, ids_slice, hps.train.segment_size // hps.data.hop_length)
|
||||||
|
y_hat_mel = mel_spectrogram_torch(
|
||||||
|
y_hat.squeeze(1),
|
||||||
|
hps.data.filter_length,
|
||||||
|
hps.data.n_mel_channels,
|
||||||
|
hps.data.sampling_rate,
|
||||||
|
hps.data.hop_length,
|
||||||
|
hps.data.win_length,
|
||||||
|
hps.data.mel_fmin,
|
||||||
|
hps.data.mel_fmax
|
||||||
|
)
|
||||||
|
|
||||||
|
y = commons.slice_segments(y, ids_slice * hps.data.hop_length, hps.train.segment_size) # slice
|
||||||
|
|
||||||
|
# Discriminator
|
||||||
|
y_d_hat_r, y_d_hat_g, _, _ = net_d(y, y_hat.detach())
|
||||||
|
with autocast(enabled=False):
|
||||||
|
loss_disc, losses_disc_r, losses_disc_g = discriminator_loss(y_d_hat_r, y_d_hat_g)
|
||||||
|
loss_disc_all = loss_disc
|
||||||
|
optim_d.zero_grad()
|
||||||
|
scaler.scale(loss_disc_all).backward()
|
||||||
|
scaler.unscale_(optim_d)
|
||||||
|
grad_norm_d = commons.clip_grad_value_(net_d.parameters(), None)
|
||||||
|
scaler.step(optim_d)
|
||||||
|
|
||||||
|
with autocast(enabled=hps.train.fp16_run):
|
||||||
|
# Generator
|
||||||
|
y_d_hat_r, y_d_hat_g, fmap_r, fmap_g = net_d(y, y_hat)
|
||||||
|
with autocast(enabled=False):
|
||||||
|
loss_dur = torch.sum(l_length.float())
|
||||||
|
loss_mel = F.l1_loss(y_mel, y_hat_mel) * hps.train.c_mel
|
||||||
|
loss_kl = kl_loss(z_p, logs_q, m_p, logs_p, z_mask) * hps.train.c_kl
|
||||||
|
|
||||||
|
loss_fm = feature_loss(fmap_r, fmap_g)
|
||||||
|
loss_gen, losses_gen = generator_loss(y_d_hat_g)
|
||||||
|
loss_gen_all = loss_gen + loss_fm + loss_mel + loss_dur + loss_kl
|
||||||
|
optim_g.zero_grad()
|
||||||
|
scaler.scale(loss_gen_all).backward()
|
||||||
|
scaler.unscale_(optim_g)
|
||||||
|
grad_norm_g = commons.clip_grad_value_(net_g.parameters(), None)
|
||||||
|
scaler.step(optim_g)
|
||||||
|
scaler.update()
|
||||||
|
|
||||||
|
if rank==0:
|
||||||
|
if global_step % hps.train.log_interval == 0:
|
||||||
|
lr = optim_g.param_groups[0]['lr']
|
||||||
|
losses = [loss_disc, loss_gen, loss_fm, loss_mel, loss_dur, loss_kl]
|
||||||
|
logger.info('Train Epoch: {} [{:.0f}%]'.format(
|
||||||
|
epoch,
|
||||||
|
100. * batch_idx / len(train_loader)))
|
||||||
|
logger.info([x.item() for x in losses] + [global_step, lr])
|
||||||
|
|
||||||
|
scalar_dict = {"loss/g/total": loss_gen_all, "loss/d/total": loss_disc_all, "learning_rate": lr, "grad_norm_g": grad_norm_g}
|
||||||
|
scalar_dict.update({"loss/g/fm": loss_fm, "loss/g/mel": loss_mel, "loss/g/dur": loss_dur, "loss/g/kl": loss_kl})
|
||||||
|
|
||||||
|
scalar_dict.update({"loss/g/{}".format(i): v for i, v in enumerate(losses_gen)})
|
||||||
|
scalar_dict.update({"loss/d_r/{}".format(i): v for i, v in enumerate(losses_disc_r)})
|
||||||
|
scalar_dict.update({"loss/d_g/{}".format(i): v for i, v in enumerate(losses_disc_g)})
|
||||||
|
image_dict = {
|
||||||
|
"slice/mel_org": utils.plot_spectrogram_to_numpy(y_mel[0].data.cpu().numpy()),
|
||||||
|
"slice/mel_gen": utils.plot_spectrogram_to_numpy(y_hat_mel[0].data.cpu().numpy()),
|
||||||
|
"all/mel": utils.plot_spectrogram_to_numpy(mel[0].data.cpu().numpy()),
|
||||||
|
"all/attn": utils.plot_alignment_to_numpy(attn[0,0].data.cpu().numpy())
|
||||||
|
}
|
||||||
|
utils.summarize(
|
||||||
|
writer=writer,
|
||||||
|
global_step=global_step,
|
||||||
|
images=image_dict,
|
||||||
|
scalars=scalar_dict)
|
||||||
|
|
||||||
|
if global_step % hps.train.eval_interval == 0:
|
||||||
|
evaluate(hps, net_g, eval_loader, writer_eval)
|
||||||
|
utils.save_checkpoint(net_g, None, hps.train.learning_rate, epoch, os.path.join(hps.model_dir, "G_{}.pth".format(global_step)))
|
||||||
|
utils.save_checkpoint(net_g, None, hps.train.learning_rate, epoch,
|
||||||
|
os.path.join(hps.model_dir, "G_latest.pth".format(global_step)))
|
||||||
|
# utils.save_checkpoint(net_d, optim_d, hps.train.learning_rate, epoch, os.path.join(hps.model_dir, "D_{}.pth".format(global_step)))
|
||||||
|
old_g=os.path.join(hps.model_dir, "G_{}.pth".format(global_step-4000))
|
||||||
|
# old_d=os.path.join(hps.model_dir, "D_{}.pth".format(global_step-400))
|
||||||
|
if os.path.exists(old_g):
|
||||||
|
os.remove(old_g)
|
||||||
|
# if os.path.exists(old_d):
|
||||||
|
# os.remove(old_d)
|
||||||
|
global_step += 1
|
||||||
|
if epoch > hps.max_epochs:
|
||||||
|
print("Maximum epoch reached, closing training...")
|
||||||
|
exit()
|
||||||
|
|
||||||
|
if rank == 0:
|
||||||
|
logger.info('====> Epoch: {}'.format(epoch))
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(hps, generator, eval_loader, writer_eval):
|
||||||
|
generator.eval()
|
||||||
|
with torch.no_grad():
|
||||||
|
for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers) in enumerate(eval_loader):
|
||||||
|
x, x_lengths = x.cuda(0), x_lengths.cuda(0)
|
||||||
|
spec, spec_lengths = spec.cuda(0), spec_lengths.cuda(0)
|
||||||
|
y, y_lengths = y.cuda(0), y_lengths.cuda(0)
|
||||||
|
speakers = speakers.cuda(0)
|
||||||
|
|
||||||
|
# remove else
|
||||||
|
x = x[:1]
|
||||||
|
x_lengths = x_lengths[:1]
|
||||||
|
spec = spec[:1]
|
||||||
|
spec_lengths = spec_lengths[:1]
|
||||||
|
y = y[:1]
|
||||||
|
y_lengths = y_lengths[:1]
|
||||||
|
speakers = speakers[:1]
|
||||||
|
break
|
||||||
|
y_hat, attn, mask, *_ = generator.module.infer(x, x_lengths, speakers, max_len=1000)
|
||||||
|
y_hat_lengths = mask.sum([1,2]).long() * hps.data.hop_length
|
||||||
|
|
||||||
|
mel = spec_to_mel_torch(
|
||||||
|
spec,
|
||||||
|
hps.data.filter_length,
|
||||||
|
hps.data.n_mel_channels,
|
||||||
|
hps.data.sampling_rate,
|
||||||
|
hps.data.mel_fmin,
|
||||||
|
hps.data.mel_fmax)
|
||||||
|
y_hat_mel = mel_spectrogram_torch(
|
||||||
|
y_hat.squeeze(1).float(),
|
||||||
|
hps.data.filter_length,
|
||||||
|
hps.data.n_mel_channels,
|
||||||
|
hps.data.sampling_rate,
|
||||||
|
hps.data.hop_length,
|
||||||
|
hps.data.win_length,
|
||||||
|
hps.data.mel_fmin,
|
||||||
|
hps.data.mel_fmax
|
||||||
|
)
|
||||||
|
image_dict = {
|
||||||
|
"gen/mel": utils.plot_spectrogram_to_numpy(y_hat_mel[0].cpu().numpy())
|
||||||
|
}
|
||||||
|
audio_dict = {
|
||||||
|
"gen/audio": y_hat[0,:,:y_hat_lengths[0]]
|
||||||
|
}
|
||||||
|
if global_step == 0:
|
||||||
|
image_dict.update({"gt/mel": utils.plot_spectrogram_to_numpy(mel[0].cpu().numpy())})
|
||||||
|
audio_dict.update({"gt/audio": y[0,:,:y_lengths[0]]})
|
||||||
|
|
||||||
|
utils.summarize(
|
||||||
|
writer=writer_eval,
|
||||||
|
global_step=global_step,
|
||||||
|
images=image_dict,
|
||||||
|
audios=audio_dict,
|
||||||
|
audio_sampling_rate=hps.data.sampling_rate
|
||||||
|
)
|
||||||
|
generator.train()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
from moviepy.editor import AudioFileClip
|
||||||
|
import whisper
|
||||||
|
import os
|
||||||
|
import torchaudio
|
||||||
|
import librosa
|
||||||
|
import torch
|
||||||
|
import argparse
|
||||||
|
parent_dir = "./denoised_audio/"
|
||||||
|
filelist = list(os.walk(parent_dir))[0][2]
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--languages", default="CJE")
|
||||||
|
args = parser.parse_args()
|
||||||
|
if args.languages == "CJE":
|
||||||
|
lang2token = {
|
||||||
|
'zh': "[ZH]",
|
||||||
|
'ja': "[JA]",
|
||||||
|
"en": "[EN]",
|
||||||
|
}
|
||||||
|
elif args.languages == "CJ":
|
||||||
|
lang2token = {
|
||||||
|
'zh': "[ZH]",
|
||||||
|
'ja': "[JA]",
|
||||||
|
}
|
||||||
|
model = whisper.load_model("large")
|
||||||
|
speaker_annos = []
|
||||||
|
for file in filelist:
|
||||||
|
print(f"transcribing {parent_dir + file}...\n")
|
||||||
|
options = dict(beam_size=5, best_of=5)
|
||||||
|
transcribe_options = dict(task="transcribe", **options)
|
||||||
|
result = model.transcribe(parent_dir + file, **transcribe_options)
|
||||||
|
segments = result["segments"]
|
||||||
|
# result = model.transcribe(parent_dir + file)
|
||||||
|
lang = result['language']
|
||||||
|
if result['language'] not in list(lang2token.keys()):
|
||||||
|
print(f"{lang} not supported, ignoring...\n")
|
||||||
|
# segment audio based on segment results
|
||||||
|
character_name = file.rstrip(".wav").split("_")[0]
|
||||||
|
code = file.rstrip(".wav").split("_")[1]
|
||||||
|
if not os.path.exists("./segmented_character_voice/" + character_name):
|
||||||
|
os.mkdir("./segmented_character_voice/" + character_name)
|
||||||
|
wav, sr = torchaudio.load(parent_dir + file, frame_offset=0, num_frames=-1, normalize=True,
|
||||||
|
channels_first=True)
|
||||||
|
|
||||||
|
for i, seg in enumerate(result['segments']):
|
||||||
|
start_time = seg['start']
|
||||||
|
end_time = seg['end']
|
||||||
|
text = seg['text']
|
||||||
|
text = lang2token[lang] + text.replace("\n", "") + lang2token[lang]
|
||||||
|
text = text + "\n"
|
||||||
|
wav_seg = wav[:, int(start_time*sr):int(end_time*sr)]
|
||||||
|
wav_seg_name = f"{character_name}_{code}_{i}.wav"
|
||||||
|
savepth = "./segmented_character_voice/" + character_name + "/" + wav_seg_name
|
||||||
|
speaker_annos.append(savepth + "|" + character_name + "|" + text)
|
||||||
|
# trimmed_wav_seg = librosa.effects.trim(wav_seg.squeeze().numpy())
|
||||||
|
# trimmed_wav_seg = torch.tensor(trimmed_wav_seg[0]).unsqueeze(0)
|
||||||
|
torchaudio.save(savepth, wav_seg, 22050, channels_first=True)
|
||||||
|
|
||||||
|
with open("long_character_anno.txt", 'w', encoding='utf-8') as f:
|
||||||
|
for line in speaker_annos:
|
||||||
|
f.write(line)
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
import os
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--add_auxiliary_data", type=bool, help="Whether to add extra data as fine-tuning helper")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
new_annos = []
|
||||||
|
# Source 1: transcribed short audios
|
||||||
|
if os.path.exists("short_character_anno.txt"):
|
||||||
|
with open("short_character_anno.txt", 'r', encoding='utf-8') as f:
|
||||||
|
short_character_anno = f.readlines()
|
||||||
|
new_annos += short_character_anno
|
||||||
|
# Source 2: transcribed long audio segments
|
||||||
|
if os.path.exists("long_character_anno.txt"):
|
||||||
|
with open("long_character_anno.txt", 'r', encoding='utf-8') as f:
|
||||||
|
long_character_anno = f.readlines()
|
||||||
|
new_annos += long_character_anno
|
||||||
|
|
||||||
|
# Get all speaker names
|
||||||
|
speakers = []
|
||||||
|
for line in new_annos:
|
||||||
|
path, speaker, text = line.split("|")
|
||||||
|
if speaker not in speakers:
|
||||||
|
speakers.append(speaker)
|
||||||
|
assert (len(speakers) != 0), "no speaker found"
|
||||||
|
# Source 3 (Optional): sampled audios as extra training helpers
|
||||||
|
if args.add_auxiliary_data:
|
||||||
|
with open("sampled_audio4ft.txt", 'r', encoding='utf-8') as f:
|
||||||
|
old_annos = f.readlines()
|
||||||
|
num_old_voices = len(old_annos)
|
||||||
|
num_new_voices = len(new_annos)
|
||||||
|
# STEP 1: balance number of new & old voices
|
||||||
|
cc_duplicate = num_old_voices // num_new_voices
|
||||||
|
if cc_duplicate == 0:
|
||||||
|
cc_duplicate = 1
|
||||||
|
|
||||||
|
|
||||||
|
# STEP 2: modify config file
|
||||||
|
with open("./configs/finetune_speaker.json", 'r', encoding='utf-8') as f:
|
||||||
|
hps = json.load(f)
|
||||||
|
|
||||||
|
# assign ids to new speakers
|
||||||
|
speaker2id = {}
|
||||||
|
for i, speaker in enumerate(speakers):
|
||||||
|
speaker2id[speaker] = hps['data']["n_speakers"] + i
|
||||||
|
# modify n_speakers
|
||||||
|
hps['data']["n_speakers"] = hps['data']["n_speakers"] + len(speakers)
|
||||||
|
# add speaker names
|
||||||
|
for speaker in speakers:
|
||||||
|
hps['speakers'][speaker] = speaker2id[speaker]
|
||||||
|
hps['train']['log_interval'] = 100
|
||||||
|
hps['train']['eval_interval'] = 1000
|
||||||
|
hps['train']['batch_size'] = 16
|
||||||
|
hps['data']['training_files'] = "final_annotation_train.txt"
|
||||||
|
hps['data']['validation_files'] = "final_annotation_val.txt"
|
||||||
|
# save modified config
|
||||||
|
with open("./configs/modified_finetune_speaker.json", 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(hps, f, indent=2)
|
||||||
|
|
||||||
|
# STEP 3: clean annotations, replace speaker names with assigned speaker IDs
|
||||||
|
import text
|
||||||
|
cleaned_new_annos = []
|
||||||
|
for i, line in enumerate(new_annos):
|
||||||
|
path, speaker, txt = line.split("|")
|
||||||
|
if len(txt) > 150:
|
||||||
|
continue
|
||||||
|
cleaned_text = text._clean_text(txt, hps['data']['text_cleaners'])
|
||||||
|
cleaned_text += "\n" if not cleaned_text.endswith("\n") else ""
|
||||||
|
cleaned_new_annos.append(path + "|" + str(speaker2id[speaker]) + "|" + cleaned_text)
|
||||||
|
# merge with old annotation
|
||||||
|
final_annos = old_annos + cc_duplicate * cleaned_new_annos
|
||||||
|
# save annotation file
|
||||||
|
with open("final_annotation_train.txt", 'w', encoding='utf-8') as f:
|
||||||
|
for line in final_annos:
|
||||||
|
f.write(line)
|
||||||
|
# save annotation file for validation
|
||||||
|
with open("final_annotation_val.txt", 'w', encoding='utf-8') as f:
|
||||||
|
for line in cleaned_new_annos:
|
||||||
|
f.write(line)
|
||||||
|
print("finished")
|
||||||
|
else:
|
||||||
|
# Do not add extra helper data
|
||||||
|
# STEP 1: modify config file
|
||||||
|
with open("./configs/finetune_speaker.json", 'r', encoding='utf-8') as f:
|
||||||
|
hps = json.load(f)
|
||||||
|
|
||||||
|
# assign ids to new speakers
|
||||||
|
speaker2id = {}
|
||||||
|
for i, speaker in enumerate(speakers):
|
||||||
|
speaker2id[speaker] = i
|
||||||
|
# modify n_speakers
|
||||||
|
hps['data']["n_speakers"] = len(speakers)
|
||||||
|
# overwrite speaker names
|
||||||
|
hps['speakers'] = speaker2id
|
||||||
|
hps['train']['log_interval'] = 10
|
||||||
|
hps['train']['eval_interval'] = 100
|
||||||
|
hps['train']['batch_size'] = 16
|
||||||
|
hps['data']['training_files'] = "final_annotation_train.txt"
|
||||||
|
hps['data']['validation_files'] = "final_annotation_val.txt"
|
||||||
|
# save modified config
|
||||||
|
with open("./configs/modified_finetune_speaker.json", 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(hps, f, indent=2)
|
||||||
|
|
||||||
|
# STEP 2: clean annotations, replace speaker names with assigned speaker IDs
|
||||||
|
import text
|
||||||
|
|
||||||
|
cleaned_new_annos = []
|
||||||
|
for i, line in enumerate(new_annos):
|
||||||
|
path, speaker, txt = line.split("|")
|
||||||
|
if len(txt) > 150:
|
||||||
|
continue
|
||||||
|
cleaned_text = text._clean_text(txt, hps['data']['text_cleaners'])
|
||||||
|
cleaned_text += "\n" if not cleaned_text.endswith("\n") else ""
|
||||||
|
cleaned_new_annos.append(path + "|" + str(speaker2id[speaker]) + "|" + cleaned_text)
|
||||||
|
|
||||||
|
final_annos = cleaned_new_annos
|
||||||
|
# save annotation file
|
||||||
|
with open("final_annotation_train.txt", 'w', encoding='utf-8') as f:
|
||||||
|
for line in final_annos:
|
||||||
|
f.write(line)
|
||||||
|
# save annotation file for validation
|
||||||
|
with open("final_annotation_val.txt", 'w', encoding='utf-8') as f:
|
||||||
|
for line in cleaned_new_annos:
|
||||||
|
f.write(line)
|
||||||
|
print("finished")
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
import torch
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--model_dir", type=str, default="./OUTPUT_MODEL/G_latest.pth")
|
||||||
|
parser.add_argument("--config_dir", type=str, default="./configs/modified_finetune_speaker.json")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
model_sd = torch.load(args.model_dir, map_location='cpu')
|
||||||
|
with open(args.config_dir, 'r', encoding='utf-8') as f:
|
||||||
|
hps = json.load(f)
|
||||||
|
|
||||||
|
valid_speakers = list(hps['speakers'].keys())
|
||||||
|
if hps['data']['n_speakers'] > len(valid_speakers):
|
||||||
|
new_emb_g = torch.zeros([len(valid_speakers), 256])
|
||||||
|
old_emb_g = model_sd['model']['emb_g.weight']
|
||||||
|
for i, speaker in enumerate(valid_speakers):
|
||||||
|
new_emb_g[i, :] = old_emb_g[hps['speakers'][speaker], :]
|
||||||
|
hps['speakers'][speaker] = i
|
||||||
|
hps['data']['n_speakers'] = len(valid_speakers)
|
||||||
|
model_sd['model']['emb_g.weight'] = new_emb_g
|
||||||
|
with open("./finetune_speaker.json", 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(hps, f, indent=2)
|
||||||
|
torch.save("./G_latest.pth", args.model_dir)
|
||||||
|
# save another config file copy in MoeGoe format
|
||||||
|
hps['speakers'] = valid_speakers
|
||||||
|
with open("./moegoe_config.json", 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(hps, f, indent=2)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
import whisper
|
||||||
|
import os
|
||||||
|
import torchaudio
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
lang2token = {
|
||||||
|
'zh': "[ZH]",
|
||||||
|
'ja': "[JA]",
|
||||||
|
"en": "[EN]",
|
||||||
|
}
|
||||||
|
def transcribe_one(audio_path):
|
||||||
|
# load audio and pad/trim it to fit 30 seconds
|
||||||
|
audio = whisper.load_audio(audio_path)
|
||||||
|
audio = whisper.pad_or_trim(audio)
|
||||||
|
|
||||||
|
# make log-Mel spectrogram and move to the same device as the model
|
||||||
|
mel = whisper.log_mel_spectrogram(audio).to(model.device)
|
||||||
|
|
||||||
|
# detect the spoken language
|
||||||
|
_, probs = model.detect_language(mel)
|
||||||
|
print(f"Detected language: {max(probs, key=probs.get)}")
|
||||||
|
lang = max(probs, key=probs.get)
|
||||||
|
# decode the audio
|
||||||
|
options = whisper.DecodingOptions()
|
||||||
|
result = whisper.decode(model, mel, options)
|
||||||
|
|
||||||
|
# print the recognized text
|
||||||
|
print(result.text)
|
||||||
|
return lang, result.text
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--languages", default="CJE")
|
||||||
|
args = parser.parse_args()
|
||||||
|
if args.languages == "CJE":
|
||||||
|
lang2token = {
|
||||||
|
'zh': "[ZH]",
|
||||||
|
'ja': "[JA]",
|
||||||
|
"en": "[EN]",
|
||||||
|
}
|
||||||
|
elif args.languages == "CJ":
|
||||||
|
lang2token = {
|
||||||
|
'zh': "[ZH]",
|
||||||
|
'ja': "[JA]",
|
||||||
|
}
|
||||||
|
model = whisper.load_model("large")
|
||||||
|
parent_dir = "./custom_character_voice/"
|
||||||
|
speaker_names = list(os.walk(parent_dir))[0][1]
|
||||||
|
speaker_annos = []
|
||||||
|
# resample audios
|
||||||
|
for speaker in speaker_names:
|
||||||
|
for i, wavfile in enumerate(list(os.walk(parent_dir + speaker))[0][2]):
|
||||||
|
# try to load file as audio
|
||||||
|
if wavfile.startswith("processed_"):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
wav, sr = torchaudio.load(parent_dir + speaker + "/" + wavfile, frame_offset=0, num_frames=-1, normalize=True,
|
||||||
|
channels_first=True)
|
||||||
|
wav = wav.mean(dim=0).unsqueeze(0)
|
||||||
|
if sr != 22050:
|
||||||
|
wav = torchaudio.transforms.Resample(orig_freq=sr, new_freq=22050)(wav)
|
||||||
|
if wav.shape[1] / sr > 20:
|
||||||
|
print(f"{wavfile} too long, ignoring\n")
|
||||||
|
save_path = parent_dir + speaker + "/" + f"processed_{i}.wav"
|
||||||
|
torchaudio.save(save_path, wav, 22050, channels_first=True)
|
||||||
|
# transcribe text
|
||||||
|
lang, text = transcribe_one(save_path)
|
||||||
|
if lang not in list(lang2token.keys()):
|
||||||
|
print(f"{lang} not supported, ignoring\n")
|
||||||
|
text = lang2token[lang] + text + lang2token[lang] + "\n"
|
||||||
|
speaker_annos.append(save_path + "|" + speaker + "|" + text)
|
||||||
|
except:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# # clean annotation
|
||||||
|
# import argparse
|
||||||
|
# import text
|
||||||
|
# from utils import load_filepaths_and_text
|
||||||
|
# for i, line in enumerate(speaker_annos):
|
||||||
|
# path, sid, txt = line.split("|")
|
||||||
|
# cleaned_text = text._clean_text(txt, ["cjke_cleaners2"])
|
||||||
|
# cleaned_text += "\n" if not cleaned_text.endswith("\n") else ""
|
||||||
|
# speaker_annos[i] = path + "|" + sid + "|" + cleaned_text
|
||||||
|
# write into annotation
|
||||||
|
with open("short_character_anno.txt", 'w', encoding='utf-8') as f:
|
||||||
|
for line in speaker_annos:
|
||||||
|
f.write(line)
|
||||||
|
|
||||||
|
# import json
|
||||||
|
# # generate new config
|
||||||
|
# with open("./configs/finetune_speaker.json", 'r', encoding='utf-8') as f:
|
||||||
|
# hps = json.load(f)
|
||||||
|
# # modify n_speakers
|
||||||
|
# hps['data']["n_speakers"] = 1000 + len(speaker2id)
|
||||||
|
# # add speaker names
|
||||||
|
# for speaker in speaker_names:
|
||||||
|
# hps['speakers'][speaker] = speaker2id[speaker]
|
||||||
|
# # save modified config
|
||||||
|
# with open("./configs/modified_finetune_speaker.json", 'w', encoding='utf-8') as f:
|
||||||
|
# json.dump(hps, f, indent=2)
|
||||||
|
# print("finished")
|
||||||
+3
-2
@@ -30,14 +30,15 @@ def text_to_sequence(text, symbols, cleaner_names):
|
|||||||
return sequence
|
return sequence
|
||||||
|
|
||||||
|
|
||||||
def cleaned_text_to_sequence(cleaned_text):
|
def cleaned_text_to_sequence(cleaned_text, symbols):
|
||||||
'''Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
|
'''Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
|
||||||
Args:
|
Args:
|
||||||
text: string to convert to a sequence
|
text: string to convert to a sequence
|
||||||
Returns:
|
Returns:
|
||||||
List of integers corresponding to the symbols in the text
|
List of integers corresponding to the symbols in the text
|
||||||
'''
|
'''
|
||||||
sequence = [_symbol_to_id[symbol] for symbol in cleaned_text if symbol in _symbol_to_id.keys()]
|
symbol_to_id = {s: i for i, s in enumerate(symbols)}
|
||||||
|
sequence = [symbol_to_id[symbol] for symbol in cleaned_text if symbol in symbol_to_id.keys()]
|
||||||
return sequence
|
return sequence
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
|
|||||||
logger = logging
|
logger = logging
|
||||||
|
|
||||||
|
|
||||||
def load_checkpoint(checkpoint_path, model, optimizer=None):
|
def load_checkpoint(checkpoint_path, model, optimizer=None, drop_speaker_emb=False):
|
||||||
assert os.path.isfile(checkpoint_path)
|
assert os.path.isfile(checkpoint_path)
|
||||||
checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
|
checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
|
||||||
iteration = checkpoint_dict['iteration']
|
iteration = checkpoint_dict['iteration']
|
||||||
@@ -31,8 +31,10 @@ def load_checkpoint(checkpoint_path, model, optimizer=None):
|
|||||||
for k, v in state_dict.items():
|
for k, v in state_dict.items():
|
||||||
try:
|
try:
|
||||||
if k == 'emb_g.weight':
|
if k == 'emb_g.weight':
|
||||||
|
if drop_speaker_emb:
|
||||||
|
new_state_dict[k] = v
|
||||||
|
continue
|
||||||
v[:saved_state_dict[k].shape[0], :] = saved_state_dict[k]
|
v[:saved_state_dict[k].shape[0], :] = saved_state_dict[k]
|
||||||
# v[999, :] = saved_state_dict[k][154, :]
|
|
||||||
new_state_dict[k] = v
|
new_state_dict[k] = v
|
||||||
else:
|
else:
|
||||||
new_state_dict[k] = saved_state_dict[k]
|
new_state_dict[k] = saved_state_dict[k]
|
||||||
@@ -154,6 +156,7 @@ def get_hparams(init=True):
|
|||||||
help='Model name')
|
help='Model name')
|
||||||
parser.add_argument('-n', '--max_epochs', type=int, default=50,
|
parser.add_argument('-n', '--max_epochs', type=int, default=50,
|
||||||
help='finetune epochs')
|
help='finetune epochs')
|
||||||
|
parser.add_argument('--drop_speaker_embed', type=bool, default=False, help='whether to drop existing characters')
|
||||||
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
model_dir = os.path.join("./", args.model)
|
model_dir = os.path.join("./", args.model)
|
||||||
@@ -176,6 +179,7 @@ def get_hparams(init=True):
|
|||||||
hparams = HParams(**config)
|
hparams = HParams(**config)
|
||||||
hparams.model_dir = model_dir
|
hparams.model_dir = model_dir
|
||||||
hparams.max_epochs = args.max_epochs
|
hparams.max_epochs = args.max_epochs
|
||||||
|
hparams.drop_speaker_embed = args.drop_speaker_embed
|
||||||
return hparams
|
return hparams
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
from moviepy.editor import AudioFileClip
|
||||||
|
import os
|
||||||
|
video_dir = "./video_data/"
|
||||||
|
audio_dir = "./raw_audio/"
|
||||||
|
filelist = list(os.walk(video_dir))[0][2]
|
||||||
|
for file in filelist:
|
||||||
|
if file.endswith(".mp4"):
|
||||||
|
my_audio_clip = AudioFileClip(video_dir + file)
|
||||||
|
my_audio_clip.write_audiofile(audio_dir + file.rstrip(".mp4") + ".wav")
|
||||||
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
from moviepy.editor import AudioFileClip
|
|
||||||
import whisper
|
|
||||||
import os
|
|
||||||
import torchaudio
|
|
||||||
parent_dir = "../"
|
|
||||||
filelist = ["taffy1.mp4", "taffy2.mp4"]
|
|
||||||
for file in filelist:
|
|
||||||
my_audio_clip = AudioFileClip(parent_dir + file)
|
|
||||||
my_audio_clip.write_audiofile(parent_dir + file.rstrip(".mp4") + ".wav")
|
|
||||||
for file in filelist:
|
|
||||||
file = file.replace(".mp4", ".wav")
|
|
||||||
os.system(f"demucs --two-stems=vocals {parent_dir}{file}")
|
|
||||||
for file in filelist:
|
|
||||||
file = file.strip(".mp4")
|
|
||||||
wav, sr = torchaudio.load(f"./separated/htdemucs/{file}/vocals.wav", frame_offset=0, num_frames=-1, normalize=True,
|
|
||||||
channels_first=True)
|
|
||||||
# merge two channels into one
|
|
||||||
wav = wav.mean(dim=0).unsqueeze(0)
|
|
||||||
if sr != 22050:
|
|
||||||
wav = torchaudio.transforms.Resample(orig_freq=sr, new_freq=22050)(wav)
|
|
||||||
torchaudio.save(file + ".wav", wav, 22050, channels_first=True)
|
|
||||||
model = whisper.load_model("medium")
|
|
||||||
def transcribe_one(audio_path):
|
|
||||||
# load audio and pad/trim it to fit 30 seconds
|
|
||||||
audio = whisper.load_audio(audio_path)
|
|
||||||
audio = whisper.pad_or_trim(audio)
|
|
||||||
|
|
||||||
# make log-Mel spectrogram and move to the same device as the model
|
|
||||||
mel = whisper.log_mel_spectrogram(audio).to(model.device)
|
|
||||||
|
|
||||||
# detect the spoken language
|
|
||||||
_, probs = model.detect_language(mel)
|
|
||||||
lang = max(probs, key=probs.get)
|
|
||||||
# decode the audio
|
|
||||||
options = whisper.DecodingOptions()
|
|
||||||
result = whisper.decode(model, mel, options)
|
|
||||||
|
|
||||||
# print the recognized text
|
|
||||||
return result
|
|
||||||
|
|
||||||
result = model.transcribe("taffy2.wav")
|
|
||||||
# segment audio based on segment results
|
|
||||||
|
|
||||||
+25
-11
@@ -1,14 +1,28 @@
|
|||||||
from google.colab import files
|
from google.colab import files
|
||||||
import shutil
|
import shutil
|
||||||
import os
|
import os
|
||||||
basepath = os.getcwd()
|
import argparse
|
||||||
uploaded = files.upload() # 上传文件
|
if __name__ == "__main__":
|
||||||
upload_path = "./custom_character_voice/"
|
parser = argparse.ArgumentParser()
|
||||||
if not os.path.exists(upload_path):
|
parser.add_argument("--type", type=str, required=True, help="type of file to upload")
|
||||||
os.mkdir(upload_path)
|
args = parser.parse_args()
|
||||||
for filename in uploaded.keys():
|
file_type = args.type
|
||||||
#将上传的文件移动到指定的位置上
|
|
||||||
if filename.endswith(".zip"):
|
basepath = os.getcwd()
|
||||||
shutil.move(os.path.join(basepath, filename), os.path.join(upload_path, "custom_character_voice.zip"))
|
uploaded = files.upload() # 上传文件
|
||||||
elif filename.endswith(".rar"):
|
assert(file_type in ['zip', 'audio', 'video'])
|
||||||
shutil.move(os.path.join(basepath, filename), os.path.join(upload_path, "custom_character_voice.rar"))
|
if file_type == "zip":
|
||||||
|
upload_path = "./custom_character_voice/"
|
||||||
|
for filename in uploaded.keys():
|
||||||
|
#将上传的文件移动到指定的位置上
|
||||||
|
shutil.move(os.path.join(basepath, filename), os.path.join(upload_path, "custom_character_voice.zip"))
|
||||||
|
elif file_type == "audio":
|
||||||
|
upload_path = "./raw_audio/"
|
||||||
|
for filename in uploaded.keys():
|
||||||
|
#将上传的文件移动到指定的位置上
|
||||||
|
shutil.move(os.path.join(basepath, filename), os.path.join(upload_path, filename))
|
||||||
|
elif file_type == "video":
|
||||||
|
upload_path = "./video_data/"
|
||||||
|
for filename in uploaded.keys():
|
||||||
|
# 将上传的文件移动到指定的位置上
|
||||||
|
shutil.move(os.path.join(basepath, filename), os.path.join(upload_path, filename))
|
||||||
Reference in New Issue
Block a user