azalea/VITS-fast-fine-tuning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated pipeline

Browse Source
This commit is contained in:
Plachta
2023-02-26 20:22:50 +08:00
parent 626ace0132
commit 46cc84aec2
2 changed files with 27 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+22 -40
View File
@@ -1,19 +1,23 @@
[中文文档请点击这里](https://github.com/Plachtaa/VITS-fast-fine-tuning/blob/main/README_ZH.md) [中文文档请点击这里](https://github.com/Plachtaa/VITS-fast-fine-tuning/blob/main/README_ZH.md)
# VITS Fast Fine-tuning # VITS Fast Fine-tuning
This repo will guide you to add your own character voices, or even your own voice, into an existing VITS TTS model This repo will guide you to add your own character voices, or even your own voice, into existing VITS TTS model
to make it able to do the following tasks in less than 1 hour: to make it able to do the following tasks in less than 1 hour:
1. Any-to-any voice conversion between you & any characters you added & preset characters 1. Many-to-many voice conversion between any characters you added & preset characters in the model.
2. English, Japanese & Chinese Text-to-Speech synthesis with the characters you added & preset characters 2. English, Japanese & Chinese Text-to-Speech synthesis with the characters you added & preset characters
Welcome to play around with the base model, a Trilingual Anime VITS! Welcome to play around with the base models!
[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer) Chinese & English & Japanese：[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer)
Chinese & Japanese：[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai)
### Currently Supported Tasks: ### Currently Supported Tasks:
- [x] Convert user's voice to characters listed [here](https://github.com/SongtingLiu/VITS_voice_conversion/blob/main/configs/finetune_speaker.json) - [x] Clone character voice from 10+ short audios
- [x] Chinese, English, Japanese TTS with user's voice - [x] Clone character voice from long audio(s) >= 3 minutes (one audio should contain single speaker only)
- [x] Chinese, English, Japanese TTS with custom characters! - [x] Clone character voice from videos(s) >= 3 minutes (one video should contain single speaker only)
- [x] Clone character voice from BILIBILI video links (one video should contain single speaker only)
### Currently Supported Characters for TTS & VC: ### Currently Supported Characters for TTS & VC:
- [x] Umamusume Pretty Derby (Used as base model pretraining) - [x] Umamusume Pretty Derby (Used as base model pretraining)
@@ -25,48 +29,22 @@ Welcome to play around with the base model, a Trilingual Anime VITS!
## Fine-tuning ## Fine-tuning
It's recommended to perform fine-tuning on [Google Colab](https://colab.research.google.com/drive/1omMhfYKrAAQ7a6zOCsyqpla-wU-QyfZn?usp=sharing) It's recommended to perform fine-tuning on [Google Colab](https://colab.research.google.com/drive/1pn1xnFfdLK63gVXDwV4zCXfVeo8c-I-0?usp=sharing)
because the original VITS has some dependencies that are difficult to configure. because the original VITS has some dependencies that are difficult to configure.
### How long does it take? ### How long does it take?
1. Install dependencies (2 min) 1. Install dependencies (3 min)
2. Record at least 20 your own voice, the content to read will be presented in UI, less than 20 words per sentence. (5~10 min) 2. Choose pretrained model to start. The detailed differences between them are described in [Colab Notebook](https://colab.research.google.com/drive/1pn1xnFfdLK63gVXDwV4zCXfVeo8c-I-0?usp=sharing)
3. Upload your character voices, which should be a `.zip` file, 3. Upload the voice samples of the characters you wish to add，see [DATA.MD](https://github.com/Plachtaa/VITS-fast-fine-tuning/blob/main/DATA.MD) for detailed uploading options.
it's file structure should be like: 4. Start fine-tuning. Time taken varies from 20 minutes ~ 2 hours, depending on the number of voices you uploaded.
```
Your-zip-file.zip
├───Character_name_1
├ ├───xxx.wav
├ ├───...
├ ├───yyy.mp3
├ └───zzz.wav
├───Character_name_2
├ ├───xxx.wav
├ ├───...
├ ├───yyy.mp3
├ └───zzz.wav
├───...
├
└───Character_name_n
├───xxx.wav
├───...
├───yyy.mp3
└───zzz.wav
```
Note that the format & name of the audio files does not matter as long as they are audio files.
Audio quality requirements: >=2s, <=10s per audio, background noise should be as less as possible.
Audio quantity requirements: at least 10 per character, better if 20+ per character.
You can either choose to perform step 2, 3, or both, depending on your needs.
4. Fine-tune (30 min)
After everything is done, download the fine-tuned model & model config
## Inference or Usage (Currently support Windows only) ## Inference or Usage (Currently support Windows only)
0. Remember to download your fine-tuned model! 0. Remember to download your fine-tuned model!
1. Download the latest release 1. Download the latest release
2. Put your model & config file into the folder `inference`, make sure to rename the model to `G_latest.pth` and config file to `finetune_speaker.json` 2. Put your model & config file into the folder `inference`, which are named `G_latest.pth` and `finetune_speaker.json`, respectively.
3. The file structure should be as follows: 3. The file structure should be as follows:
```shell ```
inference inference
├───inference.exe ├───inference.exe
├───... ├───...
@@ -74,3 +52,7 @@ inference
└───G_latest.pth └───G_latest.pth
``` ```
4. run `inference.exe`, the browser should pop up automatically. 4. run `inference.exe`, the browser should pop up automatically.
## Use in MoeGoe
0. Prepare downloaded model & config file, which are named `G_latest.pth` and `moegoe_config.json`, respectively.
1. Follow [MoeGoe](https://github.com/CjangCjengh/MoeGoe) page instructions to install, configure path, and use.
README_ZH.md
+5 -5
View File
@@ -14,9 +14,9 @@ English Documentation Please Click [here](https://github.com/Plachtaa/VITS-fast-
### 目前支持的任务: ### 目前支持的任务:
- [x] 从 10条以上的短音频 克隆角色声音 - [x] 从 10条以上的短音频 克隆角色声音
- [x] 从 3分钟以上的长音频 克隆角色声音 - [x] 从 3分钟以上的长音频（单个音频只能包含单说话人） 克隆角色声音
- [x] 从 3分钟以上的视频（只包含单说话人） 克隆角色声音 - [x] 从 3分钟以上的视频（单个视频只能包含单说话人） 克隆角色声音
- [x] 通过输入 bilibili视频链接（只包含单说话人） 克隆角色声音 - [x] 通过输入 bilibili视频链接（单个视频只能包含单说话人） 克隆角色声音
### 目前支持声线转换和中日英三语TTS的角色 ### 目前支持声线转换和中日英三语TTS的角色
- [x] 赛马娘 （仅已实装角色）（预训练时使用的角色） - [x] 赛马娘 （仅已实装角色）（预训练时使用的角色）
@@ -32,7 +32,7 @@ English Documentation Please Click [here](https://github.com/Plachtaa/VITS-fast-
进行微调任务，因为VITS在多语言情况下的某些环境依赖相当难以配置。 进行微调任务，因为VITS在多语言情况下的某些环境依赖相当难以配置。
### 在Google Colab里，我需要花多长时间？ ### 在Google Colab里，我需要花多长时间？
1. 安装依赖 (3 min) 1. 安装依赖 (3 min)
2. 选择预训练模型，详细区别参见Colab笔记本页面。 2. 选择预训练模型，详细区别参见[Colab 笔记本页面](https://colab.research.google.com/drive/1pn1xnFfdLK63gVXDwV4zCXfVeo8c-I-0?usp=sharing)。
3. 上传你希望加入的其它角色声音，详细上传方式见[DATA.MD](https://github.com/Plachtaa/VITS-fast-fine-tuning/blob/main/DATA.MD) 3. 上传你希望加入的其它角色声音，详细上传方式见[DATA.MD](https://github.com/Plachtaa/VITS-fast-fine-tuning/blob/main/DATA.MD)
4. 进行微调，根据选择的微调方式和样本数量不同，花费时长可能在20分钟到2小时不等。 4. 进行微调，根据选择的微调方式和样本数量不同，花费时长可能在20分钟到2小时不等。
@@ -41,7 +41,7 @@ English Documentation Please Click [here](https://github.com/Plachtaa/VITS-fast-
## 本地运行和推理 ## 本地运行和推理
0. 记得下载微调好的模型和config文件！ 0. 记得下载微调好的模型和config文件！
1. 下载最新的Release包（在Github页面的右侧） 1. 下载最新的Release包（在Github页面的右侧）
2. 把下载的模型和config文件放在 `inference`文件夹下, 确保模型的文件名为 `G_latest.pth` ，config文件名为 `finetune_speaker.json` 2. 把下载的模型和config文件放在 `inference`文件夹下, 其文件名分别为 `G_latest.pth` 和 `finetune_speaker.json`。
3. 一切准备就绪后，文件结构应该如下所示: 3. 一切准备就绪后，文件结构应该如下所示:
``` ```
inference inference