Benjamin Elizalde
114 lines
4.0 KiB
Markdown
114 lines
4.0 KiB
Markdown
# CLAP
|
|
|
|
CLAP (Contrastive Language-Audio Pretraining) is a neural network model that learns acoustic concepts from natural language supervision. It achieves SoTA in “Zero-Shot” classification, Audio-Text & Text-Audio Retrieval, and in some datasets when finetuned.
|
|
|
|
<img width="832" alt="clap_diagram_v3" src="https://user-images.githubusercontent.com/26778834/199842089-39ef6a2e-8abb-4338-bdfe-680abab70f53.png">
|
|
|
|
## Citation
|
|
https://arxiv.org/pdf/2206.04769.pdf
|
|
```
|
|
@article{elizalde2022clap,
|
|
title={Clap: Learning audio concepts from natural language supervision},
|
|
author={Elizalde, Benjamin and Deshmukh, Soham and Ismail, Mahmoud Al and Wang, Huaming},
|
|
journal={arXiv preprint arXiv:2206.04769},
|
|
year={2022}
|
|
}
|
|
```
|
|
|
|
## Request CLAP weights:
|
|
```
|
|
https://forms.office.com/r/ULb4k9GL1F
|
|
```
|
|
|
|
|
|
### Usage
|
|
- Load model
|
|
```python
|
|
from CLAP_API import CLAP
|
|
|
|
clap_model = CLAP("<PATH TO WEIGHTS>", use_cuda=False)
|
|
```
|
|
|
|
- Extract text embeddings
|
|
```python
|
|
|
|
text_embeddings = clap_model.get_text_embeddings(class_labels: List[str])
|
|
```
|
|
|
|
- Extract audio embeddings
|
|
```python
|
|
|
|
audio_embeddings = clap_model.get_audio_embeddings(file_paths: List[str])
|
|
```
|
|
|
|
- Compute similarity
|
|
```python
|
|
# For using the below function, DO NOT normalize the text and audio embeddings
|
|
sim = clap_model.compute_similarity(audio_embeddings, text_embeddings)
|
|
```
|
|
|
|
### Zero-Shot Classification of [ESC50 dataset](https://github.com/karolpiczak/ESC-50)
|
|
|
|
```python
|
|
from CLAP_API import CLAP
|
|
from esc50 import ESC50
|
|
import torch.nn.functional as F
|
|
import numpy as np
|
|
from tqdm import tqdm
|
|
from sklearn.metrics import accuracy_score
|
|
|
|
# Load CLAP
|
|
weights_path = # Add weight path here
|
|
clap_model = CLAP(weights_path, use_cuda=False)
|
|
|
|
# Load dataset
|
|
dataset = ESC50(root='path/ESC-50-master', download=False)
|
|
prompt = 'this is a sound of '
|
|
Y = [prompt + x for x in dataset.classes]
|
|
|
|
# Computing text embeddings
|
|
text_embeddings = clap_model.get_text_embeddings(Y)
|
|
|
|
# Computing audio embeddings
|
|
y_preds, y_labels = [], []
|
|
for i in tqdm(range(len(dataset))):
|
|
x, _, one_hot_target = dataset.__getitem__(i)
|
|
|
|
audio_embeddings = clap_model.get_audio_embeddings([x], resample=True)
|
|
similarity = clap_model.compute_similarity(audio_embeddings, text_embeddings)
|
|
y_pred = F.softmax(similarity.detach().cpu(), dim=1).numpy()
|
|
y_preds.append(y_pred)
|
|
y_labels.append(one_hot_target.detach().cpu().numpy())
|
|
|
|
y_labels, y_preds = np.concatenate(y_labels, axis=0), np.concatenate(y_preds, axis=0)
|
|
acc = accuracy_score(np.argmax(y_labels, axis=1), np.argmax(y_preds, axis=1))
|
|
print('ESC50 Accuracy {}'.format(acc))
|
|
```
|
|
The output:
|
|
|
|
```
|
|
ESC50 Accuracy: 82.6%
|
|
```
|
|
|
|
## Contributing
|
|
|
|
This project welcomes contributions and suggestions. Most contributions require you to agree to a
|
|
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
|
|
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
|
|
|
|
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
|
|
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
|
|
provided by the bot. You will only need to do this once across all repos using our CLA.
|
|
|
|
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
|
|
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
|
|
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
|
|
|
|
## Trademarks
|
|
|
|
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
|
|
trademarks or logos is subject to and must follow
|
|
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
|
|
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
|
|
Any use of third-party trademarks or logos are subject to those third-party's policies.
|