CMT-LLM

The source code of "CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models"

Installation

git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout tags/v4.35.2
pip install -e .
cd ..
git clone https://github.com/huggingface/peft.git
cd peft
git checkout tags/v0.6.0
pip install -e .
cd ..
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
git clone https://github.com/JiajunHe1025/CMT-LLM.git
cd CMT-LLM
pip install  -e .

Usage

Data preparation

You need to prepare the data jsonl in this format.

{"key": "1272-128104-0000_2035-147961-0014", "source": "/data/espnet/egs2/librimix/sot_asr1/data/LibriMix/libri_mix_single_offset/Libri2Mix/wav16k/max/dev/mix_both/1272-128104-0000_2035-147961-0014.wav", "target": "MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL <sc> FROM OUR BENCH WE COULD SEE WHAT A HOLLOW CASE HIS BODY WAS", "rare_words": "QUILTER;APOSTLE;GOSPEL"}
{"key": "1272-128104-0003_2035-147961-0016", "source": "/data/espnet/egs2/librimix/sot_asr1/data/LibriMix/libri_mix_single_offset/Libri2Mix/wav16k/max/dev/mix_both/1272-128104-0003_2035-147961-0016.wav", "target": "HE HAS GRAVE DOUBTS WHETHER SIR FREDERICK LEIGHTON'S WORK IS REALLY GREEK AFTER ALL AND CAN DISCOVER IN IT BUT LITTLE OF ROCKY ITHACA <sc> WITHOUT A WORD PETER GOT UP AND LIT HIS LANTERN", "rare_words": "LEIGHTON'S;ITHACA"}
...

Decode with checkpoints

bash decode_wavlm_large_linear_vicuna_7b.sh

Modify the path including speech_encoder_path, llm_path, output_dir, ckpt_path, val_data_path and decode_log in the script when you run the shell script.

Train a new model

Use whisper as the encoder

bash finetune_whisper_large_linear_vicuna_7b.sh

Whisper takes mel as input. Pay attention to the key dataset_config.mel_size for different version of the whisper model family.

Use self-supervised model(such as WavLM) as the encoder

bash finetune_wavlm_large_linear_vicuna_7b.sh

WavLM takes raw wavform as input. Pay attention to the key dataset_config.normalize and model_config.normalize for different version of the SSL models for different SSL models are different in these keys.