VERSA
VERSADocs

Metric catalog

Configuration keys, outputs, implementations, and references.

On this page

Optional discrete-speech backend

SpeechBERTScore, SpeechBLEU, and SpeechTokenDistance are supported with additional installation; they are not included in the base package or the external extra. From a repository checkout, install the pinned fork:

python -m pip install -r tools/requirements-external.txt

Use egs/separate_metrics/discrete_speech.yaml to run these metrics separately. The general speech recipes exclude them. Their individual tests require VERSA_RUN_REAL_MODEL_TESTS=1 and may download checkpoints; see the installation guide for commands.

Independent Metrics

We include x mark if the metric is auto-installed in versa.

Number Auto-Install Metric Name (Auto-Install) Key in config Key in report Code Source References
1 x Deep Noise Suppression MOS Score of P.835 (DNSMOS) pseudo_mos dnsmos_overall speechmos (MS) paper
2 x Deep Noise Suppression MOS Score of P.808 (DNSMOS) pseudo_mos dnsmos_p808 speechmos (MS) paper
3 x Non-intrusive Speech Quality and Naturalness Assessment (NISQA) nisqa {nisqa_mos_pred, nisqa_noi_pred, nisqa_dis_pred, nisqa_col_pred, nisqa_loud_pred} NISQA paper
4 x UTokyo-SaruLab System for VoiceMOS Challenge 2022 (UTMOS) pseudo_mos utmos speechmos paper
5 x Packet Loss Concealment-related MOS Score (PLCMOS) pseudo_mos plcmos speechmos (MS) paper
6 x PESQ in TorchAudio-Squim squim_no_ref torch_squim_pesq torch_squim paper
7 x STOI in TorchAudio-Squim squim_no_ref torch_squim_stoi torch_squim paper
8 x SI-SDR in TorchAudio-Squim squim_no_ref torch_squim_si_sdr torch_squim paper
9 x Singing voice MOS singmos singmos singmos paper
10 x Sheet SSQA MOS Models sheet_ssqa sheet_ssqa Sheet paper
11 UTMOSv2: UTokyo-SaruLab MOS Prediction System utmosv2 utmosv2 UTMOSv2 paper
12 Speech Contrastive Regression for Quality Assessment without reference (ScoreQ) scoreq_nr scoreq_nr ScoreQ paper
13 x Speech enhancement-based SI-SNR se_snr se_si_snr ESPnet
14 x Speech enhancement-based CI-SDR se_snr se_ci_sdr ESPnet
15 x Speech enhancement-based SAR se_snr se_sar ESPnet
16 x Speech enhancement-based SDR se_snr se_sdr ESPnet
17 x PAM: Prompting Audio-Language Models for Audio Quality Assessment pam pam PAM Paper
18 Speech-to-Reverberation Modulation energy Ratio (SRMR) srmr srmr SRMRpy Paper
19 x Voice Activity Detection (VAD) vad vad_info SileroVAD
20 Speaker Turn Taking (SPK-TT)
21 x Speaking Word/Character Rate (SWR) speaking_rate speaking_rate - -
22 x Auti-spoofing Score (SpoofS) with AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks asvspoof_score asvspoof_score AASIST Paper
23 x Language Identification lid language ESPnet Paper
24 Audiobox Aesthetics audiobox_aesthetics {audiobox_aesthetics_CE, audiobox_aesthetics_CU, audiobox_aesthetics_PC, audiobox_aesthetics_PQ} Audiobox-Aesthetics Paper
25 x Qwen2 Speaker Characteristics - Count qwen2_speaker_count_metric qwen2_speaker_count_metric Qwen2 Audio paper
26 x Qwen2 Speaker Characteristics - Gender qwen2_speaker_gender_metric qwen2_speaker_gender_metric Qwen2 Audio paper
27 x Qwen2 Speaker Characteristics - Age qwen2_speaker_age_metric qwen2_speaker_age_metric Qwen2 Audio paper
28 x Qwen2 Speaker Characteristics - Speech Impairment qwen2_speech_impairment_metric qwen2_speech_impairment_metric Qwen2 Audio paper
29 x Qwen2 Voice Properties - Pitch qwen2_voice_pitch_metric qwen2_voice_pitch_metric Qwen2 Audio paper
30 x Qwen2 Voice Properties - Pitch Range qwen2_pitch_range_metric qwen2_pitch_range_metric Qwen2 Audio paper
31 x Qwen2 Voice Properties - Voice Type qwen2_voice_type_metric qwen2_voice_type_metric Qwen2 Audio paper
32 x Qwen2 Voice Properties - Volume Level qwen2_speech_volume_level_metric qwen2_speech_volume_level_metric Qwen2 Audio paper
33 x Qwen2 Speech Content - Language qwen2_language_metric qwen2_language_metric Qwen2 Audio paper
34 x Qwen2 Speech Content - Register qwen2_speech_register_metric qwen2_speech_register_metric Qwen2 Audio paper
35 x Qwen2 Speech Content - Vocabulary Complexity qwen2_vocabulary_complexity_metric qwen2_vocabulary_complexity_metric Qwen2 Audio paper
36 x Qwen2 Speech Content - Purpose qwen2_speech_purpose_metric qwen2_speech_purpose_metric Qwen2 Audio paper
37 x Qwen2 Speech Delivery - Emotion qwen2_speech_emotion_metric qwen2_speech_emotion_metric Qwen2 Audio paper
38 x Qwen2 Speech Delivery - Clarity qwen2_speech_clarity_metric qwen2_speech_clarity_metric Qwen2 Audio paper
39 x Qwen2 Speech Delivery - Rate qwen2_speech_rate_metric qwen2_speech_rate_metric Qwen2 Audio paper
40 x Qwen2 Speech Delivery - Style qwen2_speaking_style_metric qwen2_speaking_style_metric Qwen2 Audio paper
41 x Qwen2 Speech Delivery - Emotional Vocalizations qwen2_laughter_crying_metric qwen2_laughter_crying_metric Qwen2 Audio paper
42 x Qwen2 Interaction Patterns - Overlapping Speech qwen2_overlapping_speech_metric qwen2_overlapping_speech_metric Qwen2 Audio paper
43 x Qwen2 Recording Environment - Background qwen2_speech_background_environment_metric qwen2_speech_background_environment_metric Qwen2 Audio paper
44 x Qwen2 Recording Environment - Quality qwen2_recording_quality_metric qwen2_recording_quality_metric Qwen2 Audio paper
45 x Qwen2 Recording Environment - Channel Type qwen2_channel_type_metric qwen2_channel_type_metric Qwen2 Audio paper
46 x Qwen2.5-Omni Speech and Singing Characteristics qwen_omni_{sub_metrics}, qwen_omni_{sub_metrics}_metric qwen_omni_{sub_metrics} Qwen2.5-Omni -
47 x Dimensional Emotion emo_vad arousal_emo_vad, valence_emo_vad, dominance_emo_vad w2v2-how-to paper
48 x Uni-VERSA (Versatile Speech Assessment with a Unified Framework) universa, universa_noref, universa_audioref, universa_textref, universa_fullref universa_{sub_metrics} Uni-VERSA paper
49 ARECHO (Audio Reference Echo Cancellation and Codec Quality Assessment) - No Reference arecho, arecho_noref arecho_{sub_metrics} ARECHO paper
50 x DNSMOS Pro: A Reduced-Size DNN for Probabilistic MOS of Speech pseudo_mos dnsmos_pro_bvcc DNSMOSPro paper
51 x DNSMOS Pro: A Reduced-Size DNN for Probabilistic MOS of Speech pseudo_mos dnsmos_pro_nisqa DNSMOSPro paper
52 x DNSMOS Pro: A Reduced-Size DNN for Probabilistic MOS of Speech pseudo_mos dnsmos_pro_vcc2018 DNSMOSPro paper
53 WV-MOS (MOS score prediction by fine-tuned wav2vec2.0 model) wvmos wvmos wvmos paper
54 SIG-MOS sigmos {SIGMOS_COL, SIGMOS_DISC, SIGMOS_LOUD, SIGMOS_REVERB, SIGMOS_SIG, SIGMOS_OVRL} sigmos paper
55 x VQScore (Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech) vqscore vqscore VQScore paper
56 x Singing voice MOS pseudo_mos singmos_pro singmos paper
57 SongEval (A Benchmark Dataset for Song Aesthetics Evaluation) songeval {songeval_coherence, songeval_musicality, songeval_memorability, songeval_clarity, songeval_naturalness} SongEval paper
58 Multivariate Probabilistic Assessment of Speech Quality multigauss multigauss_{mos,noi,col,dis,loud}, multigauss_covariance MultiGauss paper
59 Korean Mean Opinion Score (KOMOS) pseudo_mos komos KOMOS paper
60 WhiSQA single/multi-head (checkpoint parity pending) whisqa whisqa_mos; optional quality dimensions Setup and validation paper
61 DNSMOS-C dataset-specific (checkpoint parity pending) dnsmos_c dnsmos_c_mean, dnsmos_c_variance Setup and validation paper
62 Distill-MOS variant 7 (adapter; checkpoint parity pending) distill_mos distill_mos Setup and validation paper

Dependent Metrics

Number Auto-Install Metric Name (Auto-Install) Key in config Key in report Code Source References
1 x Mel Cepstral Distortion (MCD) mcd_f0 mcd espnet and s3prl-vc paper
2 x F0 Correlation mcd_f0 f0_corr espnet and s3prl-vc paper
3 x F0 Root Mean Square Error mcd_f0 f0_rmse espnet and s3prl-vc paper
4 x Signal-to-interference Ratio (SIR) signal_metric sir espnet -
5 x Signal-to-artifact Ratio (SAR) signal_metric sar espnet -
6 x Signal-to-distortion Ratio (SDR) signal_metric sdr espnet -
7 x Convolutional scale-invariant signal-to-distortion ratio (CI-SDR) signal_metric ci-sdr ci_sdr paper
8 x Scale-invariant signal-to-noise ratio (SI-SNR) signal_metric si-snr espnet paper
9 x Perceptual Evaluation of Speech Quality (PESQ) pesq pesq pesq paper
10 x Short-Time Objective Intelligibility (STOI) stoi stoi pystoi paper
11 Manual Speech BERT Score discrete_speech speech_bert discrete speech metric paper
12 Manual Discrete Speech BLEU Score discrete_speech speech_bleu discrete speech metric paper
13 Manual Discrete Speech Token Edit Distance discrete_speech speech_token_distance discrete speech metric paper
14 Dynamic Time Warping Cost Metric warpq warpq WARP-Q paper
15 Speech Contrastive Regression for Quality Assessment with reference (ScoreQ) scoreq_ref scoreq_ref ScoreQ paper
16 2f-Model
17 x Log-Weighted Mean Square Error log_wmse log_wmse log_wmse
18 x ASR-oriented Mismatch Error Rate (ASR-Mismatch) asr_match asr_match_error_rate - -
19 Virtual Speech Quality Objective Listener (VISQOL) visqol visqol google-visqol paper
20 Frequency-Weighted SEGmental SNR (FWSEGSNR) pysepm pysepm_fwsegsnr pysepm Paper
21 Weighted Spectral Slope (WSS) pysepm pysepm_wss pysepm Paper
22 Cepstrum Distance Objective Speech Quality Measure (CD) pysepm pysepm_cd pysepm Paper
23 Composite Objective Speech Quality (composite) pysepm pysepm_Csig, pysepm_Cbak, pysepm_Covl pysepm Paper
24 Coherence and speech intelligibility index (CSII) pysepm pysepm_csii_high, pysepm_csii_mid, pysepm_csii_low pysepm Paper
25 Normalized-covariance measure (NCM) pysepm pysepm_ncm pysepm Paper
26 x Uni-VERSA (Versatile Speech Assessment with a Unified Framework) with Paired Reference universa universa_{sub_metrics} Uni-VERSA paper
27 x Chroma-related Alignment chroma_alignment chroma_{stft,cqt,cens}_{cosine, euclidean}_dtw{"", _log, _raw} - -
28 x Deep Perceptual Audio Metric (DPAM) dpam dpam_distance PerceptualAudio_Pytorch paper
29 x Contrastive learning-based Deep Perceptual Audio Metric (CDPAM) cdpam cdpam_distance PerceptualAudio paper
30 MAPSS Perceptual Separation and Perceptual Match mapss mapss_{ps,pm}_{source}; frame-level tables in the MAPSS result directory MAPSS paper

MAPSS uses a dedicated ordered multi-source input path because each estimate must remain paired with its corresponding reference and at least two pairs are required. Install the pinned optional backend with tools/install_mapss.sh, then pass one source-specific SCP per position through --pred_sources and --gt_sources; see egs/separate_metrics/mapss.yaml. MAPSS 1.1.2 supports Python 3.10--3.12 and requires transformers<4.53, so it is intentionally not part of the VERSA base installation. Its first use downloads the selected pretrained backbone. VERSA retains the frame-level tables because the diagnostic PS mean is not the paper's formal utterance-level aggregation.

Non-match Metrics

Number Auto-Install Metric Name (Auto-Install) Key in config Key in report Code Source References
1 NORESQA : A Framework for Speech Quality Assessment using Non-Matching References noresqa noresqa Noresqa Paper
2 x MOS in TorchAudio-Squim squim_ref torch_squim_mos torch_squim paper
3 x ESPnet Speech Recognition-based Error Rate espnet_wer espnet_wer ESPnet paper
4 x ESPnet-OWSM Speech Recognition-based Error Rate owsm_wer owsm_wer ESPnet paper
5 x OpenAI-Whisper Speech Recognition-based Error Rate (WER/CER, optional PER) whisper_wer whisper_wer Whisper paper
6 Faster-Whisper Speech Recognition-based Error Rate fwhisper_wer fwhisper_wer Faster-Whisper -
7 NVIDIA Conformer-Transducer X-Large Speech Recognition-based Error Rate nemo_wer nemo_wer NeMo paper
8 x Facebook Hubert-Large-Finetuned Speech Recognition-based Error Rate hubert_wer hubert_wer HuBERT paper
9 Emotion2vec similarity (emo2vec) emo2vec_similarity emotion_similarity emo2vec paper
10 x Speaker Embedding Similarity speaker spk_similarity espnet, transformers x-vector (e.g. WavLM) paper
11 NOMAD: Unsupervised Learning of Perceptual Embeddings For Speech Enhancement and Non-Matching Reference Audio Quality Assessment nomad nomad Nomad paper
12 Contrastive Language-Audio Pretraining Score (CLAP Score) clap_score clap_score frechet-audio-distance paper
13 Accompaniment Prompt Adherence (APA) apa apa Sony-audio-metrics paper
14 Log Likelihood Ratio (LLR) pysepm pysepm_llr pysepm Paper
15 x Uni-VERSA (Versatile Speech Assessment with a Unified Framework) with Paired Text universa universa_{sub_metrics} Uni-VERSA paper
16 Singer Embedding Similarity singer singer_similarity SSL-Singer-Identity paper

Distributional Metrics (in verifying)

Number Auto-Install Metric Name (Auto-Install) Key in config Key in report Code Source References
1 Frechet Audio Distance (FAD) fad fad fadtk paper
2 Individual Frechet Audio Distance individual_fad individual_fad fadtk paper
3 Kullback-Leibler Divergence on Embedding Distribution kl_embedding kl_embedding Stability-AI
4 Audio Density Score audio_density_coverage audio_density Sony-audio-metrics paper
5 Audio Coverage Score audio_density_coverage audio_coverage Sony-audio-metrics paper
6 KID : Kernel Distance Metric for Audio/Music Quality kid kid_mean, kid_std KID Paper

Acknowledgement

We sincerely thank all the open-source implementations listed in https://github.com/shinjiwlab/versa/tree/main#list-of-metrics

Uni-VERSA and ARECHO (ESPnet AQA)

These backends use espnet2.bin.aqa_inference.AqaInference. Install ESPnet with the AQA task on Python 3.12 or 3.13; the external extra pins an upstream commit containing AQA. Older espnet_inference fork installations need updating. The VERSA metric names, YAML configs, Hugging Face model IDs, and score prefixes remain unchanged. See the AQA recipe guide for current inference and fine-tuning instructions.