VERSA
VERSADocs

Usage guide

Run evaluations locally or across a compute cluster.

On this page

Basic Usage

Use --cache_folder to place downloads for all metrics below one shareable root. VERSA shares subdirectories for metrics using the same model backend (such as Hugging Face, Whisper, ESPnet, and Torch Hub) and isolates other metric-specific files. An explicit cache_dir in the score YAML takes precedence:

python versa/bin/scorer.py \
    --score_config egs/speech_cpu.yaml \
    --pred test/test_samples/test2 \
    --gt test/test_samples/test1 \
    --output_file test_result \
    --io dir \
    --cache_folder /shared/versa_cache

# Direct usage with file paths
python versa/bin/scorer.py \
    --score_config egs/speech_cpu.yaml \
    --gt test/test_samples/test1 \
    --pred test/test_samples/test2 \
    --output_file test_result \
    --io dir

# With SCP-style input
python versa/bin/scorer.py \
    --score_config egs/speech_cpu.yaml \
    --gt test/test_samples/test1.scp \
    --pred test/test_samples/test2.scp \
    --output_file test_result \
    --io soundfile \
    --num_workers 4

# With Kaldi-ARK style input (compatible with ESPnet)
python versa/bin/scorer.py \
    --score_config egs/speech_cpu.yaml \
    --gt test/test_samples/test1.scp \
    --pred test/test_samples/test2.scp \
    --output_file test_result \
    --io kaldi

# Including text transcription information
python versa/bin/scorer.py \
    --score_config egs/separate_metrics/wer_tiny.yaml \
    --gt test/test_samples/test1.scp \
    --pred test/test_samples/test2.scp \
    --output_file test_result \
    --text test/test_samples/text \
    --io soundfile

# Resume an interrupted utterance-level scoring run
python versa/bin/scorer.py \
    --score_config egs/speech_cpu.yaml \
    --gt test/test_samples/test1.scp \
    --pred test/test_samples/test2.scp \
    --output_file test_result \
    --io soundfile \
    --resume

# Load and score one metric at a time to reduce peak GPU memory
python versa/bin/scorer.py \
    --score_config egs/speech_gpu.yaml \
    --gt test/test_samples/test1.scp \
    --pred test/test_samples/test2.scp \
    --output_file test_result \
    --io soundfile \
    --use_gpu True \
    --scoring_mode metric

--resume reads existing JSONL rows from --output_file and continues the run from the per-metric completion records they carry. An utterance is skipped only when every configured metric completed successfully, or explicitly abstained, under the same metric name, evaluation configuration, and input identity; missing and failed metrics are recomputed and merged into the stored row, and previously successful metrics are never recomputed. This is useful for long-running evaluations that are interrupted or restarted.

Each row carries a _versa_completion record with the schema version, the input identity, and the status of every attempted metric (success, failed, skipped, or abstained) together with a short error category for failures. The field name starts with an underscore, so it is excluded from score summaries, report columns, and rankings; existing score keys are unchanged.

Rows written before this record existed have no identity, so --resume recomputes them by default. Pass --legacy_resume trust to keep the historical behavior of trusting any row that already has a value. --input_identity selects how changed inputs are detected: path (the default) compares input locations, while content also hashes each input file so audio edited in place is rescored, at the cost of reading every input once per run.

When resuming, the result file is first rewritten atomically as one record per stored key, which collapses repeated keys and drops a truncated final record; every readable row is kept, including partially completed utterances and keys outside the current input set. Recomputed rows are appended as they finish, and a final atomic rewrite keeps only the newest record per utterance. A resumed run that is interrupted again therefore loses nothing: the superseded records it leaves behind are resolved by the next resume, so aggregate a result file only after a run has finished. Retained rows keep their original positions while new keys are appended in input key order, so a resumed JSONL file may not be globally sorted by input key; the returned scores are ordered by input key.

--strict exits unsuccessfully when a metric fails to load, fails to run, or an utterance is skipped. The results that did complete are still written, including a requested --report, because the completeness check runs last. The default tolerant mode records the same status counts and exits normally. Both modes log a run summary naming requested and loaded metrics, and scored, resumed, and skipped utterances.

--num_workers runs utterance-level CPU scoring in local worker processes while preserving input key order in newly written JSONL output for non-resume runs. GPU scoring, metric-oriented scoring (--scoring_mode metric), and corpus/distributional metrics remain serial in this first implementation; --num_workers > 1 cannot be combined with --use_gpu, --scoring_mode metric, or corpus-only configurations.

--scoring_mode metric loads and runs one metric at a time, then releases metric resources before moving to the next metric. This can reduce peak GPU memory when many model-backed metrics are configured together.

Metric Discovery

Use the versa-score CLI to inspect available metrics and generate starter configs without reading the full metrics table manually:

# List metrics registered in the current environment
versa-score --list-metrics

# Show metadata, dependencies, references, and aliases for one metric
versa-score --describe-metric pesq

# Print a recommended YAML score config for a task/device pair
versa-score --recommend-config --task tts --device gpu

--list-metrics also supports --metric-category and --metric-type filters, for example:

versa-score --list-metrics --metric-category dependent --metric-type float

Distributed Evaluation with Slurm

The launcher defaults to 4 CPUs and a 12-hour limit per job. It checks the actual chunk count against MAX_JOBS (default 50) and the combined requested CPU time against MAX_TOTAL_CPU_HOURS (default 5000) before submitting. These limits apply to one invocation, not all jobs already running on the cluster. Use a fresh score directory; existing split files are rejected. Submission requires confirmation; pass --yes for unattended runs (resource checks still apply).

Override resources with CPUS, MEM (MB per CPU), CPU_TIME, and GPU_TIME. Numeric settings must be integers from 1 to 999999999. Time limits accept Slurm's finite positive time formats, including minutes, minutes:seconds, hours:minutes:seconds, and days-hours[:minutes[:seconds]]. Seconds round up to a minute for the estimate. Jobs request one task on one node; cluster allocation and billing policies may differ from the requested CPU estimate. CPU_OTHER_OPTS and GPU_OTHER_OPTS accept space-separated --name=value options for account, qos, constraint, reservation, dependency, exclude, nodelist, mail-type, mail-user, and comment. Resource overrides, arrays, and other implicit SBATCH_* settings are rejected to keep the estimate valid.

# Option 1: With ground truth speech
./launch_slurm.sh \
  <pred_speech_scp> \
  <gt_speech_scp> \
  <score_dir> \
  <split_job_num> 

# Option 2: Without ground truth speech
./launch_slurm.sh \
  <pred_speech_scp> \
  None \
  <score_dir> \
  <split_job_num>

# Aggregate results
cat <score_dir>/result/*.result.cpu.txt > <score_dir>/utt_result.cpu.txt
cat <score_dir>/result/*.result.gpu.txt > <score_dir>/utt_result.gpu.txt

# Visualize results
python scripts/show_result.py <score_dir>/utt_result.cpu.txt
python scripts/show_result.py <score_dir>/utt_result.gpu.txt

Explore egs/*.yaml for configuration examples for different evaluation scenarios.