CMI-Bench

dataset·music instruction following·active

A test-only benchmark converting diverse MIR tasks into standardized music instruction-following evaluation.

Verified facts

Official site	https://github.com/nicolaus625/CMI-bench ↗
Geography	Global
arxiv	2506.12285
huggingface repository	nicolaus625/CMI-bench
size	repository size gb: 57.7 · public source datasets: 20
scope	Genre, emotion, instruments, pitch, key, lyrics, melody, techniques, captioning and beat tasks.
genres	multiple
license	CC BY-NC 4.0 aggregate; source terms also apply
steward	Queen Mary University of London / Yamaha Corporation
creators	Yinghao Ma; Siyou Li; Juntao Yu; Emmanouil Benetos; Akira Maezawa
languages	English instructions; source datasets vary
modalities	audio; natural-language instructions; reference labels; evaluation code
geographies	Global
access method	GitHub evaluation toolkit and Hugging Face test audio.
intended uses	music instruction-following evaluation; audio-text LLM evaluation
consent claims	Inherited from source datasets.
related papers	CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following
annotation method	Source labels are mapped to instructions while retaining task-standard metrics.
collection method	Reformats twenty public MIR datasets as instruction-following tests.
provenance claims	Repository and paper enumerate source datasets and warn that their terms continue to apply.
current availability	Repository and test set available.
commercial use limits	Aggregate release is noncommercial and some sources impose further restrictions.
us market scope basis	Publicly available or materially used in US-facing music/audio-AI research.
limitations biases disputes	Paper reports cultural, chronological and gender bias.; Test-only design does not provide a training corpus.