Hugging Face has introduced Community Evals, a new initiative designed to enhance the transparency and community involvement in AI model benchmarking. This innovative platform empowers contributors to actively participate in the evaluation of AI models, thereby promoting openness and shared accountability in assessing and comparing these models.
Community Evals is launched as a beta feature that enables benchmark datasets hosted on the Hugging Face Hub to maintain their own leaderboards. This system automatically aggregates evaluation results from various model repositories, fostering a more standardized and traceable approach to benchmarking.
Benchmarks on the platform are designed to ensure reproducibility by defining their evaluation specifications using an <code>eval.yaml</code> file in the Inspect AI format. This standardized approach allows model authors and community members to submit their scores through pull requests, which are meticulously versioned via Git for complete traceability and transparency of changes.
Initial supported benchmarks in this effort include MMLU-Pro, GPQA, and HLE. Community Evals aims to tackle the existing inconsistencies in reported benchmark scores that often arise across research papers and platforms. By decentralizing the evaluation reporting process, Hugging Face seeks to introduce a more transparent and standardized method, ensuring that AI model assessments are reliable and equitable.

Collections

Hugging Face has launched Community Evals, a beta platform that lets contributors participate in AI model benchmarking by hosting benchmark datasets with their own leaderboards on the Hugging Face Hub. Evaluation specs are defined via an `eval.yaml` file in Inspect AI format, with scores submitted through pull requests versioned by Git for full traceability. Initial supported benchmarks include MMLU-Pro, GPQA, and HLE. The initiative aims to address inconsistencies in reported benchmark scores across research papers and platforms by decentralizing and standardizing the evaluation process.

Hugging Face Launches Community Evals for Transparent AI Model Benchmarking