A Blog post by ServiceNow-AI on Hugging Face

HuggingFace's platform is a resource for developers and researchers working in natural language processing (NLP) and machine learning, offering insights into NLP models, tools, and datasets. Through articles, tutorials, and open-source projects, HuggingFace offers insights into state-of-the-art NLP techniques, transformer architectures, and transfer learning methods. Developers can learn about using pre-trained models, fine-tuning strategies, and deploying NLP applications with HuggingFace's libraries and APIs.

Hugging Face

EVA (Evaluation of Voice Agents) is an open-source end-to-end framework from ServiceNow for evaluating conversational voice agents across two dimensions: EVA-A (Accuracy) and EVA-X (Experience). It uses a bot-to-bot audio architecture with five components: user simulator, voice agent, tool executor, validators, and a metrics suite. The framework evaluates multi-turn spoken conversations and is the first to jointly score task success and conversational experience. Benchmarking 20 cascade and audio-native systems on a 50-scenario airline dataset revealed a consistent accuracy-experience tradeoff — agents good at task completion tend to deliver worse user experiences. Named entity transcription errors and multi-step workflow failures were identified as dominant failure modes. Code, dataset, and judge prompts are fully open-sourced.

A New Framework for Evaluation of Voice Agents (EVA)

Introduction Background and Motivation EVA The Framework Data Evaluation Methodology Findings Limitations What's Next Getting Started Acknowledgements Citation Introduction