SpeechEval : A Domain-Independent User Simulation Platform for Spoken Dialog System Evaluation

Abstract

In this paper, we present the SpeechEval user simulation platform for spoken dialog system (SDS) evaluation. Our user simulation realizes a complete end-to-end user test system with spoken input and output that calls speech-based dialog systems via telephone, and is intended to be used in the development and evaluation of real-life SDSs. We give an overview of the system architecture and present the results of applying our approach to two different kinds of systems in two domains.User simulations for SDS are developed for many different purposes, often to test or train specific system components, like dialog manager, language model or the speech recognition itself. Many user simulations are developed in the context of research systems, which differ from commercially used systems in many aspects. In contrast, our user simulation uses a speech interface, can be carried over to new domains easily by learning characteristics of user behavior from a corpus, and treats the evaluated SDS as a black box.