Event

A human benchmark for automatic language recognition

David van Leeuwen and Rosemary Orr

AMIDA

Monday, June 30, 2008
12:30

In speech technology systems are designed to perform specific tasks that humans are believed to be good at, such as speech recognition. Every once in a while, it is good to assess what the performance of humans actually is compared to the state of the art of machines. At ICSI, we assessed the capability of human subjects to recognize a variety of languages. For this, we used the same material and evaluation measures as were used in the NIST 2007 Language Recognition Evaluation (LRE). In a pilot study performed prior to this work, the level of proficiency of a subject in a particular language turned out to be an important factor of the subject's performance to detect that language. In this study we have recruited subjects to fill 5 categories of language proficiency in all of the 14 target languages of LRE-2007. We will present the first analyses of the results, together with a comparison to system performance in 2007. A further study aimed at determining the discriminative capabilities of human subjects when they are presented with two speech fragments of language(s) they don't know, and the task is to determine whether these fragments are spoken in the same language or not. In this very challenging task, humans still seem to show a superior kind of cognitive processing abilities.