A Study of Multimodal Addressee Detection in Human-Human-Computer Interaction

TitleA Study of Multimodal Addressee Detection in Human-Human-Computer Interaction
Publication TypeJournal Article
Year of Publication2015
AuthorsTsai, T.. J., Stolcke A., & Slaney M.
Published inIEEE Transactions on Multimedia
Other Numbers3818

The goal of addressee detection is to answer the question , “Are you talking to me?” When a dialogue system interacts with multiple users, it is crucial to detect when a user is speaking to the system as opposed to another person. We study this problem in a multimodal scenario, using lexical, acoustic, visual, dialogue state, and beamforming information. Using data from a multiparty dialogue system, we quantify the benefits of using multiple modalities over using a single modality. We also assess the relative importance of the various modalities, as well as of key individual features, in estimating the addressee. We find that energy-based acoustic features are by far the most important, that information from speech recognition and system state is useful as well, and that visual and beamforming features provide little additional benefit. While we find that head pose is affected by whom the speaker is addressing, it yields little nonredundant information due to the system acting as a situational attractor. Our findings would be relevant to multiparty, open-world dialogue systems in which the agent plays an active, conversational role, such as an interactive assistant deployed in a public, open space. For these scenarios , our study suggests that acoustic, lexical, and system-state information is an effective and practical combination of modalities to use for addressee detection. We also consider how our analyses might be affected by the ongoing development of more realistic, natural dialogue systems.


This work was partially supported by funding provided to ICSI through Microsoft Research. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of Microsoft Research.

Bibliographic Notes

IEEE Transactions on Multimedia, Vol. 17, Issue 9, pp. 1550-1561

Abbreviated Authors

T.J. Tsai, A. Stolcke, and M. Slaney

ICSI Research Group


ICSI Publication Type

Article in journal or magazine