Reference in Visual Domains

Donna Byron
OSU-CSE Faculty

Fri., Oct. 22nd
3:30pm, 263 Dreese Labs
All interested parties are invited.
Pizza will be served after the talk.

Abstract:
Interpreting noun phrases in natural language requires complex reasoning to track both the flow or topic of the discourse and the semantic properties of items that might be mentioned. Computational models that interpret noun phrases in text have reached a fairly high level of maturity, but spoken language, and especially dialogue, remains a challenge. The current state of the art works for spoken dialogue systems that perform very simple tasks, but does not yet support the integration of information presented in different modalities, such as when a task includes computer-generated graphics or when it is carried out in a physical setting. In these tasks, the discourse, the task, and the visual information all compete to structure the context and provide referents for noun phrases in the dialogue.

One recently launched project in the OSU SLATE lab seeks to build a computational model for interpreting noun phrases in collaborative dialogue for tasks taking place in a visually-rich setting provided by a first-person-shooter computer game. The data collected as part of this task has just begun to be analyzed. Our initial analysis indicates that several extensions to current technology will be required to successfully work with this data. This talk will lay out the fundamental issues at work in interpreting noun phrases, the theory that can be brought to bear, and the extensions necessary to allow existing models, designed for text, to work in multimodal domains.