Research Repository

Visual Complexity and Its Effects on Referring Expression Generation

Elsner, Micha and Clarke, Alasdair DF and Rohde, Hannah (2018) 'Visual Complexity and Its Effects on Referring Expression Generation.' Cognitive Science, 42 (S4). pp. 940-973. ISSN 0364-0213

realtime.pdf - Accepted Version

Download (1MB) | Preview


Speakers’ perception of a visual scene influences the language they use to describe it—which objects they choose to mention and how they characterize the relationships between them. We show that visual complexity can either delay or facilitate description generation, depending on how much disambiguating information is required and how useful the scene’s complexity can be in providing, for example, helpful landmarks. To do so, we measure speech onset times, eye gaze, and utterance content in a reference production experiment in which the target object is either unique or non-unique in a visual scene of varying size and complexity. Speakers delay speech onset if the target object is non-unique and requires disambiguation, and we argue that this reflects the cost of deciding on a high-level strategy for describing it. The eye-tracking data demonstrates that these delays increase when the speaker is able to conduct an extensive early visual search, implying that when a speaker scans too little of the scene early on, they may decide to begin speaking before becoming aware that their description is underspecified. Speak- ers’ content choices reflect the visual makeup of the scene—the number of distractors present and the availability of useful landmarks. Our results highlight the complex role of visual perception in reference production, showing that speakers can make good use of complexity in ways that reflect their visual processing of the scene.

Item Type: Article
Uncontrolled Keywords: referring expression generation; psycholinguistics;; sentence processing; visual search
Subjects: B Philosophy. Psychology. Religion > BF Psychology
Divisions: Faculty of Science and Health
Faculty of Science and Health > Psychology, Department of
SWORD Depositor: Elements
Depositing User: Elements
Date Deposited: 25 Apr 2017 09:22
Last Modified: 06 Jan 2022 13:32

Actions (login required)

View Item View Item