Research Repository

Inter-Coder Agreement for Computational Linguistics

Artstein, Ron and Poesio, Massimo (2008) 'Inter-Coder Agreement for Computational Linguistics.' Computational Linguistics, 34 (4). pp. 555-596. ISSN 0891-2017


Download (335kB) | Preview


<jats:p> This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder. </jats:p>

Item Type: Article
Subjects: P Language and Literature > P Philology. Linguistics
Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: Faculty of Science and Health
Faculty of Science and Health > Computer Science and Electronic Engineering, School of
SWORD Depositor: Elements
Depositing User: Elements
Date Deposited: 05 Mar 2013 15:38
Last Modified: 15 Jan 2022 00:34

Actions (login required)

View Item View Item