Artstein, R and Poesio, M (2008) 'Inter-coder agreement for computational linguistics.' Computational Linguistics, 34 (4). 555 - 596. ISSN 0891-2017

Preview

Text coli%2E07-034-R2.pdf Download (335kB) | Preview

Abstract

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks-but that their use makes the interpretation of the value of the coefficient even harder. © 2008 Association for Computational Linguistics.

Item Type:	Article
Subjects:	P Language and Literature > P Philology. Linguistics Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions:	Faculty of Science and Health > Computer Science and Electronic Engineering, School of
Depositing User:	Jim Jamieson
Date Deposited:	05 Mar 2013 15:38
Last Modified:	17 Aug 2017 18:02
URI:	http://repository.essex.ac.uk/id/eprint/5548

Actions (login required)

View Item