Artstein, Ron and Poesio, Massimo (2008) Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34 (4). pp. 555-596. DOI https://doi.org/10.1162/coli.07-034-r2
Artstein, Ron and Poesio, Massimo (2008) Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34 (4). pp. 555-596. DOI https://doi.org/10.1162/coli.07-034-r2
Artstein, Ron and Poesio, Massimo (2008) Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34 (4). pp. 555-596. DOI https://doi.org/10.1162/coli.07-034-r2
Abstract
<jats:p> This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder. </jats:p>
Item Type: | Article |
---|---|
Subjects: | P Language and Literature > P Philology. Linguistics Q Science > QA Mathematics > QA75 Electronic computers. Computer science |
Divisions: | Faculty of Science and Health Faculty of Science and Health > Computer Science and Electronic Engineering, School of |
SWORD Depositor: | Unnamed user with email elements@essex.ac.uk |
Depositing User: | Unnamed user with email elements@essex.ac.uk |
Date Deposited: | 05 Mar 2013 15:38 |
Last Modified: | 16 May 2024 16:47 |
URI: | http://repository.essex.ac.uk/id/eprint/5548 |
Available files
Filename: coli%2E07-034-R2.pdf