Paun, Silviu and Carpenter, Bob and Chamberlain, JD and Hovy, Dirk and Kruschwitz, Udo and Poesio, Massimo (2018) Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics, 6 (2018). pp. 571-585. DOI https://doi.org/10.1162/tacl_a_00040
Paun, Silviu and Carpenter, Bob and Chamberlain, JD and Hovy, Dirk and Kruschwitz, Udo and Poesio, Massimo (2018) Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics, 6 (2018). pp. 571-585. DOI https://doi.org/10.1162/tacl_a_00040
Paun, Silviu and Carpenter, Bob and Chamberlain, JD and Hovy, Dirk and Kruschwitz, Udo and Poesio, Massimo (2018) Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics, 6 (2018). pp. 571-585. DOI https://doi.org/10.1162/tacl_a_00040
Abstract
The analysis of crowdsourced annotations in NLP is concerned with identifying 1) gold standard labels, 2) annotator accuracies and biases, and 3) item difficulties and error patterns. Traditionally, majority voting was used for 1), and coefficients of agreement for 2) and 3). Lately, model-based analysis of corpus annotations have proven better at all three tasks. But there has been relatively little work comparing them on the same datasets. This paper aims to fill this gap by analyzing six models of annotation, covering different approaches to annotator ability, item difficulty, and parameter pooling (tying) across annotators and items. We evaluate these models along four aspects: comparison to gold labels, predictive accuracy for new annotations, annotator characterization, and item difficulty, using four datasets with varying degrees of noise in the form of random (spammy) annotators. We conclude with guidelines for model selection, application, and implementation.
Item Type: | Article |
---|---|
Uncontrolled Keywords: | annotation; annotation models; Bayesian models; computational linguistics; crowdsourcing |
Subjects: | P Language and Literature > P Philology. Linguistics Q Science > QA Mathematics > QA75 Electronic computers. Computer science |
Divisions: | Faculty of Science and Health Faculty of Science and Health > Computer Science and Electronic Engineering, School of |
SWORD Depositor: | Unnamed user with email elements@essex.ac.uk |
Depositing User: | Unnamed user with email elements@essex.ac.uk |
Date Deposited: | 09 Nov 2018 13:42 |
Last Modified: | 23 Sep 2022 19:29 |
URI: | http://repository.essex.ac.uk/id/eprint/23420 |
Available files
Filename: tacl_a_00040.pdf
Licence: Creative Commons: Attribution-Noncommercial-No Derivative Works 3.0