Crowdsourcing is the de-facto standard for gathering annotated data. While, in theory, data annotation tasks are assumed to be attempted by workers independently, in practice, data annotation tasks are often grouped into batches to be presented and annotated by workers together, in order to save on the time or cost overhead of providing instructions or necessary background. Thus, even though independence is usually assumed between annotations on data items within the same batch, in most cases, a worker's judgment on a data item can still be affected by other data items within the batch, leading to additional errors in collected labels. In this paper, we study the data annotation bias when data items are presented as batches to be judged by workers simultaneously. We propose a novel worker model to characterize the annotating behavior on data batches, and present how to train the worker model on annotation data sets. We also present a debiasing technique to remove the effect of such annotation bias from adversely affecting the accuracy of labels obtained. Our experimental results on both synthetic data and real-world data demonstrate the effectiveness of our proposed method.
Keywords: Crowdsourcing; annotation bias; worker model.
Crowd control: Effectively utilizing unscreened crowd workers for biomedical data annotation.J Biomed Inform. 2017 May;69:86-92. doi: 10.1016/j.jbi.2017.04.003. Epub 2017 Apr 4. J Biomed Inform. 2017. PMID: 28389234
Cameras for Public Health Surveillance: A Methods Protocol for Crowdsourced Annotation of Point-of-Sale Photographs.JMIR Res Protoc. 2014 Apr 9;3(2):e22. doi: 10.2196/resprot.3277. JMIR Res Protoc. 2014. PMID: 24717168 Free PMC article.
Robust Subjective Visual Property Prediction from Crowdsourced Pairwise Labels.IEEE Trans Pattern Anal Mach Intell. 2016 Mar;38(3):563-77. doi: 10.1109/TPAMI.2015.2456887. IEEE Trans Pattern Anal Mach Intell. 2016. PMID: 27046498
Harnessing the power of crowds: crowdsourcing as a novel research method for evaluation of acne treatments.Am J Clin Dermatol. 2012 Dec 1;13(6):405-16. doi: 10.2165/11634040-000000000-00000. Am J Clin Dermatol. 2012. PMID: 23039257 Review.
Adaptation of the older worker to occupational challenges.Work. 2004;22(2):71-8. Work. 2004. PMID: 15004340 Review.
Cited by 1 article
Optimizing Open-Ended Crowdsourcing: The Next Frontier in Crowdsourced Data Management.Bull Tech Comm Data Eng. 2016 Dec;39(4):26-37. Bull Tech Comm Data Eng. 2016. PMID: 28951893 Free PMC article.