Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2015 Aug;2015:1593-1602.
doi: 10.1145/2783258.2783316.

Debiasing Crowdsourced Batches

Affiliations
Free PMC article

Debiasing Crowdsourced Batches

Honglei Zhuang et al. KDD. .
Free PMC article

Abstract

Crowdsourcing is the de-facto standard for gathering annotated data. While, in theory, data annotation tasks are assumed to be attempted by workers independently, in practice, data annotation tasks are often grouped into batches to be presented and annotated by workers together, in order to save on the time or cost overhead of providing instructions or necessary background. Thus, even though independence is usually assumed between annotations on data items within the same batch, in most cases, a worker's judgment on a data item can still be affected by other data items within the batch, leading to additional errors in collected labels. In this paper, we study the data annotation bias when data items are presented as batches to be judged by workers simultaneously. We propose a novel worker model to characterize the annotating behavior on data batches, and present how to train the worker model on annotation data sets. We also present a debiasing technique to remove the effect of such annotation bias from adversely affecting the accuracy of labels obtained. Our experimental results on both synthetic data and real-world data demonstrate the effectiveness of our proposed method.

Keywords: Crowdsourcing; annotation bias; worker model.

Figures

Figure 1
Figure 1
Example of correlation between annotations on data items in the same batch. Workers are asked to label whether a review on the movie “The Imitation Game” crawled from IMDb is positive. Assign each review-movie pair to different workers separately can be costly, while assigning a batch of reviews together with a movie to workers might affect workers' judgments.
Figure 2
Figure 2
Learning worker model from the synthetic training data set.
Figure 3
Figure 3
Analysis of estimation error of parameters in the worker model under different configurations.
Figure 4
Figure 4
Learning worker model from the comments training data set.
Figure 5
Figure 5
Performance of debiasing strategies on synthetic data sets generated by setting both mL/nL and mU/nU as 2, 5, 10, 20 respectively.
Figure 6
Figure 6
Performance of debiasing strategies on synthetic data sets generated by different size of training data set nL (mL = 10nL), while the size of testing data set remains nU = 5,000 and mU = 50, 000.
Figure 7
Figure 7
Performance of debiasing strategies on comments data set where the training data set is randomly sampled from the original training set with different size of mL, while the testing data set remains the same.

Similar articles

See all similar articles

Cited by 1 article

LinkOut - more resources

Feedback