Crowdsourcing with the Drift Diffusion Model of Decision Making: A Neuroscientific Approach
Utilizing crowdsourcing to label datasets is a familiar practice, especially when the reliability of the data annotators is uncertain. This uncertainty poses a challenge in estimating the true labels within datasets. Traditionally, techniques have relied on the imposition of beta or dirichlet distributions to gauge annotator reliability. However, a groundbreaking study published in Scientific Reports takes a novel approach by incorporating a neuroscientifically validated decision-making model—the drift diffusion model—into the crowdsourcing annotator’s decision process.
The study embarked on pioneering experiments with synthetically generated data featuring sinusoidal decision boundaries, using variational inference to predict both the ground truth labels and parameters related to the annotators. These experiments confirmed the new model’s efficacy, rivaling the state-of-the-art technique, SVGPCR, in predicting labels and enhancing the capability to predict neuroscientific biomarkers of annotators.
Crowdsourcing’s vast applicability spans various fields such as medicine, engineering, and behavioral science, enriching it with diverse data annotation tasks. By focusing on annotator reliability—especially through the lens of sensitivity and specificity in binary labeling—researchers have aimed to create classifiers that generalize to new data. Techniques range from frequentist approaches like the Expectation Maximization algorithm to Bayesian approaches involving beta or dirichlet distributions as priors.
Distinctively, this paper proposes adopting the drift diffusion model as a prior on annotator reliability. This model extensively applies to two-choice decisions, mirroring the quick, binary decisions annotators make, and has confirmed neurological correlates, including connections to the prefrontal cortex, subthalamic nucleus, basal ganglia, and dopaminergic activity in perceptual decision-making.
Previous literature merely scrapes the surface of incorporating psychological or neuroscientifically validated models in crowdsourcing. Some studies have explored using functional near-infrared spectroscopy (fNIRS) and brain-computer interfaces (BCI) in crowdsourcing tasks, while others have collected psychological and neurological properties of annotators. These studies hint at the integration of behavioral and health economics into crowdsourcing, providing a fertile area for further exploration.
The drift diffusion model operates on choosing between two options, modeled by a ‘evidence level,’ a latent stochastic process reflecting the accumulated evidence for a decision. This process evolves over time towards an upper or lower boundary, signifying the correct or incorrect decisions. The paper argues for using this model to predict annotator behaviors and decisions in binary labeling tasks, employing variational inference for parameter and ground-truth label estimation.
In conclusion, by leveraging a neuroscientifically validated decision-making model, this approach not only pioneers in predicting annotator reliability but also opens doors to understanding the neurological and psychological underpinnings of decision-making in crowdsourcing. While the current study focuses on binary labeling, its success paves the way for future investigations into non-binary labeling scenarios, promising far-reaching implications in the understanding and application of crowdsourcing.
