Speaker: Alex Ihler, UC Irvine

Title:  Belief Propagation Algorithms for Crowdsourcing

Abstract:

Obtaining expert information has always been a major bottleneck for machine learning methods.  "Crowdsourcing" techniques such as Amazon's Mechanical Turk have become a popular mechanism to access the power of human intelligence, for example to label large datasets.  However, this raises the computational task of properly aggregating the crowdsourced predictions provided by a collection of unreliable and diverse annotators.

On the other side, graphical models are powerful tools for reasoning about systems with complicated dependency structures. In this talk, we approach the crowdsourcing problem by transforming it into a standard inference problem in graphical models, and apply powerful inference algorithms such as belief propagation (BP). We show both the naive majority voting and a recent algorithm by Karger, Oh, and Shah are special cases of our BP algorithm when taking particular modeling choices. With more careful modeling choices, we show that our simple BP method performs very well on both simulated and real-world datasets, competitively with state-of-the-art algorithms based on more complicated modeling assumptions. Our work sheds light on the important tradeoff between better modeling choices and better inference algorithms.

Joint work with Qiang Liu