TY - GEN

T1 - On the Verification Complexity of Group Decision-Making Tasks

AU - Amir, Ofra

AU - Shahar, Yuval

AU - Gal, Yaakov

AU - Ilany, Litan

N1 - Publisher Copyright: Copyright © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

PY - 2013/11/10

Y1 - 2013/11/10

N2 - A popular use of crowdsourcing is to collect and aggregate individual worker responses to problems to reach a correct answer. This paper studies the relationship between the computation complexity class of problems, and the ability of a group to agree on a correct solution. We hypothesized that for NP-Complete (NPC) problems, groups would be able to reach a majority-based correct solution once it was suggested by a group member and presented to the other members, due to the “easy to verify” (i.e., verification in polynomial time) characteristic of this complexity class. In contrast, when posed with PSPACE-Complete (PSC) “hard to verify” problems (i.e., verification in exponential time), groups will not necessarily be able to choose a correct solution even if such a solution has been presented. Consequently, increasing the size of the group is expected to facilitate the ability of the group to converge on a correct solution when solving NPC problems, but not when solving PSC problems. To test this hypothesis we conducted preliminary experiments in which we evaluated people’s ability to solve an analytical problem and their ability to recognize a correct solution. In our experiments, participants were significantly more likely to recognize correct and incorrect solutions for NPC problems than for PSC problems, even for problems of similar difficulties (as measured by the percentage of participants who solved the problem). This is a first step towards formalizing a relationship between the computationally complexity of a problem and the crowd’s ability to converge to a correct solution to the problem.

AB - A popular use of crowdsourcing is to collect and aggregate individual worker responses to problems to reach a correct answer. This paper studies the relationship between the computation complexity class of problems, and the ability of a group to agree on a correct solution. We hypothesized that for NP-Complete (NPC) problems, groups would be able to reach a majority-based correct solution once it was suggested by a group member and presented to the other members, due to the “easy to verify” (i.e., verification in polynomial time) characteristic of this complexity class. In contrast, when posed with PSPACE-Complete (PSC) “hard to verify” problems (i.e., verification in exponential time), groups will not necessarily be able to choose a correct solution even if such a solution has been presented. Consequently, increasing the size of the group is expected to facilitate the ability of the group to converge on a correct solution when solving NPC problems, but not when solving PSC problems. To test this hypothesis we conducted preliminary experiments in which we evaluated people’s ability to solve an analytical problem and their ability to recognize a correct solution. In our experiments, participants were significantly more likely to recognize correct and incorrect solutions for NPC problems than for PSC problems, even for problems of similar difficulties (as measured by the percentage of participants who solved the problem). This is a first step towards formalizing a relationship between the computationally complexity of a problem and the crowd’s ability to converge to a correct solution to the problem.

UR - http://www.scopus.com/inward/record.url?scp=84959864675&partnerID=8YFLogxK

M3 - منشور من مؤتمر

T3 - Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013

SP - 2

EP - 8

BT - Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013

A2 - Hartmann, Bjorn

A2 - Horvitz, Eric

T2 - 1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013

Y2 - 7 November 2013 through 9 November 2013

ER -