Using game responses to gather data

• US 8,032,483 B1
• Filed: 06/29/2005
• Issued: 10/04/2011
• Est. Priority Date: 12/03/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Haveliwala, Taher, Gomes, Benedict, Singhal, Amitabh K.
• Primary Examiner(s): Rones; Charles
• Assistant Examiner(s): Hotelling; Harold A
• Application Number:
  US11/169,279
• Time in Patent Office: 2,288 Days
• Field of Search: None
• US Class Current: 707/602
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A system provides images or questions to multiple game participants and receives labels or answers in response thereto. The system uses the labels or answers for various data gathering purposes.

Claims

1. A method performed by one or more processors associated with one or more server devices, the method comprising:

eliciting, by one or more processors associated with the one or more server devices, user participation in a game by posing queries from a query log to a user, where the query log comprises search queries previously submitted by one or more users;

gathering, by one or more processors associated with the one or more server devices, first information resulting from the user participation in the game;

gathering, by one or more processors associated with the one or more server devices, second information, which is different than the first information, resulting from the user participation in the game;

assigning, by one or more processors associated with the one or more server devices, a score to the user based on the gathered first information;

assigning, by one or more processors associated with the one or more server devices, a weight to the gathered second information based on the assigned score; and

using, by one or more processors associated with the one or more server devices, the weighted, gathered, second information to answer queries subsequently posed to the one or more users.

2. The method of claim 1, where the user participates in the game via a network connected to the one or more server devices.

3. The method of claim 1, where eliciting the user participation in the game comprises:

retrieving, from the query log, a first set of queries and retrieving answers associated with the first set of queries,

retrieving, from the query log, a second set of queries that do not have associated answers, and

posing, to the user, at least one query from the first set and at least one query from the second set.

4. The method of claim 3, where using the weighted, gathered, second information to answer queries, subsequently posed to the one or more users, comprises:

assigning, based on the score assigned to the user, the weight to the user'"'"'s answer to the at least one query from the second set;

receiving the at least one query from the second set, as a query from another user;

selecting, based on the weight, the user'"'"'s answer to the at least one query from the second set; and

providing the selected answer to the other user in response to receiving the at least one query from the second set, as a query from the other user.

5. The method of claim 3, where assigning the score to the user comprises:

assigning, to the user, the score based on whether the received answer to the at least one query from the first set matched a retrieved answer associated with the at least one query from the first set.

6. The method of claim 1, where gathering the first information comprises:

receiving answers, from the user, to the at least one query from the first set; and

gathering the second information comprises;

receiving answers, from the user, to the at least one query from the second set.

7. A computer-readable memory device comprising processing instructions executable by one or more processors, the processing instructions comprising:

instructions for retrieving a first set of queries from a query log that comprises search queries previously submitted by one or more users;

instructions for retrieving answers associated with the first set of queries from the query log;

instructions for retrieving a second set of queries from the query log that do not have associated answers in the query log;

instructions for posing, to multiple users, at least one query from the first set of queries and at least one query from the second set of queries;

instructions for receiving answers to the at least one query from the first set and the at least one query from the second set;

instructions for aggregating the answers to the at least one query from the second set;

instructions for comparing answers to the at least one query from the first set to a retrieved answer associated with the at least one query from the first set;

instructions for assigning a score to each particular user, of the multiple users, based on a quantity of answers, received from the particular user, to queries from the first set of queries that match retrieved answers associated with the first set of queries; and

instructions for selecting an answer for the at least one query from the second set of queries based on the aggregated answers and scores assigned to the multiple users.

8. The computer-readable memory device of claim 7, where the instructions for selecting the answer comprise:

instructions for selecting an answer that was received from at least a particular quantity of users.

9. The computer-readable memory device of claim 7, where the queries are associated with images.

11. The method of claim 9, where selecting one of the answers for the at least one of the second queries further comprises:

selecting the one of the answers based on (1) a weight given to each of the plurality of users, where the weight is based on the score associated with a respective user of the plurality of users, and (2) based on answers provided by a majority of the plurality of users.

10. A method performed by one or more processors of one or more computer devices, comprising:

retrieving, by at least one of the one or more processors, a set of first queries from a query log that comprises search queries previously submitted by one or more users, where each of the first queries includes a question for which an answer has been previously identified;

retrieving, by at least one of the one or more processors, a set of second queries from the query log, where each of the second queries includes a question that does not have an associated answer in the query log;

posing, by at least one of the one or more processors, a plurality of questions to a plurality of users, where the plurality of questions includes a plurality of the first queries and at least one of the second queries;

receiving, by at least one of the one or more processors, answers to the plurality of questions from the plurality of users;

determining, by at least one of the one or more processors, a score reflecting an accuracy of each of the plurality of users based on the answers, provided by each of the plurality of users, that correspond to the first queries and that match the answers previously identified for the first queries; and

selecting, by at least one of the one or more processors, one of the answers for the at least one of the second queries based on the answers provided by the plurality of users and the scores for the plurality of users.

12. The method of claim 10, further comprising:

verifying that a user, of the plurality of users, is human based on the answers that correspond to the first queries.

13. The method of claim 10, further comprising:

generating the answers to the first queries based on the answers that correspond to the second queries.

14. A system comprising:

at least one processor to;

elicit user participation in a game by posing queries from a query log to a user, where the query log comprises search queries previously submitted by one or more users;

gather first information resulting from the user participation in the game;

gather second information, which is different than the first information, resulting from the user participation in the game;

assign a score to the user based on the gathered first information;

assign a weight to the gathered second information based on the assigned score; and

answer queries subsequently posed to the one or more users based on the weighted, gathered, second information.

15. The system of claim 14, where the user participates in the game via a network connected to the one or more server devices.

16. The system of claim 14, where, when the at least one processor is to elicit the user participation in the game, the at least one processor further is to:

retrieve, from the query log, a first set of queries and retrieve answers associated with the first set of queries, retrieve, from the query log, a second set of queries that do not have associated answers, and pose, to the user, at least one query from the first set and at least one query from the second set.

17. The system of claim 16, where, when the at least one processor is to answer the queries, subsequently posed to the one or more users, the at least one processor further is to:

assign, based on the score assigned to the user, the weight to the user'"'"'s answer to the at least one query from the second set;

receive the at least one query from the second set, as a query from another user;

select, based on the weight, the user'"'"'s answer to the at least one query from the second set; and

provide the selected answer to the other user in response to receiving the at least one query from the second set, as a query from the other user.

18. The system of claim 16, where, when the at least one processor is to assign the score to the user, the at least one processor further is to:

assign, to the user, the score based on whether the received answer to the at least one query from the first set matched a retrieved answer associated with the at least one query from the first set.

19. The system of claim 14, where, when the at least one processor is to gather the first information, the at least one processor further is to:

receive, from the user, answers, to the at least one query from the first set; and

when the at least one processor is to gather the second information, the at least one processor further is to;

receive, from the user, answers to the at least one query from the second set.

20. A system comprising:

at least one processor to;

retrieve a first set of queries from a query log that comprises search queries previously submitted by one or more users;

retrieve answers associated with the first set of queries from the query log;

retrieve a second set of queries from the query log that do not have associated answers in the query log;

pose, to multiple users, at least one query from the first set of queries and at least one query from the second set of queries;

receive answers to the at least one query from the first set and the at least one query from the second set;

aggregate the answers to the at least one query from the second set;

compare answers to the at least one query from the first set to a retrieved answer associated with the at least one query from the first set;

assign a score to each particular user, of the multiple users, based on a quantity of answers, received from the particular user, to queries from the first set of queries that match retrieved answers associated with the first set of queries; and

select an answer for the at least one query from the second set of queries based on the aggregated answers and scores assigned to the multiple users.

21. The system of claim 20, where, when the at least one processor is to select the answer, the at least one processor further is to:

select an answer that was received from at least a particular quantity of users.

22. The system of claim 20, where the queries are associated with images.

23. A system comprising:

at least one processor to;

retrieve a set of first queries from a query log that comprises search queries previously submitted by one or more users, where each of the first queries includes a question for which an answer has been previously identified;

retrieve a set of second queries from the query log, where each of the second queries includes a question that does not have an associated answer in the query log;

pose a plurality of questions to a plurality of users, where the plurality of questions includes a plurality of the first queries and at least one of the second queries;

receive answers to the plurality of questions from the plurality of users;

determine a score reflecting an accuracy of each of the plurality of users based on the answers, provided by each of the plurality of users, that correspond to the first queries and that match the answers previously identified for the first queries; and

select one of the answers for the at least one of the second queries based on the answers provided by the plurality of users and the scores for the plurality of users.

24. The system of claim 23, where, when the at least one processor is to select the one of the answers for the at least one of the second queries, the at least one processor further is to:

select the one of the answers based on (1) a weight given to each of the plurality of users, where the weight is based on the score associated with a respective user of the plurality of users, and (2) based on answers provided by a majority of the plurality of users.

25. The system of claim 23, where the at least one processor further is to:

verify that a user, of the plurality of users, is human based on the answers that correspond to the first queries.

26. The system of claim 23, where the at least one processor further is to:

generate the answers to the first queries based on the answers that correspond to the second queries.

27. A non-transitory computer program product comprising:

one or more programming instructions to;

elicit user participation in a game by posing queries from a query log to a user, where the query log comprises search queries previously submitted by one or more users;

gather first information resulting from the user participation in the game;

gather second information, which is different than the first information, resulting from the user participation in the game;

assign a score to the user based on the gathered first information;

assign a weight to the gathered second information based on the assigned score; and

answer queries subsequently posed to the one or more users based on the weighted, gathered, second information.

28. The computer program product of claim 27, where the user participates in the game via a network connected to the one or more server devices.

29. The computer program product of claim 27, where the one or more programming instructions to elicit the user participation in the game further comprises one or more programming instructions to:

retrieve, from the query log, a first set of queries and retrieve answers associated with the first set of queries,

retrieve, from the query log, a second set of queries that do not have associated answers, and

pose, to the user, at least one query from the first set and at least one query from the second set.

30. The computer program product of claim 29, where the one or more programming instructions to answer the queries, subsequently posed to the one or more users, further comprises one or more programming instructions to:

assign, based on the score assigned to the user, the weight to the user'"'"'s answer to the at least one query from the second set;

receive the at least one query from the second set, as a query from another user;

select, based on the weight, the user'"'"'s answer to the at least one query from the second set; and

provide the selected answer to the other user in response to receiving the at least one query from the second set, as a query from the other user.

31. The computer program product of claim 29, where the one or more programming instructions to assign the score to the user further comprises one or more programming instructions to:

assign, to the user, the score based on whether the received answer to the at least one query from the first set matched a retrieved answer associated with the at least one query from the first set.

32. The computer program product of claim 27, where the one or more programming instructions to gather the first information further comprises one or more programming instructions to:

receive, from the user, answers to the at least one query from the first set; and

the one or more programming instructions to gather the second information further comprises one or more programming instructions to;

receive, from the user, answers to the at least one query from the second set.

33. A non-transitory computer program product comprising:

one or more programming instructions to;

retrieve a first set of queries from a query log that comprises search queries previously submitted by one or more users;

retrieve answers associated with the first set of queries from the query log;

retrieve a second set of queries from the query log that do not have associated answers in the query log;

pose, to multiple users, at least one query from the first set of queries and at least one query from the second set of queries;

receive answers to the at least one query from the first set and the at least one query from the second set;

aggregate the answers to the at least one query from the second set;

compare answers to the at least one query from the first set to a retrieved answer associated with the at least one query from the first set;

assign a score to each particular user, of the multiple users, based on a quantity of answers, received from the particular user, to queries from the first set of queries that match retrieved answers associated with the first set of queries; and

select an answer for the at least one query from the second set of queries based on the aggregated answers and scores assigned to the multiple users.

34. The computer program product of claim 33, where the one or more programming instructions to select the answer further comprises one or more programming instructions to:

select an answer that was received from at least a particular quantity of users.

35. The computer program product of claim 33, where the queries are associated with images.

36. A non-transitory computer program product comprising:

one or more programming instructions to;

retrieve a set of first queries from a query log that comprises search queries previously submitted by one or more users, where each of the first queries includes a question for which an answer has been previously identified;

retrieve a set of second queries from the query log, where each of the second queries includes a question that does not have an associated answer in the query log;

pose a plurality of questions to a plurality of users, where the plurality of questions includes a plurality of the first queries and at least one of the second queries;

receive answers to the plurality of questions from the plurality of users;

determine a score reflecting an accuracy of each of the plurality of users based on the answers, provided by each of the plurality of users, that correspond to the first queries and that match the answers previously identified for the first queries; and

select one of the answers for the at least one of the second queries based on the answers provided by the plurality of users and the scores for the plurality of users.

37. The computer program product of claim 36, where the one or more programming instructions to select the one of the answers for the at least one of the second queries further comprises one or more programming instructions to:

select the one of the answers based on (1) a weight given to each of the plurality of users, where the weight is based on the score associated with a respective user of the plurality of users, and (2) based on answers provided by a majority of the plurality of users.

38. The computer program product of claim 36, further comprising one or more programming instructions to:

verify that a user, of the plurality of users, is human based on the answers that correspond to the first queries.

39. The computer program product of claim 36, further comprising one or more programming instructions to:

generate the answers to the first queries based on the answers that correspond to the second queries.