Systems and methods for assignment of human reviewers using probabilistic prioritization

• US 8,214,373 B1
• Filed: 02/18/2011
• Issued: 07/03/2012
• Est. Priority Date: 02/18/2011
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bakir, Gökhan, Kulshreshtha, Ashutosh
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US13/030,389
• Time in Patent Office: 501 Days
• Field of Search: 707/748, 707/802
• US Class Current: 707/748
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The present application discloses systems and methods for using probabilistic prioritization to assign human reviewers to review data stored in or indexed by an information system. Some embodiments include accessing an index of data items, where individual data items have a corresponding probability f of having a problem, a cost to review the data item, a penalty if a problem associated with the data item is not remedied, and a gain if a problem associated with the data item is remedied; identifying a subset of data items having a corresponding f that is greater than or equal to a decision threshold based on the data item'"'"'s corresponding cost, penalty, and gain; and ranking at least a portion of the subset of data items based at least in part on their corresponding cost, f, and gain.

Claims

1. A method comprising:

accessing an index of data items stored on tangible, non-transitory computer readable media, wherein data items have a corresponding probability (f_i) of having a problem, a cost to review the data item (cost_i), a penalty if a problem in the data item is not remedied (penalty_i), and a gain if a problem in the data item is remedied (gain_i);

identifying a subset of data items of the index of data items, wherein the subset comprises at least one data item with a corresponding f_i that is greater than or equal to a decision threshold;

ranking at least a portion of the subset of data items based at least in part on the data items'"'"' corresponding cost_i, f_i, and gain_i; and

storing the ranked portion of the subset of data items in a queue on the tangible, non-transitory computer readable media.

2. The method of claim 1, wherein an individual data item'"'"'s decision threshold is based on the individual data item'"'"'s corresponding cost_i, penalty_i, and gain_i.

3. The method of claim 2, wherein the decision threshold is equal to cost_i/(penalty_i+gain_i).

4. The method of claim 1, wherein ranking at least a portion of the subset of data items based at least in part on the data items'"'"' corresponding cost_i, f_i, and gain_i comprises:

assigning corresponding scores to individual data items in the at least a portion of the subset of data items, wherein the corresponding score is equal to (cost_i)−

(f_i)×

(gain_i); and

ranking the at least a portion of the subset of data items based on the corresponding scores of the data items in the subset.

5. The method of claim 1, wherein a data item'"'"'s corresponding cost_i is based at least in part on an amount of time for a human reviewer to review the data item.

6. The method of claim 1, wherein at least one data item has a corresponding importance score, prominence score, and corresponding likely problem severity score, and wherein the at least one data item'"'"'s corresponding penalty_i is based on at least one of the data item'"'"'s importance score, prominence score, and likely problem severity score, and wherein the at least one data item'"'"'s corresponding gain_i is based on at least one of the individual data item'"'"'s importance score and prominence score.

7. The method of claim 1, wherein a data item'"'"'s corresponding gain_i is based at least in part on expected revenue associated with the data item.

8. The method of claim 1, wherein a data item'"'"'s corresponding f_i is based on one or more of an indication received from a computing device configured to assign a corresponding f_i to the data item, an indication received from an information system user, and/or an indication received from an assigned human reviewer.

9. The method of claim 1, further comprising updating a first data item'"'"'s corresponding f_i based on a change in a reference data item'"'"'s corresponding f_i.

10. The method of claim 9, wherein updating the first data item'"'"'s corresponding f_i based on a change in the reference data item'"'"'s corresponding f_i comprises:

receiving one or more of (i) an indication of whether there is a suspected problem associated with the first data item, (ii) an indication of whether there is a suspected problem with the reference data item, and/or (iii) an indication from a human reviewer of whether the reference data item has a problem;

revising the first data item'"'"'s f_i based on at least one indication associated with the reference data item and at least one similarity between the reference data item and the first data item; and

re-ranking at least a portion of the data items in the subset according to the corresponding cost_i, f_i, and gain_i of the data items based on the revision to the first data item'"'"'s corresponding f_i.

11. The method of claim 10, wherein the one or more similarities between the reference data item and the first data item are based on similarities between metadata associated with the first data item and the reference data item.

12. A system comprising:

one or more computing devices configured to access an index of data items, wherein individual data items in the index have a corresponding probability (f_i) of having a problem, a cost to review the data item (cost_i), a penalty if a problem in the data item is not remedied (penalty_i), and a gain if a problem in the data item is remedied (gain_i),

wherein the one or more computing devices are further configured to identify a subset of data items of the index, wherein the subset comprises one or more data items having a corresponding f_i that is greater than or equal to a decision threshold, and

wherein the one or more computing devices are further configured to rank at least a portion of the subset of data items based at least in part on individual data items'"'"' corresponding cost_i, f_i, and gain_i, and store the ranked data items in a queue.

13. The system of claim 12, wherein the decision threshold for an individual data item is based on the individual data item'"'"'s corresponding cost_i, penalty_i, and gain_i.

14. The system of claim 13, wherein an individual data item'"'"'s corresponding decision threshold is equal to cost_i/(penalty_i+gain_i).

15. The system of claim 12, wherein the one or more computing devices are further configured to assign corresponding scores to individual data items in the at least a portion of the subset of data items, wherein an individual data item'"'"'s corresponding score is equal to (cost_i)−

(f_i)×

(gain_i), and wherein the one or more computing devices are further configured to rank the at least a portion of the subset of data items based on the data items'"'"' corresponding scores.

16. The system of claim 12, wherein a data item'"'"'s corresponding cost_i is based at least in part on an amount of time for a human reviewer to review the data item.

17. The system of claim 12, wherein at least some individual data items have a corresponding importance score, a corresponding prominence score, and corresponding likely problem severity score, wherein an individual data item'"'"'s corresponding penalty_i is based on at least one of the individual data item'"'"'s corresponding importance and likely problem severity scores, and wherein an individual data item'"'"'s corresponding gain_i is based on at least one of the individual data item'"'"'s corresponding importance, prominence, and likely problem severity scores.

18. The system of claim 12, wherein an individual data item'"'"'s corresponding gain_i is based at least in part on expected revenue associated with the data item.

19. The system of claim 12, wherein an individual data item'"'"'s corresponding f_i is based on one or more of an indication received from a computing device configured to assign a corresponding probability of having a problem to a data item, an indication received from an information system user, and/or an indication received from an assigned human reviewer.

20. The system of claim 12, wherein the one or more computing devices are further configured to update a first data item'"'"'s corresponding f_i based on a change in a reference data item'"'"'s corresponding f_i.

21. The system of claim 20, wherein the one or more computing devices are further configured to receive an indication of whether the reference data item contains an actual or suspected problem, to revise the first data item'"'"'s f_i based on one or more similarities between the reference data item and the first data item in response to receiving the indication, and to re-rank at least a portion of the data items in the subset according to their corresponding cost_i, f_i, and gain_i based on the revision to the first data item'"'"'s corresponding f_i.

22. The system of claim 21, wherein the one or more similarities between the reference data item and the first data item are based on similarities between metadata associated with the first data item and the reference data item.

23. An article of manufacture including a tangible computer-readable media having computer-readable instructions encoded thereon, the instructions comprising:

instructions for accessing an index of data items, wherein individual data items in the index have a corresponding probability (f_i) of having a problem, a cost to review the data item (cost_i), a penalty if a problem in the data item is not remedied (penalty_i), and a gain if a problem in the data item is remedied (gain_i);

instructions for identifying a subset of data items of the index of data items, wherein the subset comprises at least one data item with a corresponding f_i that is greater than or equal to a decision threshold;

instructions for ranking at least a portion of the subset of data items based at least in part on individual data items'"'"' corresponding cost_i, f_i, and gain_i; and

instructions for storing the ranked portion of the subset of data items in a queue on the tangible, non-transitory computer readable media.

24. The article of manufacture of claim 23, wherein an individual data item'"'"'s decision threshold is based on the individual data item'"'"'s corresponding cost_i, penalty_i, and gain_i.

25. The article of manufacture of claim 24, wherein an individual data item'"'"'s corresponding decision threshold is equal to cost_i/(penalty_i+gain_i).

26. The article of manufacture of claim 23, wherein the instructions for ranking at least a portion of the subset of data items based at least in part on individual data items'"'"' corresponding cost_i, f_i, and gain_i comprises:

instructions for assigning corresponding scores to individual data items in the at least a portion of the subset of data items, wherein an individual data item'"'"'s corresponding score is equal to (cost_i)−

(f_i)×

(gain_i); and

instructions for ranking the at least a portion of the subset of data items based on the corresponding scores.

27. The article of manufacture of claim 23, wherein a data item'"'"'s corresponding cost_i is based at least in part on an amount of time for a human reviewer to review the data item.

28. The article of manufacture of claim 23, wherein at least some individual data items have a corresponding importance score, a prominence score, and a likely problem severity score, and wherein an individual data item'"'"'s corresponding penalty_i is based on at least one of the individual data item'"'"'s importance score and likely problem severity score, and wherein an individual data item'"'"'s corresponding gain_i is based on at least one of the individual data item'"'"'s importance score, prominence score, and likely problem severity score.

29. The article of manufacture of claim 23, wherein an individual data item'"'"'s corresponding gain_i is based at least in part on expected revenue associated with the data item.

30. The article of manufacture of claim 23, wherein an individual data item'"'"'s corresponding f_i is based on one or more of:

(i) an indication of a likely problem received from a computing device configured to assign a corresponding probability of having a problem to the data item;

(ii) an indication of a likely problem received from an information system user, and/or (iii) an indication from a human reviewer of whether a related data has been identified as having a problem.

31. The article of manufacture of claim 23, further comprising instructions for updating a first data item'"'"'s corresponding f_i based on a change in a reference data item'"'"'s corresponding f_i.

32. The article of manufacture of claim 31, wherein the instructions for updating the first data item'"'"'s corresponding f_i based on a change in the reference data item'"'"'s corresponding f_i comprises:

instructions for receiving one or more of (i) an indication of whether there is a suspected problem associated with the first data item, (ii) an indication of whether there is a suspected problem with the reference data item, and/or (iii) an indication from a human reviewer of whether the reference data item has a problem;

instructions for revising the first data item'"'"'s f_i based on at least one indication associated with the reference data item and at least one similarity between the reference data item and the first data item; and

instructions for re-ranking at least a portion of the data items in the subset according to the corresponding cost_i, f_i, and gain_i of the data items based on the revision to the first data item'"'"'s corresponding f_i.

33. The article of manufacture of claim 32, wherein the one or more similarities between the reference data item and the first data item are based on similarities between metadata associated with the first data item and the reference data item.

34. A system comprising:

means for accessing an index of data items, wherein individual data items in the index have a corresponding probability (f_i) of having a problem, a cost to review the data item (cost_i), a penalty if a problem in the data item is not remedied (penalty_i), and a gain if a problem in the data item is remedied (gain_i);

means for identifying a subset of data items of the index, wherein the subset comprises data items having a corresponding f_i that is greater than or equal to a decision threshold, wherein the decision threshold for a data item is based on the data item'"'"'s corresponding cost_i, penalty_i, and gain_i; and

means for ranking at least a portion of the subset of data items based at least in part on the corresponding cost_i, f_i, and gain_i of the data items.