Automatic identification of unreliable user ratings

• US 7,519,562 B1
• Filed: 03/31/2005
• Issued: 04/14/2009
• Est. Priority Date: 03/31/2005
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gloy, Nikolas C., Ghosh, Arijit, Vander Mey, Christopher D.
• Primary Examiner(s): Rosen; Nicholas D
• Application Number:
  US11/096,699
• Time in Patent Office: 1,475 Days
• Field of Search: 705/12, 705/26, 705/27, 705/500
• US Class Current: 705/500
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 99/00 Subject matter not provided for in other groups of this subclass

Abstract

Techniques are described for enhancing the reliability of information received from users in various ways, such as by analyzing votes or other ratings supplied by a user in order to identify fraudulent and other unreliable ratings. Such analysis may be based at least in part on prior ratings submitted by the user, such as based on one or more patterns indicating that the user has a bias for or against one or more of various aspects. Unreliable ratings can then be excluded from use in various ways. This abstract is provided to comply with the rules requiring it, and is submitted with the intention that it not limit the scope of the claims.

Claims

1. A method for a computing system to automatically detect unreliability of a rating from a user based at least in part on a pattern identified using prior ratings of the user, the method comprising:

receiving an indication of a rating that reflects an evaluation by a user related to a target item that is available for acquisition, the indicated rating being a rating by the user that is of a review of the target item provided by another user;

obtaining one or more indications of multiple other ratings by the user that reflect evaluations related to other items;

automatically analyzing by the computing system the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating, wherein the computing system is configured with instructions to perform the automatic analyzing of the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating; and

if the unreliability of the indicated rating is identified based on the one or more patterns, providing an indication that the indicated rating is unreliable so as to exclude use of the indicated rating as part of an aggregate rating related to the target item, and otherwise including the use of the indicated rating as part of the aggregate rating related to the target item.

2. The method of claim 1 wherein the one or more patterns include a pattern indicating the user has a bias for and/or against a source associated with the target item.

3. The method of claim 1 wherein the indicated rating reflects an evaluation by the user of information related to the target item that is provided by another user, and wherein the one or more patterns include a pattern indicating the user has a bias for and/or against the another user.

4. The method of claim 1 including obtaining one or more indications related to other activities of the user, and wherein the automatic analyzing further includes analyzing the other activities in order to determine whether the user is using a shell account.

5. The method of claim 1 wherein the one or more patterns include a pattern indicating a user is generating blocks of ratings.

6. The method of claim 1 wherein the one or more patterns include a pattern indicating the user has a consistent positive or negative bias relative to ratings of other users.

7. The method of claim 1 further comprising:

obtaining one or more indications of multiple other ratings by other users that reflect evaluations related to the target item;

automatically analyzing the indicated rating and the multiple other ratings by the other users to identify an inconsistency of the indicated rating that is sufficient to exceed an inconsistency threshold; and

providing an indication that the indicated rating is inconsistent so as to exclude use of the indicated rating for the target item.

8. The method of claim 1 further comprising:

obtaining one or more indications of multiple other ratings by other users that reflect evaluations related to the other items;

automatically analyzing the other ratings by the user and the other ratings by the other users to identify an inconsistency of the other ratings by the user that is sufficient to exceed an inconsistency threshold; and

providing an indication that the ratings of the user are inconsistent with the ratings of other users so as to exclude use of the indicated rating for the target item.

9. The method of claim 1 further comprising, if the unreliability of the indicated rating is identified based on the one or more patterns, automatically determining whether to suppress the user as being unreliable based at least in part on whether one or more of the other ratings by the user are identified as unreliable.

10. The method of claim 9 including, if it is determined to suppress the user as being unreliable, automatically excluding use of at least some other ratings previously provided by the user and/or automatically excluding use of at least some additional ratings if provided by the user in the future.

11. The method of claim 9 including, if it is determined to suppress the user as being unreliable, automatically determining at a later time whether to remove the suppression of the user based at least in part on activities of the user after the suppression.

12. The method of claim 1 wherein the indicated rating related to the target item is a rating of the target item.

13. The method of claim 1 wherein the evaluation of the user related to the target item reflects a subjective opinion of the user.

14. The method of claim 1 wherein the indicated rating is a boolean vote.

15. The method of claim 1 wherein the indicated rating is a numerical rating.

16. The method of claim 15 wherein the numerical rating is a rating of the target item.

17. The method of claim 1 wherein the indicated rating includes textual comments provided by the user.

18. The method of claim 1 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and wherein the method further comprises, if the unreliability of the indicated rating is identified based on the one or more patterns, automatically analyzing each of at least some other ratings previously provided by the user to determine whether an assessed degree of unreliability of that other rating exceeds a second unreliability threshold that is lower than the first unreliability threshold, and otherwise not performing the automatic analyzing of the at least some other ratings previously provided by the user.

19. The method of claim 1 wherein the indicated rating and the other ratings by the user are provided in an anonymous manner such that an identity of the user is unknown.

20. The method of claim 19 including further assessing the unreliability of the indicated rating by comparing the indicated rating to ratings of other users whose identities are unknown.

21. The method of claim 1 including further assessing the unreliability of the indicated rating based at least in part on additional information obtained about the user.

22. The method of claim 1 wherein the one or more patterns include a pattern indicating that at least some of the other ratings are provided by an automated program on behalf of the user.

23. The method of claim 1 including automatically identifying the indicated rating as being unreliable if prior activities of the user do not satisfy specified criteria.

24. The method of claim 1 including automatically identifying whether other ratings indicated as being from other users were provided by the user, and identifying the indicated rating as being unreliable if one or more of the other ratings indicated as being from other users were identified as being provided by the user.

25. The method of claim 1 including automatically analyzing ratings from multiple users related to each of multiple items to determine whether at least some of the multiple users are collaborating together to provide consistent ratings, and when it is determined that at least some of the multiple users are collaborating together to provide consistent ratings, identifying those consistent ratings as being unreliable.

26. The method of claim 1 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and including when the assessed degree of unreliability of the indicated rating does not exceed the first unreliability threshold but does exceed a second lower unreliability threshold, providing an indication that the indicated rating is suspect so as to restrict use of the indicated rating.

27. The method of claim 26 wherein the restricted use of the indicated rating includes using the indicated rating as part of the aggregate rating in such a manner that the indicated rating is given a lower weight than other ratings that are not suspect.

28. The method of claim 26 wherein the restricted use of the indicated rating includes temporarily excluding use of the indicated rating as part of the aggregate rating until additional analysis of the indicated rating is completed.

29. The method of claim 1 wherein the providing of the indication that the indicated rating is unreliable so as to exclude use of the indicated rating is performed in such a manner as to obfuscate the excluding from the user.

30. The method of claim 1 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and including automatically revising the first unreliability threshold based at least in part on an automated analysis of ratings identified as unreliable so as to improve further identifications of unreliable ratings.

31. The method of claim 1 wherein the target item is a product and/or a service offered by an online retailer.

32. The method of claim 1 wherein identifying that the indicated rating is unreliable reflects an assessment of intentional fraudulent activity by the user.

33. The method of claim 1 including, if the unreliability of the indicated rating is not identified, presenting information regarding the target item to other users that is based at least in part on the indicated rating.

34. The method of claim 33 including, if the unreliability of the indicated rating is not identified, automatically generating the aggregate rating based on the indicated rating and on ratings from other users, and wherein the presented information includes an indication of the aggregate rating.

35. The method of claim 1 wherein the obtaining of the indications of the multiple other ratings by the user includes obtaining information from a second computing system, and wherein the indication that the indicated rating is unreliable is provided to the second computing system.

36. A method for a computing system to automatically detect unreliability of a rating from a user based at least in part on a pattern identified using prior ratings of the user, the method comprising:

receiving an indication of a rating that reflects an evaluation by a user related to a target item that is available for acquisition;

obtaining one or more indications of multiple other ratings by the user that reflect evaluations related to other items;

automatically analyzing by the computing system the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating, the one or more patterns including a pattern indicating a user is generating blocks of ratings, wherein the blocks of ratings by the user are indicated by one of the patterns as being prolific block rating by a new rater, and wherein the computing system is configured with instructions to perform the automatic analyzing of the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating, the one or more patterns including a pattern indicating a user is generating blocks of ratings; and

if the unreliability of the indicated rating is identified based on the one or more patterns, providing an indication that the indicated rating is unreliable so as to exclude use of the indicated rating as part of an aggregate rating related to the target item, and otherwise including the use of the indicated rating as part of the aggregate rating related to the target item.

37. The method of claim 36 wherein the one or more patterns further include at least one of a pattern indicating the user has a bias for and/or against a source associated with the target item, of a pattern indicating the user has a bias for and/or against another user who provided information to which the evaluation corresponds, and of a pattern indicating the user has a consistent positive or negative bias relative to ratings of other users.

38. The method of claim 36 wherein the one or more patterns further include a pattern indicating that at least some of the other ratings are provided by an automated program on behalf of the user.

39. The method of claim 36 further comprising obtaining one or more indications related to other activities of the user, and wherein the automatic analyzing further includes analyzing the other activities in order to determine whether the user is using a shell account.

40. The method of claim 36 further comprising, if the unreliability of the indicated rating is identified based on the one or more patterns, automatically determining to suppress the user as being unreliable based at least in part on one or more of the other ratings by the user being identified as unreliable, so as to restrict use of at least some of the ratings provided by the user.

41. The method of claim 36 further comprising automatically identifying whether other ratings indicated as being from other users were provided by the user, and identifying the indicated rating as being unreliable if one or more of the other ratings indicated as being from other users were identified as being provided by the user.

42. The method of claim 36 further comprising automatically analyzing ratings from multiple users related to each of multiple items to determine whether at least some of the multiple users are collaborating together to provide consistent ratings, and identifying consistent ratings as being unreliable if it is determined that at least some of the multiple users are collaborating together to provide those consistent ratings.

43. The method of claim 36 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and wherein the method further comprises, if the assessed degree of unreliability of the indicated rating does not exceed the first unreliability threshold but does exceed a second lower unreliability threshold, providing an indication that the indicated rating is suspect so as to restrict use of the indicated rating, the restricted use of the indicated rating including at least one of using the indicated rating as part of the aggregate rating in such a manner that the indicated rating is given a lower weight than other ratings that are not suspect, and of temporarily excluding use of the indicated rating as part of the aggregate rating until additional analysis of the indicated rating is completed.

44. The method of claim 36 further comprising, if the unreliability of the indicated rating is not identified, automatically generating the aggregate rating based on the indicated rating and on ratings from multiple other users, and presenting an indication of the generated aggregate rating to one or more users.

45. A method for a computing system to automatically detect unreliability of a rating from a user based at least in part on a pattern identified using prior ratings of the user, the method comprising:

receiving an indication of a rating that reflects an evaluation by a user related to a target item that is available for acquisition, the indicated rating being a numerical rating of a usefulness of a review of the target item by another user;

obtaining one or more indications of multiple other ratings by the user that reflect evaluations related to other items;

automatically analyzing by the computing system the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating, wherein the computing system is configured with instructions to perform the automatic analyzing of the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating; and

if the unreliability of the indicated rating is identified based on the one or more patterns, providing an indication that the indicated rating is unreliable so as to exclude use of the indicated rating as part of an aggregate rating related to the target item, and otherwise including the use of the indicated rating as part of the aggregate rating related to the target item.

46. The method of claim 45 wherein the one or more patterns include multiple of a pattern indicating the user has a bias for and/or against a source associated with the target item, of a pattern indicating the user has a bias for and/or against the another user who provided the review to which the evaluation corresponds, and of a pattern indicating the user has a consistent positive or negative bias relative to ratings of other users.

47. The method of claim 45 further comprising obtaining one or more indications related to other activities of the user, and wherein the automatic analyzing further includes analyzing the other activities in order to identify whether the indicated rating is unreliable.

48. The method of claim 45 further comprising:

obtaining one or more indications of multiple other ratings by other users that reflect evaluations of the usefulness of the review of the target item;

automatically analyzing the indicated rating and the multiple other ratings by the other users to identify an inconsistency of the indicated rating; and

providing an indication that the indicated rating is inconsistent so as to exclude use of the indicated rating for the target item.

49. The method of claim 45 further comprising, if the unreliability of the indicated rating is identified based on the one or more patterns, automatically determining whether to suppress the user as being unreliable so as to restrict use of at least some ratings provided by the user.

50. The method of claim 45 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and wherein the method further comprises, if the assessed degree of unreliability of the indicated rating does not exceed the first unreliability threshold but does exceed a second lower unreliability threshold, providing an indication that the indicated rating is suspect so as to restrict use of the indicated rating, the restricted use of the indicated rating including at least one of using the indicated rating as part of the aggregate rating in such a manner that the indicated rating is given a lower weight than other ratings that are not suspect, and of temporarily excluding use of the indicated rating as part of the aggregate rating until additional analysis of the indicated rating is completed.

51. The method of claim 45 further comprising, if the unreliability of the indicated rating is not identified, automatically generating the aggregate rating based on the indicated rating and on ratings from multiple other users, the generated aggregate rating being an aggregate rating of the target item review from the another user, and selecting the target item review to provide to other users based at least in part on the generated aggregate rating.

52. A method for a computing system to automatically detect unreliability of a rating from a user based at least in part on a pattern identified using prior ratings of the user, the method comprising:

receiving an indication of a rating that reflects an evaluation by a user related to a target item that is available for acquisition, and wherein the computing system is operated on behalf of an online retailer who offers the target item;

obtaining one or more indications of multiple other ratings by the user that reflect evaluations related to other items;

automatically analyzing by the computing system the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating, the automatic analyzing including automatically identifying the indicated rating as being unreliable if prior activities of the user do not satisfy specified criteria, wherein the specified criteria include a specified number of one or more prior purchases by the user from the online retailer, and wherein the computing system is configured with instructions to perform the automatic analyzing of the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating, the automatic analyzing including automatically identifying the indicated rating as being unreliable if prior activities of the user do not satisfy specified criteria; and

if the unreliability of the indicated rating is identified based on the one or more patterns, providing an indication that the indicated rating is unreliable so as to exclude use of the indicated rating as part of an aggregate rating related to the target item, and otherwise including the use of an indicated rating as part of the aggregate rating related to the target item.

53. The method of claim 52 wherein the one or more patterns include a pattern indicating the user has a bias for and/or against a source associated with the target item, of a pattern indicating the user has a bias for and/or against another user who provided information to which the evaluation corresponds, and of a pattern indicating the user has a consistent positive or negative bias relative to ratings of other users.

54. The method of claim 52 further comprising obtaining one or more indications related to other activities of the user, and wherein the automatic analyzing further includes analyzing the other activities in order to identify whether the indicated rating is unreliable.

55. The method of claim 52 further comprising:

obtaining one or more indications of multiple other ratings by other users that reflect evaluations related to the target item;

automatically analyzing the indicated rating and the multiple other ratings by the other users to identify an inconsistency of the indicated rating; and

providing an indication that the indicated rating is inconsistent so as to exclude use of the indicated rating for the target item.

56. The method of claim 52 further comprising, if the unreliability of the indicated rating is identified based on the one or more patterns, automatically determining whether to suppress the user as being unreliable so as to restrict use of at least some ratings provided by the user.

57. The method of claim 52 further comprising automatically identifying whether other ratings indicated as being from other users were provided by the user, and identifying the indicated rating as being unreliable if one or more of the other ratings indicated as being from other users were identified as being provided by the user.

58. The method of claim 52 further comprising automatically analyzing ratings from multiple users related to each of multiple items to determine whether at least some of the multiple users are collaborating together to provide consistent ratings, and identifying consistent ratings as being unreliable if it is determined that at least some of the multiple users are collaborating together to provide those consistent ratings.

59. The method of claim 52 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and wherein the method further comprises, if the assessed degree of unreliability of the indicated rating does not exceed the first unreliability threshold but does exceed a second lower unreliability threshold, providing an indication that the indicated rating is suspect so as to restrict use of the indicated rating, the restricted use of the indicated rating including at least one of using the indicated rating as part of the aggregate rating in such a manner that the indicated rating is given a lower weight than other ratings that are not suspect, and of temporarily excluding use of the indicated rating as part of the aggregate rating until additional analysis of the indicated rating is completed.

60. The method of claim 52 further comprising, if the unreliability of the indicated rating is not identified, automatically generating the aggregate rating based on the indicated rating and on ratings from multiple other users, and presenting an indication of the generated aggregate rating to one or more customers of the online retailer to facilitate potential purchases by the one or more customers of the target item.

61. A method for a computing system of an online retailer to automatically detect fraudulent votes received from customers who are evaluating item reviews supplied by other customers, the method comprising:

presenting to a first customer a first item review for an item available from the merchant, the first item review provided by a distinct second customer;

receiving a first vote from the first customer that reflects an evaluation by the first customer of a usefulness of the first item review;

automatically assessing by the computing system whether the first vote is fraudulent by,

retrieving information regarding a vote history of the first customer that includes multiple prior votes of the first customer, each prior vote reflecting an evaluation by the first customer of a usefulness of an item review distinct from the first item review that was provided by a customer distinct from the first customer;

retrieving information regarding multiple prior votes of other customers that reflect evaluations by those other customers of usefulness of the first item review;

evaluating unreliability of the first vote by comparing the first vote and the vote history of the first customer to multiple predefined patterns associated with unreliable vote histories and by comparing the first vote to the information regarding the prior votes of the other customers for the first item review; and

if the evaluated unreliability of the first vote exceeds a predefined unreliability threshold, identifying the first vote as fraudulent, and otherwise not identifying the first vote as fraudulent,

wherein the computing system is configured with instructions to perform the automatic assessing by retrieving information regarding a vote history of the first customer that includes multiple prior votes of the first customer, each prior vote reflecting an evaluation by the first customer of a usefulness of an item review distinct from the first item review that was provided by a customer distinct from the first customer, retrieving information regarding multiple prior votes of other customers that reflect evaluations by those other customers of usefulness of the first item review, and evaluating unreliability of the first vote by comparing the first vote and the vote history of the first customer to multiple predefined patterns associated with unreliable vote histories and by comparing the first vote to the information regarding the prior votes of the other customers for the first item review; and

when the first vote is identified as fraudulent, automatically excluding by the computing system the first vote from a group of votes used to rate the usefulness of the first item review,

wherein the computing system is further configured with instructions to perform the automatic excluding of the first votes when the first vote is identified as fraudulent.

62. The method of claim 61 further comprising automatically rating the usefulness of the first item review based on the group of votes, and automatically determining whether to present the first item review to another customer based at least in part on the usefulness rating.

63. The method of claim 62 wherein the first vote is automatically included in the group of votes used to rate the usefulness of the first item review when the first vote is not identified as fraudulent.

64. The method of claim 61 wherein the multiple predefined patterns associated with unreliable vote histories include at least two of:

a pattern indicating the first customer has a bias against or in favor of a source associated with the available item;

a pattern indicating the first customer has a bias against or in favor of the second customer;

a pattern indicating the first customer is using a shell account that is used only for voting; and

a pattern indicating the first customer is engaging in block voting that includes multiple instances of casting numerous votes over small periods of time and of not casting any votes over other longer periods of time.

65. The method of claim 61 wherein the evaluating of the unreliability of the first vote by comparing the first vote and the vote history of the first customer to multiple predefined patterns associated with unreliable vote histories includes, for each match that is detected to a pattern in the multiple predefined patterns, incrementing an unreliability score for the first vote by an amount that is based at least in part on the matched pattern.

66. The method of claim 61 including, for each of one or more of the distinct item reviews evaluated by the first customer in the multiple prior votes of the first customer, retrieving information regarding multiple prior votes of other customers that reflect evaluations by those other customers of usefulness of the distinct item review, and wherein the evaluating of the unreliability of the first vote further comprises comparing the one or more prior votes of the first customer for those distinct item reviews to corresponding votes by other customers for those distinct item reviews.

67. The method of claim 61 wherein the retrieved information regarding the vote history of the first customer includes multiple other prior votes by the first customer that each reflect evaluations by the first customer of other than a usefulness of an item review, and including automatically evaluating the other votes by comparing the other votes to one or more other predefined patterns.

68. The method of claim 61 further comprising, if the first vote is identified as fraudulent, automatically evaluating one or more prior votes of the first customer to determine whether those prior votes are fraudulent, and if the first vote is identified as fraudulent and if one or more other prior votes from the first customer are identified as fraudulent, automatically determining whether to exclude all prior votes from the first customer and/or all future votes from the first customer based on an assessed unreliability of the first customer.

69. The method of claim 61 wherein the retrieved information regarding the multiple prior votes of the other customers includes information summarizing those prior votes.

70. The method of claim 61 wherein an identification that the first vote is fraudulent reflects an assessment that the first vote does not accurately represent the first customer'"'"'s evaluation of the first item review.

71. A computing system configured to automatically detect unreliability of a rating from a user based at least in part on a pattern identified using prior ratings of the user, comprising:

one or more memories;

a reliability assessor system configured to include software instructions stored in at least one of the one or more memories and that when executed by a processor cause the computing system to receive an indication of a rating that reflects an evaluation by a user related to a target item that is available for acquisition, the indicated rating being a rating by the user that is of a review of the target item provided by another user, to obtain one or more indications of multiple other ratings by the user that reflect evaluations related to other items, and to automatically analyze the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating;

and a content manager system configured to include software instructions stored in at least one of the one or more memories and that when executed by a processor cause the computing system to manage the indicated rating by, if the unreliability of the indicated rating is identified based on the one or more patterns, excluding use of the indicated rating as part of an aggregate rating related to the target item, and otherwise including the use of the indicated rating as part of the aggregate rating related to the target item.

72. The computing system of claim 71, wherein the reliability assessor system includes a user assessor component configured to generate a usefulness assessment of the user based on an analysis of the indicated rating and of the multiple other ratings by the user, and wherein the content manager system is further configured to manage ratings provided by the user based at least in part on the generated usefulness assessment of the user.

73. The computing system of claim 71 wherein the reliability assessor system is configured to use multiple patterns to assess whether the indicated rating is unreliable by including multiple assessor components that are each configured to assess unreliability of the indicated rating corresponding to one of the multiple patterns, wherein the reliability assessor component further includes a rating assessment aggregator component to aggregate unreliability assessments from the multiple assessor components, and wherein the reliability assessor system is further configured to receive indications of one or more new patterns of reliability and to use the new patterns of reliability as part of further generating of aggregate reliability assessments for ratings.

74. The computing system of claim 71 wherein the content manager system is further configured to generate the aggregate rating related to the target item, the generated aggregate rating being an aggregate rating of the target item review provided by the another user, and to select the provided target item review to provide to other users based at least in part on the generated aggregate rating.

75. The computing system of claim 71 wherein the reliability assessor system consists of a means for receiving an indication of a rating that reflects an evaluation by a user related to a target item that is available for acquisition, the indicated rating being a rating by the user that is of a review of the target item provided by another user, for obtaining one or more indications of multiple other ratings by the user that reflect evaluations related to other items, and for automatically analyzing the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating.

76. The computing system of claim 71 wherein the one or more patterns include at least one of a pattern indicating the user has a bias for and/or against a source associated with the target item, of a pattern indicating the user has a bias for and/or against the another user who provided the target item review, and of a pattern indicating the user has a consistent positive or negative bias relative to ratings of other users.

77. The computing system of claim 76 wherein the one or more patterns further include a pattern indicating that at least some of the other ratings are provided by an automated program on behalf of the user.

78. The computing system of claim 71 wherein the reliability assessor system is further configured to automatically analyze obtained information related to other activities of the user in order to identify whether the indicated rating is likely to be unreliable.

79. The computing system of claim 71 wherein the reliability assessor system is further configured to:

obtain one or more indications of multiple other ratings by other users that reflect evaluations of the target item review;

automatically analyze the indicated rating and the multiple other ratings by the other users to identify an inconsistency of the indicated rating with respect to the multiple other ratings; and

provide an indication that the indicated rating is inconsistent so as to exclude use of the indicated rating for the target item.

80. The computing system of claim 71 wherein the reliability assessor system is further configured to:

obtain one or more indications of multiple other ratings by other users that reflect evaluations related to the other items;

automatically analyze the other ratings by the user and the other ratings by the other users to identify an inconsistency of the other ratings by the user; and

provide an indication that the other ratings of the user are inconsistent so as to exclude use of the indicated rating for the target item.

81. The computing system of claim 71 wherein the reliability assessor system includes a user assessment aggregator component that is configured to, if the unreliability of the indicated rating is identified based on the one or more patterns, automatically determine whether to suppress the user as being unreliable so as to restrict use of at least some ratings provided by the user.

82. The computing system of claim 71 wherein the reliability assessor system is further configured to automatically identify whether other ratings indicated as being from other users were provided by the user, and to identify the indicated rating as being unreliable if one or more of the other ratings indicated as being from other users are identified as being provided by the user.

83. The computing system of claim 71 wherein the reliability assessor system is further configured to automatically analyze ratings from multiple users related to each of multiple items to determine whether at least some of the multiple users are collaborating together to provide consistent ratings, and to identify consistent ratings as being unreliable if it is determined that at least some of the multiple users are collaborating together to provide those consistent ratings.

84. The computing system of claim 71 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and wherein the reliability assessor system is further configured to, if the assessed degree of unreliability of the indicated rating does not exceed the first unreliability threshold but does exceed a second lower unreliability threshold, indicate that the indicated rating is suspect so as to restrict use of the indicated rating, the restricted use of the indicated rating including at least one of using the indicated rating as part of the aggregate rating in such a manner that the indicated rating is given a lower weight than other ratings that are not suspect, and of temporarily excluding use of the indicated rating as part of the aggregate rating until additional analysis of the indicated rating is completed.

85. The computing system of claim 71 wherein the one or more patterns include a pattern indicating the user is generating prolific blocks of ratings by a new rater, and wherein the reliability assessor system is further configured to identify the indicated rating as being unreliable based on the generating of the prolific blocks of ratings.

86. The computing system of claim 71 wherein the indicated rating is a numerical rating of usefulness of the target item review.

87. The computing system of claim 71 wherein the computing system is operated on behalf of an online retailer who offers the target item, and wherein the automatic analyzing includes automatically identifying the indicated rating as being unreliable if prior activities of the user related to shopping interactions by the user with the online retailer do not satisfy specified criteria.

88. A computer-readable storage medium whose contents are instructions that when executed cause a computing device to automatically detect unreliability of a rating from a user based at least in part on a pattern using prior ratings of the user, by performing a method comprising:

receiving an indication of a rating that reflects an evaluation by a user related to a target item that is available for acquisition, the indicated rating being a rating by the user that is of a review of the target item provided by another user;

obtaining one or more indications of multiple other ratings by the user that reflect evaluations related to other items;

automatically analyzing the indicated rating and the other ratings to determine whether one or more patterns in the indicated rating and other ratings identify an unreliability of the indicated rating; and

if the unreliability of the indicated rating is identified based on the one or more patterns, indicating that the indicated rating is unreliable so as to exclude use of the indicated rating as part of an aggregate rating related to the target item, and otherwise including the use of the indicated rating as part of the aggregate rating related to the target item.

89. The computer-readable storage medium of claim 88 wherein the one or more patterns include at least one of a pattern indicating the user has a bias for and/or against a source associated with the target item, of a pattern indicating the user has a bias for and/or against the another user who provided the target item review, and of a pattern indicating the user has a consistent positive or negative bias relative to ratings of other users.

90. The computer-readable storage medium of claim 88 wherein the one or more patterns include a pattern indicating that at least some of the other ratings are provided by an automated program on behalf of the user.

91. The computer-readable storage medium of claim 88 wherein the method further comprises obtaining one or more indications related to other activities of the user, and wherein the automatic analyzing further includes analyzing the other activities in order to identify whether the indicated rating is unreliable.

92. The computer-readable storage medium of claim 88 wherein the method further comprises:

obtaining one or more indications of multiple other ratings by other users that reflect evaluations of the target item review;

automatically analyzing the indicated rating and the multiple other ratings by the other users to identify an inconsistency of the indicated rating with respect to the multiple other ratings; and

providing an indication that the indicated rating is inconsistent so as to exclude use of the indicated rating for the target item.

93. The computer-readable storage medium of claim 88 wherein the method further comprises:

obtaining one or more indications of multiple other ratings by other users that reflect evaluations related to the other items;

automatically analyzing the other ratings by the user and the other ratings by the other users to identify an inconsistency of the other ratings by the user; and

providing an indication that the other ratings of the user are inconsistent so as to exclude use of the indicated rating for the target item.

94. The computer-readable storage medium of claim 88 wherein the method further comprises, if the unreliability of the indicated rating is identified based on the one or more patterns, automatically determining whether to suppress the user as being unreliable so as to restrict use of at least some ratings provided by the user.

95. The computer-readable storage medium of claim 88 wherein the method further comprises automatically identifying whether other ratings indicated as being from other users were provided by the user, and identifying the indicated rating as being unreliable if one or more of the other ratings indicated as being from other users are identified as being provided by the user.

96. The computer-readable storage medium of claim 88 wherein the method further comprises automatically analyzing ratings from multiple users related to each of multiple items to determine whether at least some of the multiple users are collaborating together to provide consistent ratings, and identifying consistent ratings as being unreliable if it is determined that at least some of the multiple users are collaborating together to provide those consistent ratings.

97. The computer-readable storage medium of claim 88 wherein the identifying of the indicated rating as unreliable includes determining that an assessed degree of unreliability of the indicated rating exceeds a first unreliability threshold, and wherein the method further comprises, if the assessed degree of unreliability of the indicated rating does not exceed the first unreliability threshold but does exceed a second lower unreliability threshold, providing an indication that the indicated rating is suspect so as to restrict use of the indicated rating, the restricted use of the indicated rating including at least one of using the indicated rating as part of the aggregate rating in such a manner that the indicated rating is given a lower weight than other ratings that are not suspect, and of temporarily excluding use of the indicated rating as part of the aggregate rating until additional analysis of the indicated rating is completed.

98. The computer-readable storage medium of claim 88 wherein the method further comprises, if the unreliability of the indicated rating is not identified, automatically generating the aggregate rating related to the target item based on the indicated rating and on ratings from other users, the generated aggregate rating being an aggregate rating of the target item review provided by the another user, and selecting the provided target item review to provide to other users based at least in part on the generated aggregate rating.

99. The computer-readable storage medium of claim 88 wherein the one or more patterns include a pattern indicating the user is generating prolific blocks of ratings by a new rater, and wherein the method further comprises identifying the indicated rating as being unreliable based on the generating of the prolific blocks of ratings.

100. The computer-readable storage medium of claim 88 wherein the indicated rating is a numerical rating of usefulness of the target item review.

101. The computer-readable storage medium of claim 88 wherein the computing device is operated on behalf of an online retailer who offers the target item, and wherein the automatic analyzing includes automatically identifying the indicated rating as being unreliable if prior activities of the user related to shopping interactions by the user with the online retailer do not satisfy specified criteria.

102. The computer-readable storage medium of claim 88 wherein the contents include one or more data structures for use in analyzing reliability of ratings, the data structure comprising multiple entries that each correspond to a defined pattern of unreliability of ratings and that each contain information for use in analyzing ratings in order to identify whether that pattern of unreliability is present in those ratings.