Assessing user-supplied evaluations

• US 9,009,082 B1
• Filed: 06/30/2008
• Issued: 04/14/2015
• Est. Priority Date: 06/30/2008
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Marshall, Anne R., Robinson, James G., Rajyaguru, Sameer R.
• Primary Examiner(s): Zeender, Ryan
• Assistant Examiner(s): ORTIZ ROMAN, DENISSE Y
• Application Number:
  US12/165,398
• Time in Patent Office: 2,479 Days
• Field of Search: None
• US Class Current: 705/500
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 99/00 Subject matter not provided for in other groups of this subclass

Abstract

Techniques are described for assessing information supplied by users in various ways, such as to assess the reliability and/or other attributes of the user-supplied information. In at least some situations, the user-supplied information includes votes or other evaluations supplied by users related to items available from an online merchant, such as ratings of usefulness or other attributes of item reviews for the items or of other types of content pieces that are provided by other users. If user-supplied information is assessed as being sufficiently reliable and/or to have other desired attributes of interest, such as based on an automated analysis of the information, the user-supplied information may be used in various ways in various embodiments, such as to rate the quality or other attributes of the evaluated content pieces, and/or to rate quality or other attributes of the content-providing users who provide the content pieces.

Claims

1. A method for a computing system of an online merchant to assess reliability of evaluations supplied by users of the online merchant, the method comprising:

receiving multiple pieces of content that are created by multiple author users and that are supplied to the online merchant for use by customers of the online merchant, the author users being a subset of the customers of the online merchant and the pieces of content each being a customer-generated textual item review for one of multiple items available from the online merchant, each of the multiple author users creating one or more of the multiple item review content pieces;

assessing the multiple item review content pieces by,

receiving multiple evaluations of the multiple item review content pieces that are supplied by multiple evaluator users who are customers of the online merchant, each of the received evaluations being from one of the evaluator users for one of the item review content pieces and including a numerical rating of the item review content piece for each of one or more of multiple predefined rating dimensions, each rating dimension related to an aspect of the item review content piece such that a numerical rating for the rating dimension indicates an assessment by the evaluator user of a degree to which that aspect of the item review content piece is satisfied, the received evaluations including one or more evaluations for each of the multiple item review content pieces;

identifying one or more of the multiple evaluations that are unreliable by, for each combination of an evaluator user and an author user who created one or more item review content pieces evaluated by the evaluator user,

determining a subset of the received evaluations supplied by the evaluator user for the item review content pieces created by the author user; and

automatically assessing the evaluations of the determined subset to identify whether any of the evaluations of the determined subset are unreliable based at least in part on bias of the evaluator user towards the author user being detected, the detecting of the bias of the evaluator user towards the author user being based at least in part on analysis of the numerical ratings included in the evaluations of the determined subset; and

automatically determining quality ratings for each of the multiple item review content pieces and for at least one of the multiple rating dimensions based on the numerical ratings of the received multiple evaluations other than the identified unreliable evaluations; and

providing one or more indications of at least some of the determined quality ratings.

2. The method of claim 1 further comprising:

receiving multiple requests from the customers of the online merchant for information regarding the multiple items available from the online merchant, each of the requests being received from one of the customers and corresponding to one or more of the multiple available items; and

responding to each of the multiple requests by selecting one or more of the item review content pieces that include item reviews for at least one of the one or more items to which the request corresponds and by providing the selected one or more item review content pieces to the customer from whom the request was received, the selecting of the one or more item review content pieces being based at least in part on the determined quality ratings of the one or more item review content pieces.

3. The method of claim 1 wherein the detecting of the bias of an evaluator user towards an author user includes determining that a bias relationship exists between the evaluator user and the author user if consistency of the numerical ratings included in a series of multiple evaluations supplied by the evaluator user for the item review content pieces created by the author user exceeds a predefined threshold, and includes selecting each of the multiple evaluations of the series as an identified unreliable evaluation if the bias relationship is determined to exist, and wherein the providing of the one or more indications of the at least some determined quality ratings includes providing information to at least some of the customers of the online merchant about the at least some determined quality ratings and about one or more determined bias relationships between evaluator users and author users.

4. A computer-implemented method for assessing reliability of evaluations supplied by users, the method comprising:

receiving multiple evaluations from an evaluator user, each of the received evaluations being for one of multiple content pieces that are supplied by an author user distinct from the evaluator user and including a quantitative rating of that content piece with respect to an indicated content rating dimension;

automatically assessing the received evaluations to identify one or more of the evaluations that are unreliable, the identifying being based at least in part on a determination that a bias relationship between the evaluator user and the author user exists at one or more times during which the identified one or more evaluations are received; and

providing an indication of the identified unreliable one or more evaluations, so that use of the identified unreliable one or more evaluations is inhibited.

5. The method of claim 4 further comprising receiving a plurality of supplied pieces of content that are created by a plurality of author users and that are supplied for use by other users, the multiple content pieces supplied by the author user being a subset of the plurality of supplied content pieces, and receiving a plurality of evaluations of the plurality of supplied content pieces from the evaluator user, the multiple evaluations being a subset of the plurality of evaluations, and wherein the automatic assessing of evaluations is performed for each of multiple of the plurality of author users based on a determined subset of the plurality of evaluations from the evaluator user that are for one or more content pieces created by that author user.

6. The method of claim 5 further comprising receiving a plurality of additional evaluations of the plurality of supplied content pieces that are each from one of multiple other evaluator users distinct from the evaluator user, at least one of the other evaluator users acting as one of the plurality of author users to supply one or more of the plurality of supplied content pieces, and wherein the automatic assessing of evaluations is further performed for each of one or more combinations of one of the other evaluator users and one of the plurality of author users based on a determined subset of the plurality of additional evaluations that are supplied by that one evaluator user and that are for one or more content pieces created by that one author user.

7. The method of claim 4 further comprising automatically determining an aggregate quality rating for each of at least some of the multiple supplied content pieces based at least in part on the received multiple evaluations, the determining performed so as to give less influence to the indicated unreliable evaluations than to supplied evaluations other than the indicated unreliable evaluations.

8. The method of claim 4 wherein the determination that a bias relationship between the evaluator user and the author user exists at one or more times includes automatically determining that consistency of the quantitative ratings included in a group of at least some of the multiple evaluations exceeds a predefined consistency threshold, and wherein the evaluations identified as unreliable include the evaluations of the group.

9. The method of claim 8 further comprising receiving a plurality of additional evaluations of the supplied content pieces from multiple other evaluator users and automatically identifying one or more of the additional evaluations as being unreliable based at least in part on determining that one or more additional bias relationships exist between one or more of the other evaluator users and the author user, the determining that the one or more additional bias relationships exist including selecting one of multiple predefined consistency thresholds for use with each of the other evaluator users in determining whether a bias relationship exists for that evaluator user and the author user, the selecting being based on one or more user characteristics of that evaluator user.

10. The method of claim 9 wherein the method is performed by one or more computing systems of an online merchant that makes items available to customers, wherein the one or more user characteristics of an evaluator user on which the selecting of one of the predefined consistency thresholds is based include at least one of whether the evaluator user has previously purchased any of the available items from the online merchant and of a recency of the evaluator user supplying one or more of the received multiple evaluations, and wherein the method further comprises receiving multiple additional supplied pieces of content that are created by a plurality of other author users and that are supplied for use by other users, the received additional supplied pieces of content being distinct from the received multiple supplied content pieces, the received multiple evaluations not including any evaluations of the multiple additional supplied pieces of content.

11. The method of claim 4 wherein the received multiple evaluations from the evaluator user for the multiple content pieces supplied by the author user are a subset of a plurality of evaluations supplied by the evaluator user for a plurality of supplied content pieces created by a plurality of author users, and wherein the determination that a bias relationship between the evaluator user and the author user exists at one or more times includes at least one of automatically determining that the received multiple evaluations include evaluations of at least a threshold minimum subset of all the content pieces created by one author user, of automatically determining that the received multiple evaluations are at least a threshold minimum subset of all the plurality of evaluations supplied by the evaluator user, and of automatically determining that the plurality of evaluations include, for each of one or more content pieces created by the author user, evaluations of at least a threshold minimum subset of multiple other of the plurality of content pieces that share a common topic with the content piece created by the author user.

12. The method of claim 4 wherein the method is performed on behalf of an organization that provides content pieces to users, wherein the evaluator user uses a first user account with the organization to provide the multiple evaluations, and wherein the automatic assessment of the received evaluations to identify the one or more unreliable evaluations further includes determining whether one or more of the evaluations are disguised duplicates of one or more other evaluations supplied by the evaluator user using one or more other user accounts with the organization.

13. The method of claim 4 wherein the method is performed on behalf of an organization that provides content pieces to users, and wherein the method further comprises:

receiving a plurality of additional evaluations of the supplied content pieces from multiple other evaluator users;

automatically identifying one or more of the additional evaluations as being unreliable based at least in part on determining that one or more additional bias relationships exist between at least one of the other evaluator users and the author user; and

providing information to at least some users of the organization about one or more of the determined bias relationships for the author user.

14. The method of claim 4 further comprising, after the determination that the bias relationship exists between the evaluator user and the author user, customizing future interactions with the evaluator user to reflect the determined bias relationship.

15. The method of claim 4 wherein the indicated content rating dimension for each of at least some of the received multiple evaluations is based on usefulness of the content piece being evaluated, and wherein at least some of the supplied content pieces each include at least one of information supplied to a merchant related to one or more items available for purchase from the merchant and of information available to users of a content provision service, the information supplied to the merchant including at least one of a review of one or more of the available items, a textual description of one or more of the available items, and an image of one or more of the available items, and the information available to the users of the content provision service being in a form that includes at least one of a media file, a streaming media clip, instructional information related to one or more topics, and a list of multiple related elements.

16. The method of claim 4 wherein the received evaluations from the evaluator user that are automatically assessed are selected so as to include only evaluations that correspond to at least one of a subset of the multiple content pieces that are of one of multiple distinct categories of content, and a subset of the multiple content pieces that are of one of multiple distinct types of content.

17. A non-transitory computer-readable medium whose contents cause a computing device to assess reliability of evaluations supplied by users, by performing a method comprising:

obtaining multiple evaluations by a first user of multiple pieces of content provided by a second user, the multiple evaluations including ratings of one or more attributes of at least some of the multiple provided content pieces;

determining a subset of the obtained evaluations by the first user for the multiple pieces of content provided by the second user;

automatically assessing the evaluations of the determined subset to determine whether one or more of the evaluations are unreliable based on at least one of a bias relationship existing between the first and second users when the one or more evaluations are supplied by the first user and of the one or more evaluations being disguised duplicates of one or more other evaluations supplied by the first user; and

providing an indication of the one or more evaluations if the one or more evaluations are determined to be unreliable.

18. The non-transitory computer-readable medium of claim 17 wherein the automatic assessing of the evaluations of the determined subset includes determining that the bias relationship exists between the first and second users for the one or more evaluations based at least in part on the ratings included in the evaluations, wherein the second user is an author user who creates at least some of the multiple provided content pieces, wherein the first user is an evaluator user distinct from the author user, wherein the multiple evaluations are each for one of the multiple provided content pieces and each include one or more quantitative ratings for the content piece being evaluated, such that the ratings of the one or more attributes of the at least some provided content pieces include the quantitative ratings, and wherein the provided indication includes an indication of the determined bias relationship.

19. The non-transitory computer-readable medium of claim 17 wherein the method is performed on behalf of an organization that provides content pieces to users, wherein the second user is an author user who creates at least some of the multiple provided content pieces for the organization, wherein the first user is a distinct evaluator user who uses a first user account with the organization to provide the multiple evaluations, wherein the automatic assessing of the evaluations of the determined subset includes determining that the one or more evaluations are disguised duplicates of the one or more other evaluations supplied by the first user based at least in part on the one or more other evaluations being provided by the first user when using one or more other user accounts with the organization, the determining including identifying that the first user used the first user account and used the one or more other user accounts, and wherein the provided indication includes an indication of the one or more other user accounts.

20. The non-transitory computer-readable medium of claim 17 wherein the computer-readable medium is a memory of a computing device containing the contents, and wherein the contents are instructions that when executed cause the computing device to perform the method.

21. A computing system configured to assess reliability of evaluations supplied by users, comprising:

one or more memories; and

an evaluation assessment system configured to, for each of one or more evaluator users who each supply multiple evaluations of objects associated with other users such that each of the supplied evaluations includes one or more ratings that each corresponds to an attribute of one of the objects;

determine a subset of the multiple evaluations supplied by the evaluator user for objects associated with one or more of the other users;

automatically assessing the determined subset of the multiple evaluations supplied by the evaluator user to determine whether one or more bias relationships exist between the evaluator user and the one or more of the other users such that one or more of the subset of evaluations are identified as unreliable based on being supplied as part of those bias relationships, the assessing being based at least in part on the ratings included in the subset of evaluations; and

providing an indication of the one or more unreliable evaluations if the one or more bias relationships are determined to exist.

22. The computing system of claim 21 wherein at least some of the objects associated with the other users are pieces of content provided by content-providing users other than the evaluator users, wherein at least some of the included ratings are quantitative ratings, wherein one or more bias relationships are determined to exist for each of at least one of the evaluator users, and wherein the evaluation assessment system is further configured to provide an indication for each of the at least one evaluator users of the one or more bias relationships determined to exist.

23. The computing system of claim 22 further comprising one or more other systems configured to generate an aggregate rating for at least one of one of the content-providing users and one of the provided content pieces based at least in part on at least some of the multiple evaluations supplied by the one or more evaluator users, the generating performed in a manner without using the one or more unreliable evaluations identified for each of the at least one evaluator users.

24. The computing system of claim 21 wherein at least some of the objects associated with the other users are items provided by the other users and made available to customers of an online merchant, wherein one or more bias relationships are determined to exist for each of at least one of the evaluator users, and wherein the online merchant provides at least some of the supplied evaluations for an item to potential customers of the item in a manner without providing the one or more unreliable evaluations identified for each of the at least one evaluator users.

25. The computing system of claim 21 wherein the evaluation assessment system includes software instructions for execution in memory of the computing system.

26. The computing system of claim 21 wherein the evaluation assessment system consists of a means for, for each of one or more evaluator users who each supply multiple evaluations of objects associated with other users such that each of the supplied evaluations includes one or more ratings that each corresponds to an attribute of one of the objects:

automatically assessing at least some of the multiple evaluations supplied by the evaluator user to determine whether one or more bias relationships exist between the evaluator user and one or more of the other users such that one or more of the subset of evaluations are identified as unreliable based on being supplied as part of those bias relationships, the assessing being based at least in part on the ratings included in the subset of evaluations; and

providing an indication of the one or more unreliable evaluations if the one or more bias relationships are determined to exist.