Assessing content based on assessed trust in users

• US 7,822,631 B1
• Filed: 11/30/2005
• Issued: 10/26/2010
• Est. Priority Date: 08/22/2003
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ghosh, Arijit, Vander Mey, Christopher D., Marsh, Brian David
• Primary Examiner(s): Jeanty; Romain
• Application Number:
  US11/290,623
• Time in Patent Office: 1,791 Days
• Field of Search: 705/10
• US Class Current: 705/7.29
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0603 Catalogue ordering

Abstract

Techniques are described for managing content by identifying content that has attributes of interest (e.g., content that is useful, humorous and/or that otherwise has a sufficiently high degree of quality) and by determining how to use such identified content. In some situations, a piece of content is automatically assessed in a manner based on automatically assessed levels of trust in users who are associated with the content, such as a user who authored or otherwise supplied the content and/or users who evaluated the content. For example, an automatically assessed level of trust for a user may be based on prior activities of the user and be used to predict future behavior of the user as a supplier of acceptable content and/or as an acceptable evaluator of supplied content, such as based on prior activities of the user that are not related to supplying and/or evaluating content.

Claims

1. A computer-implemented method for an electronic marketplace to select customer-supplied item reviews for presentation to other customers based on automatically assessed quality of the item reviews, the assessed quality of an item review being based at least in part on automatically calculated trust scores of customers associated with the item review, the method comprising:

tracking activities of customers of the electronic marketplace in performing multiple types of interactions with the electronic marketplace, the tracking being performed by one or more programmed computing systems of the electronic marketplace;

for each of the customers, automatically calculating a trust score for the customer based at least in part on the tracked activities of the customer, the calculated trust score for a customer reflecting a level of trust granted by the electronic marketplace regarding future activities of the customer, the automatic calculating of the trust score for the customer being performed by the one or more programmed computing systems;

receiving multiple textual item reviews that are each supplied by one of the customers and that are each related to one of multiple items provided by the electronic marketplace to customers; and

for each of the item reviews, using the item review by,

automatically assessing an initial quality of the item review based at least in part on the calculated trust score for the customer from whom the item review was received, the assessing such that a calculated trust score reflecting a high level of trust will produce a higher assessed initial quality than a calculated trust score reflecting a low level of trust, the calculated trust score of the customer being based at least in part on types of interactions with the electronic marketplace that are unrelated to providing content or to evaluating content, the automatic assessing of the initial quality being performed by the one or more programmed computing systems;

automatically determining whether to present the item review to other customers based at least in part on whether the automatically assessed initial quality of the item review is sufficiently high, the automatic determining being performed by the one or more programmed computing systems; and

if the item review is presented to other customers,

receiving evaluations of the item review from each of at least some of the other customers, the received evaluations including numerical ratings of the item review for multiple rating dimensions that are related to distinct aspects of the item review, the numerical rating for a rating dimension indicating an assessment by one of the other customers of a degree to which an aspect of the item review for that rating dimension is satisfied;

automatically assessing an updated quality of the item review based at least in part on the automatically assessed initial quality and based at least in part on the numerical ratings of the received evaluations, the automatic assessing performed so as to weight the numerical ratings received from customers based on calculated trust scores for those customers, the automatic assessing of the updated quality being performed by the one or more programmed computing systems; and

if it is determined to continue presenting the item review to other customers based at least in part on the automatically assessed updated quality of the item review being sufficiently high, presenting the item review to other customers, so that quality of item reviews that are supplied by and evaluated by customers may be automatically assessed in a manner to reflect automatically calculated levels of trust in the customers based on multiple types of interactions of the customers with the electronic marketplace, and wherein each of the one or more programmed computing systems includes at least one computer processor.

2. The method of claim 1 further comprising:

receiving multiple other pieces of content that are each supplied by one of the customers and that are each related to one of the multiple items provided by the electronic marketplace to customers, the multiple other customer-supplied pieces of content being of one or more types distinct from item reviews;

automatically assessing a quality of each of the other pieces of content based at least in part on a calculated trust score for the customer from whom the piece of content was received and/or on calculated trust scores for customers from whom evaluations of the piece of content are received; and

determining whether to provide each of the other pieces of content to other customers based at least in part on whether the automatically assessed quality of the piece of content is sufficiently high.

3. The method of claim 1 wherein the tracked activities of each of the customers used in calculating trust scores for the customer include multiple of prior activities involving purchasing items from the electronic marketplace, prior activities to corroborate an identity of the customer, and prior activities to contribute information to the electronic marketplace of potential benefit to other customers.

4. The method of claim 3 wherein the tracked activities of the customers used in calculating trust scores for the customers do not include activities related to providing item reviews or to evaluating item reviews.

5. The method of claim 3 wherein the automatic calculating of the trust score for each of at least some of the customers is further based at least in part on prior actions taken by the electronic marketplace involving information contributed by the customer.

6. The method of claim 3 wherein the automatic calculating of the trust score for each of at least some of the customers is further based at least in part on information received by the electronic marketplace from one or more unaffiliated organizations that have assessed a level of trust for the customer based on interactions of the customer with those organizations.

7. The method of claim 1 further comprising, for each of at least one of the item reviews supplied by a customer and before the one item review is presented to other customers, automatically determining not to present the one item review to other customers based at least in part on the automatically assessed initial quality of the one item review being sufficiently low due to the calculated trust score of the customer who supplied the one item review, and in response suppressing presentation of the one item review to other customers.

8. The method of claim 1 wherein, for each of at least some of the item reviews, automatic assessing of quality of the item review is performed in such a manner as to assess multiple distinct degrees of quality for the item review that are each associated with one or more distinct groups of customers, the assessing of the degree of quality associated with each of the groups of customers being based at least in part on evaluations received from other customers who belong to that group, and wherein the automatic determining of whether to present each of the at least some item reviews to other customers includes determining whether to present the item review to the customers in each of one or more of the distinct groups based on the assessed degree of quality for the item review that is associated with the group.

9. The method of claim 8 wherein, for each of at least some of the customers, automatic calculating of the trust score for the customer is performed in such a manner as to calculate multiple distinct trust scores for the customer that are each associated with one or more distinct activities of the customer, and wherein the automatic assessing of the quality of each of at least some of the item reviews based at least in part on the calculated trust score for one of the at least some customers is performed in such a manner as to be based on one of the distinct trust scores for the one customer that is associated with an activity of the one customer corresponding to the assessing.

10. The method of claim 1 wherein the distinct aspects of the item reviews to which the multiple rating dimensions relate include multiple of usefulness, accuracy, informativeness, inappropriateness, and humorousness, and wherein each of the multiple rating dimensions has an associated weight such that the automatic assessing of the updated quality of an item review based at least in part on the numerical ratings of the received evaluations is performed so as to weight the numerical ratings based on the associated weights for the rating dimensions to which the numerical ratings correspond.

11. The method of claim 1 further comprising performing multiple other types of interactions with customers in a manner based on the calculated trust scores of the customers such that customers whose calculated trust scores reflect a high level of trust will receive more benefits than customers whose calculated trust scores reflect a low level of trust.

12. A computer-implemented method for selecting information to provide to customers of an electronic marketplace based on an automatic assessment of quality of the information, the method comprising:

receiving from a customer a piece of content related to an available item;

automatically determining a trust score for the customer based at least in part on prior activities of the customer at the electronic marketplace, at least some of the prior activities of the customer being unrelated to providing pieces of content to the electronic marketplace, the automatic determining of the trust score being performed by at least one of one or more programmed computing systems;

for each of one or more content rating dimensions, automatically determining an initial quantitative assessment of the received piece of content for the content rating dimension based at least in part on the determined trust score for the customer, the automatic determining of the initial quantitative assessment being performed by at least one of the one or more programmed computing systems;

after providing the received piece of content to other customers, receiving multiple evaluations of the piece of content from at least some of the other customers, each received evaluation including a quantitative assessment for each of one or more of the content rating dimensions;

for each of at least one of the content rating dimensions, automatically determining an updated quantitative assessment of the received piece of content for the content rating dimension based at least in part on combining quantitative assessments for the content rating dimension from the received evaluations, the combining such that determined trust scores for the at least some other customers from prior activities of those other customers are used to weight the quantitative assessments in the evaluations received from those other customers, the automatic determining of the updated quantitative assessment being performed by at least one of the one or more programmed computing systems; and

determining whether to provide the piece of content to one or more additional customers based at least in part on one or more of the automatically determined quantitative assessments for the piece of content, and wherein each of the one or more programmed computing systems includes at least one computer processor.

13. The method of claim 12 further comprising, before the providing of the received piece of content to the other customers, determining that at least one of the automatically determined initial quantitative assessments of the received piece of content does not exceed a suppression threshold, and suppressing the received piece of content based on the suppression threshold so as to prevent the providing of the received piece of content to the other customers.

14. The method of claim 13 further comprising, after the suppressing of the received piece of content, automatically determining a revised initial quantitative assessment of the received piece of content for a content rating dimension based at least in part on a revised determined trust score for the customer from whom the piece of content was received, and halting the suppressing of the received piece of content if the automatically determined revised initial quantitative assessment exceeds the suppression threshold.

15. The method of claim 13 wherein the suppression threshold is predefined.

16. The method of claim 12 wherein the determining of whether to provide the piece of content to one or more additional customers includes determining that at least one of the automatically determined updated quantitative assessments of the received piece of content does not exceed a suppression threshold, and in response suppressing the received piece of content so that it is not provided to additional customers.

17. The method of claim 16 wherein the suppressing is further based on determining that all of the automatically determined updated quantitative assessments of the received piece of content do not exceed the suppression threshold.

18. The method of claim 12 wherein the determining of whether to provide the piece of content to one or more additional customers includes determining to provide the piece of content to additional customers, and further comprising providing the piece of content to additional customers.

19. The method of claim 18 wherein one or more quantitative assessments are automatically determined for each of numerous other pieces of content, and wherein the determining of whether to provide the piece of content to one or more additional customers is further based on a comparison of one or more of the automatically determined quantitative assessments for the piece of content to automatically determined quantitative assessments for the numerous other pieces of content.

20. The method of claim 12 wherein the piece of content includes textual information authored by the customer from whom the piece of content is received.

21. The method of claim 12 wherein the piece of content includes one or more images generated by the customer from whom the piece of content is received.

22. The method of claim 12 wherein the piece of content includes an audio clip generated by the customer from whom the piece of content is received.

23. The method of claim 12 wherein the piece of content includes a group of related pieces of information that is created by the customer from whom the piece of content is received.

24. The method of claim 12 wherein automatic determining of a quantitative assessment of the received piece of content for each of at least one of the content rating dimensions includes determining a quantitative assessment of the received piece of content for each of multiple distinct content rating dimensions.

25. The method of claim 24 wherein the automatic determining of a quantitative assessment of the received piece of content for each of at least one of the content rating dimensions further includes automatically determining an overall assessment of the received piece of content based at least in part on multiple of the determined quantitative assessments of the received piece of content.

26. The method of claim 24 wherein the multiple content rating dimensions are each related to a distinct attribute of the piece of the content.

27. The method of claim 24 wherein the multiple content rating dimensions reflect multiple of usefulness, helpfulness, humorousness, inappropriateness and accuracy.

28. The method of claim 12 wherein the quantitative assessment of the received piece of content for a content rating dimension is a content quality score that reflects a degree of quality for the piece of content for that content rating dimension.

29. The method of claim 12 wherein the automatic determining of the updated quantitative assessment of the received piece of content for one of the content rating dimensions is further based in part on the automatically determined initial quantitative assessment of the received piece of content for that one content rating dimension.

30. The method of claim 12 wherein automatic determining of a quantitative assessment of the received piece of content for one of the content rating dimensions includes determining multiple distinct quantitative assessments of the received piece of content for the one content rating dimension, each of the multiple distinct quantitative assessments being associated with one of multiple distinct groups of customers.

31. The method of claim 30 wherein the automatic determining of each of the multiple distinct quantitative assessments of the received piece of content for the one content rating dimension is based on evaluations received from customers who belong to the one group of customers with which the quantitative assessment is associated.

32. The method of claim 30 wherein the determining of whether to provide the piece of content to one or more additional customers is further based at least in part on one of the multiple distinct quantitative assessments of the received piece of content that is associated with a group of customers that includes the one or more additional customers.

33. The method of claim 30 further comprising automatically identifying the multiple distinct groups of customers so as to identify similar customers based on analysis of information related to the customers.

34. The method of claim 12 wherein automatic determining of a quantitative assessment of the received piece of content for one of the content rating dimensions includes determining multiple distinct quantitative assessments of the received piece of content for the one content rating dimension, each of the multiple distinct quantitative assessments being associated with a distinct customer.

35. The method of claim 12 wherein automatic determining of a quantitative assessment of the received piece of content based on an automatically determined trust score for a customer includes selecting one of multiple automatically determined trust scores for the customer for use in the automatic determining of the quantitative assessment, the selecting being based at least in part on a type of the received piece of content such that the selected trust score is associated with the type.

36. The method of claim 12 wherein automatic determining of a quantitative assessment of the received piece of content based on an automatically determined trust score for a customer includes selecting one of multiple automatically determined trust scores for the customer for use in the automatic determining of the quantitative assessment, the selecting being based at least in part on a role of the customer related to the received piece of content such that the selected trust score is associated with the role.

37. The method of claim 12 wherein the prior activities of the at least some other customers on which the determined trust scores for those other customers are based include prior activities of those customers at the electronic marketplace that are unrelated to evaluating pieces of content from the electronic marketplace.

38. The method of claim 37 wherein the all of the prior activities of the at least some other customers on which the determined trust scores for those other customers are based and all of the prior activities of the customer from whom the piece of content is received are unrelated to evaluating pieces of content from the electronic marketplace and are unrelated to providing pieces of content to the electronic marketplace.

39. The method of claim 12 wherein the prior activities of the at least some other customers on which the determined trust scores for those other customers are based and the prior activities of the customer from whom the piece of content is received are of multiple types that each are associated with one of multiple trustworthiness factors for use in assessing trustworthiness of the customer.

40. The method of claim 12 wherein the prior activities of the at least some other customers on which the determined trust scores for those other customers are based and the prior activities of the customer from whom the piece of content is received include multiple of prior activities involving purchasing items from the electronic marketplace, prior activities to corroborate an identity of the customer, and prior activities to contribute information to the electronic marketplace of potential benefit to other customers.

41. The method of claim 40 wherein the automatic determining of a trust score for each of at least some customers is further based at least in part on prior actions taken by the electronic marketplace involving information supplied by the customer.

42. The method of claim 40 wherein the automatic determining of a trust score for each of at least some customers is further based at least in part on information received by the electronic marketplace from one or more external entities.

43. The method of claim 42 wherein the one or more external entities are unrelated to the electronic marketplace, and wherein the information received from the one or more external entities for each of the at least some customers includes an indication of a level of trust assessed for the customer by the one or more external entities based on interactions of the customer with those entities.

44. The method of claim 12 further comprising providing to one or more external entities for use by the external entities an indication of one or more automatically determined trust scores for a customer and/or an indication of one or more automatically determined quantitative assessments for the received piece of content.

45. The method of claim 12 wherein the automatic determining of a trust score for each of at least one of the customers includes automatically determining multiple distinct trust scores for the customer.

46. The method of claim 45 wherein the automatic determining of the multiple trust scores for the customer includes determining a trust score associated with each of multiple types of activities of the customer.

47. The method of claim 45 wherein the automatic determining of the multiple trust scores for the customer includes determining a trust score associated with each of multiple types of content.

48. The method of claim 12 further comprising automatically determining trust scores for each of multiple users who are not customers of the electronic marketplace.

49. The method of claim 12 further comprising automatically determining content quality scores for each of multiple pieces of content that are not available from the electronic marketplace.

50. A computer-readable storage medium whose contents include instructions that when executed configure a computing device to select information to provide to users based on an automatic assessment of quality of the information, by performing a method comprising:

receiving an indication of information supplied by a first user;

after providing indications of the information to other users, receiving evaluations of the information from each of at least some of the other users;

automatically assessing the quality of the information based at least in part on automatically determined trust scores for the other users in such a manner that the automatically determined trust score for each of the other users is used to influence an effect of the evaluation of that other user on the assessed quality, the automatically determined trust score for each of the other users being based at least in part on prior activities of that other user that are not related to evaluating information or to supplying information, the automatic assessing of the quality being performed by the configured computing device; and

automatically determining a manner of using the information based at least in part on the automatically assessed quality of the information, the automatic determining being performed by the configured computing device.

51. The computer-readable storage medium of claim 50 wherein the automatic determining of the manner of using the information includes determining whether to provide the information to each of one or more users, and wherein the method further comprises, before the providing of the information to the other users, automatically predicting the quality of the information based at least in part on an automatically determined trust score for the first user that is based at least in part on prior activities of the first user that are not related to supplying information, and preventing the providing of the information to the other users if the automatically predicted quality of the information is sufficiently low.

52. The computer-readable storage medium of claim 51 wherein the method further comprises, before the receiving of the indication of the information supplied by the first user, automatically determining the trust scores for each of the first user and the other users based on the prior activities of that user.

53. The computer-readable storage medium of claim 50 wherein the automatic assessing of the quality of the information is performed in such a manner as to generate multiple distinct quality assessments for the information that are each associated with one or more distinct groups of users, the generating of the quality assessment associated with each of the groups of users being based at least in part on evaluations received from other users who belong to that group, and wherein the automatic determining of whether to provide the information to additional users is based on the quality assessment associated with one of the groups to which the additional users belong.

54. The computer-readable storage medium of claim 50 wherein the automatically assessed quality of the information is a quantitative assessment of the information for each of one or more rating dimensions.

55. The computer-readable storage medium of claim 50 wherein the supplied information is related to an item available from a merchant, and wherein the information is supplied by the first user for use by other users.

56. The computer-readable storage medium of claim 50 wherein the computer-readable medium is a memory of the computing device.

57. A computing system configured to select information to provide to users based on an automatic assessment of quality of the information, comprising:

one or more processors;

a content quality assessment system that is configured to, when executed by at least one of the one or more processors;

automatically assess a predicted degree of quality of a piece of content supplied by a first user for use by other users, the predicting of the degree of quality being based at least in part on a trust score for the first user that is automatically determined based at least in part on prior activities of the first user that are not related to supplying pieces of content for use by other users; and

automatically update the assessed degree of quality of the piece of content based at least in part on evaluations of the piece of content received from multiple other users to whom the piece of content was made available and on automatically determined trust scores for the other users such that the automatically determined trust score for each of the other users is used to influence an effect of the evaluation of that other user on the updated assessed degree of quality; and

a content manager system that is configured to, when executed by at least one of the one or more processors, automatically determine whether to make the supplied piece of content available to other users based at least in part on the automatically assessed predicted degree of quality of the piece of content.

58. The computing system of claim 57 wherein the content quality assessment system is further configured to automatically assess an updated degree of quality of the piece of content based at least in part on trust scores for multiple other users associated with the piece of content that are automatically determined based on prior activities of the other users.

59. The computing system of claim 58 further comprising a user trust score assessment system configured to automatically determine the trust scores for the first user and the other users based on the prior activities.

60. The computing system of claim 57 wherein the content manager system is further configured to suppress the supplied piece of content from being made available to other users if the automatically assessed predicted degree of quality of the piece of content is sufficiently low.

61. The computing system of claim 57 wherein the automatic assessing of the predicted degree of quality of the piece of content is performed in such a manner as to generate multiple distinct predicted quality assessments for the piece of content that are each associated with one or more distinct groups of users, and wherein the automatic determining of whether to provide the piece of content to other users is based on the predicted quality assessment associated with one of the groups to which the other users belong.

62. The computing system of claim 57 wherein the automatically assessed predicted degree of quality of the piece of content is a predicted quantitative assessment of the piece of content for each of one or more rating dimensions.

63. The computing system of claim 57 wherein the supplied piece of content is related to an item available from a merchant.

64. The computing system of claim 57 wherein the content quality assessment system and the content manager system each include software instructions for execution by the one or more processors of the computing system.

65. The computing system of claim 57 wherein the content quality assessment system consists of a means for automatically assessing a predicted degree of quality of a piece of content supplied by a first user for use by other users, the predicting of the degree of quality being based at least in part on a trust score for the first user that is automatically determined based at least in part on prior activities of the first user that are not related to supplying pieces of content for use by other users, and wherein the content manager system consists of a means for automatically determining whether to make the supplied piece of content available to other users based at least in part on the automatically assessed predicted degree of quality of the piece of content.