Approach for estimating user ratings of items

• US 7,617,127 B2
• Filed: 02/06/2004
• Issued: 11/10/2009
• Est. Priority Date: 04/28/2000
• Status: Active Grant

• Current Assignee: Netflix, Inc.
• Original Assignee: Netflix, Inc.
• Inventors: Shih, Lawrence Wen-Kai, Lanning, Stanley Miguel, Purcell, Shawn Michael, Hunt, Neil Duncan, Ciancutti, John Robert, Hastings, W. Reed
• Primary Examiner(s): ZURITA, JAMES H
• Application Number:
  US10/773,843
• Publication Number:
  US 20050125307A1
• Time in Patent Office: 2,104 Days
• Field of Search: 705/26, 705/37
• US Class Current: 705/26.61
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0204 Market segmentation
• CPC Class Codes: G06Q 30/0217 involving input on products or services in exchange for incentives or rewards
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange

Abstract

According to another embodiment of the invention, an approach is provided for estimating how a particular user would rate a particular item from a plurality of items. The approach is applicable to any type of items, including rental items such as movies, music and games, and the invention is not limited to any particular type of item. One or more items from the plurality of items that have ratings similar to the particular item are identified. Then, one or more other users are identified that have given ratings to the one or more items that are substantially similar to ratings given by the particular user to the one or more items. Finally, an estimation is made of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users.

Claims

1. A machine-implemented method for estimating how a particular user of a plurality of users would rate a particular item, from a plurality of items, that the particular user has not yet rated, the method comprising the steps of:

a computer system identifying, from the plurality of items, one or more reference items that have ratings similar to ratings of the particular item that the particular user has not yet rated;

a computer system identifying, based on the one or more reference items that have ratings similar to ratings of the particular item that the particular user has not yet rated, one or more other users of the plurality of users that have given ratings to the one or more reference items that are substantially similar to ratings given by the particular user to the one or more reference items;

a computer system generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given to the particular item by the one or more other users.

2. The machine implemented method as recited in claim 1, wherein the step of identifying, from the plurality of items, one or more reference items that have ratings similar to the particular item is performed off-line in advance of the other steps.

3. The machine-implemented method as recited in claim 1, wherein identifying one or more reference items from a plurality of items that have ratings similar to the particular item includes calculating a statistical correlation between the particular item and the other items from the plurality of items.

4. The machine-implemented method as recited in claim 1, further comprising the machine-implemented step of determining whether the particular item and each of the one or more reference items have at least a specified number of co-ratings.

5. The machine-implemented method as recited in claim 1, further comprising the machine-implemented step of determining whether the particular item and one of the one or more reference items have at least a specified fraction of co-ratings between the particular item and the one of the one or more reference items divided by the number of ratings of either the particular item or the one of the one or more reference items.

6. The machine-implemented method as recited in claim 1, wherein the step of identifying one or more reference items from the plurality of items that have ratings similar to the particular item includes identifying whether the particular item and the one or more reference items have a number of co-raters greater than a specified fraction of a statistically expected number, given the number of ratings of each item counted separately.

7. The machine-implemented method as recited in claim 1, wherein the step of identifying one or more reference items from the plurality of items that have ratings similar to the particular item includes consideration of one or more of a statistical correlation between the particular item and the one or more reference items, whether the particular item and the one or more reference items have at least a specified number of co-ratings, whether the particular item and the one or more reference items have at least a specified fraction of co-ratings, and whether the particular item and the one or more reference items have at least a specified fraction of the statistically expected number of co-ratings.

8. The machine-implemented method as recited in claim 1, further comprising the machine-implemented step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having at least a specified minimum number of ratings.

9. The machine-implemented method as recited in claim 1, further comprising the machine-implemented step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having less than a specified maximum number of ratings.

10. The machine-implemented method as recited in claim 1, further comprising the machine-implemented step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having an average rating within a specified range.

11. The machine-implemented method as recited in claim 1, further comprising the machine-implemented step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having a rating distribution within a specified tolerance of a Normal distribution.

12. The machine-implemented method as recited in claim 1, further comprising the machine-implemented step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having a rating standard deviation within a specified range.

13. The machine-implemented method as recited in claim 1, wherein identifying one or more other users that have given ratings to the one or more reference items that are substantially similar to ratings given by the particular user to the one or more reference items includes consideration of one or more of whether the one or more other users have at least a specified minimum number of co-ratings, whether the one or more other users have less than a specified maximum number of ratings, whether the one or more other users have an average rating within a specified range, whether the one or more other users have a rating distribution within a specified tolerance of a Normal distribution, and whether the one or more other users have a rating standard deviation within a specified range.

14. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes choosing the one or more other users for whom a statistical correlation with the particular user of co-ratings of the one or more reference items is greater than a specified threshold.

15. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes choosing the one or more other users for whom a vector cosine distance of co-ratings of the one or more reference items is smaller than a specified threshold.

16. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes choosing the one or more other users who have co-rated a subset of the one or more reference items exactly the same as the particular user.

17. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes choosing the one or more other users who have rated co-rated a subset of the one or more reference items within a specified threshold of the ratings by the particular user.

18. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes consideration of one or more of whether the one or more users have a correlation of co-ratings of the one or more reference items greater than a specified threshold, whether the one or more users have a vector cosine distance of co-ratings of the one or more reference items smaller than a specified threshold, whether the one or more users have co-rated a subset of the one or more reference items exactly the same, and whether the one or more users having co-rated a subset of the one or more reference items within a specified threshold of the ratings by the particular user.

19. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes choosing a specified number of other users most similar to the particular user.

20. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes choosing other users whose similarity to the particular user is better than a specified threshold.

21. The machine-implemented method as recited in claim 1, wherein identifying the one or more other users includes choosing up to a specified number of other users most similar to the particular user, where all the chosen other users have similarity better than a specified threshold.

22. The machine-implemented method as recited in claim 1, wherein generating an estimation of how the user would rate the particular item based upon ratings for the particular item given by the one or more other users includes determining an average rating of the particular item by the one or more other users.

23. The machine-implemented method as recited in claim 1, wherein generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes determining an average rating of the particular item by the one or more other users.

24. The machine-implemented method as recited in claim 1, further comprising the machine-implemented steps of:

generating, for each of the one or more users, a separate estimation of how the particular user would rate the particular item based upon the rating for the particular item given by each of the one or more other users, and

calculating a confidence metric based upon disparities between the separate estimations.

25. The machine-implemented method as recited in claim 1, wherein the plurality of items is a plurality of movies, the one or more reference items is one or more movies and the particular item is a particular movie.

26. The machine-implemented method as recited in claim 1, wherein the plurality of items is a plurality of music items, the one or more reference items is one or more music items and the particular item is a particular music item.

27. The machine-implemented method of claim 1, wherein:

generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes determining a similarity between the ratings given by the one or more other users to the one or more reference items and the ratings given by the particular user to the one or more reference items.

28. A volatile or non-volatile machine-readable storage medium for estimating how a particular user of a plurality of users would rate a particular item, from a plurality of items, that the particular user has not yet rated, the machine-readable storage medium storing instructions which, when processed by one or more processors, cause performance of the steps of:

identifying, from the plurality of items, one or more reference items that have ratings similar to ratings of the particular item that the particular user has not yet rated;

identifying, based on the one or more reference items that have ratings similar to ratings of the particular item that the particular user has not yet rated, one or more other users of the plurality of users that have given ratings to the one or more reference items that are substantially similar to ratings given by the particular user to the one or more reference items;

generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given to the particular item by the one or more other users.

29. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein the step of identifying, from the plurality of items, one or more other items that have ratings similar to the particular item is performed off line in advance of the other steps.

30. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying one or more reference items from a plurality of items that have ratings similar to the particular item includes calculating a statistical correlation between the particular item and the other items from the plurality of items.

31. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional step of determining whether the particular item and each of the one or more reference items have at least a specified number of co-ratings.

32. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional step of determining whether the particular item and one of the one or more reference items have at least a specified fraction of co-ratings between the particular item and the one of the one or more reference items divided by the number of ratings of either the particular item or the one of the one or more reference items.

33. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein the step of identifying one or more reference items from the plurality of items that have ratings similar to the particular item includes identifying whether the particular item and the one or more reference items have a number of co-raters greater than a specified fraction of a statistically expected number, given the number of ratings of each item counted separately.

34. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein the step of identifying one or more reference items from the plurality of items that have ratings similar to the particular item includes consideration of one or more of a statistical correlation between the particular item and the one or more reference items, whether the particular item and the one or more reference items have at least a specified number of co-ratings, whether the particular item and the one or more reference items have at least a specified fraction of co-ratings, and whether the particular item and the one or more reference items have at least a specified fraction of the statistically expected number of co-ratings.

35. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having at least a minimum number of ratings.

36. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having less than a maximum number of ratings.

37. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having an average rating within a specified range.

38. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having a rating distribution within a specified tolerance of a Normal distribution.

39. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional step of selecting the one or more other users from a plurality of other users based upon each of the one or more other users having a rating standard deviation within a specified range.

40. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying one or more other users that have given ratings to the one or more reference items that are substantially similar to ratings given by the particular user to the one or more reference items includes consideration of one or more of whether the one or more other users have at least a specified minimum number of co-ratings, whether the one or more other users have less than a specified maximum number of ratings, whether the one or more other users have an average rating within a specified range, whether the one or more other users have a rating distribution within a specified tolerance of a Normal distribution, and whether the one or more other users have a rating standard deviation within a specified range.

41. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes choosing the one or more other users for whom a statistical correlation with the particular user of co-ratings of the one or more reference items is greater than a specified threshold.

42. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes choosing the one or more other users for whom a vector cosine distance of co-ratings of the one or more reference items is smaller than a specified threshold.

43. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes choosing the one or more other users who have co-rated a subset of the one or more reference items exactly the same as the particular user.

44. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes choosing the one or more other users who have rated co-rated a subset of the one or more reference items within a specified threshold of the ratings by the particular user.

45. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes consideration of one or more of whether the one or more users have a correlation of co-ratings of the one or more reference items greater than a specified threshold, whether the one or more users have a vector cosine distance of co-ratings of the one or more reference items smaller than a specified threshold, whether the one or more users have co-rated a subset of the one or more reference items exactly the same, and whether the one or more users having co-rated a subset of the one or more reference items within a specified threshold of the ratings by the particular user.

46. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes choosing a specified number of other users most similar to the particular user.

47. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes choosing other users whose similarity to the particular user is better than a specified threshold.

48. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein identifying the one or more other users includes choosing up to a specified number of other users most similar to the particular user, where all the chosen other users have similarity better than a specified threshold.

49. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes determining an average rating of the particular item by the one or more other users.

50. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes determining an average rating of the particular item by the one or more other users.

51. The volatile or non-volatile machine-readable storage medium as recited in claim 28, further comprising one or more additional instructions which, when processed by the one or more processors, cause the one or more processors to perform the additional steps of:

generating, for each of the one or more users, a separate estimation of how the particular user would rate the particular item based upon the rating for the particular item given by each of the one or more other users, and

calculating a confidence metric based upon disparities between the separate estimations.

52. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein the plurality of items is a plurality of movies, the one or more reference items is one or more movies and the particular item is a particular movie.

53. The volatile or non-volatile machine-readable storage medium as recited in claim 28, wherein the plurality of items is a plurality of music items, the one or more reference items is one or more music items and the particular item is a particular music item.

54. The volatile or non-volatile machine-readable storage medium of claim 28, wherein:

the instructions for generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes instructions for determining a similarity between the ratings given by the one or more other users to the one or more reference items and the ratings given by the particular user to the one or more reference items.

55. An apparatus for estimating how a particular user of a plurality of users would rate a particular item, from a plurality of items, that the particular user has not yet rated, the apparatus comprising:

means for identifying, from the plurality of items, one or more reference items that have ratings similar to ratings of the particular item that the particular user has not yet rated;

means for identifying, based on the one or more reference items that have ratings similar to ratings of the particular item that the particular user has not yet rated, one or more other users of the plurality of users that have given ratings to the one or more reference items that are substantially similar to ratings given by the particular user to the one or more reference items;

means for generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given to the particular item by the one or more other users.

56. The apparatus as recited in claim 55, wherein the means for identifying, from the plurality of items, one or more other items that have ratings similar to the particular item is operable off-line in advance of the other steps.

57. The apparatus as recited in claim 55, wherein the means for identifying one or more reference items from a plurality of items that have ratings similar to the particular item includes means for calculating a statistical correlation between the particular item and the other items from the plurality of items.

58. The apparatus as recited in claim 55, further comprising:

means for determining whether the particular item and each of the one or more reference items have at least a specified number of co-ratings.

59. The apparatus as recited in claim 55, further comprising:

means for determining whether the particular item and one of the one or more reference items have at least a specified fraction of co-ratings between the particular item and the one of the one or more reference items divided by the number of ratings of either the particular item or the one of the one or more reference items.

60. The apparatus as recited in claim 55, wherein the means for identifying one or more reference items from the plurality of items that have ratings similar to the particular item includes means for identifying whether the particular item and the one or more reference items have a number of co-raters greater than a specified fraction of a statistically expected number, given the number of ratings of each item counted separately.

61. The apparatus as recited in claim 55, wherein the means for identifying one or more reference items from the plurality of items that have ratings similar to the particular item includes means for consideration of one or more of a statistical correlation between the particular item and the one or more reference items, whether the particular item and the one or more reference items have at least a specified number of co-ratings, whether the particular item and the one or more reference items have at least a specified fraction of co-ratings, and whether the particular item and the one or more reference items have at least a specified fraction of the statistically expected number of co-ratings.

62. The apparatus as recited in claim 55, further comprising:

means for selecting the one or more other users from a plurality of other users based upon each of the one or more other users having at least a minimum number of ratings.

63. The apparatus as recited in claim 55, further comprising:

means for selecting the one or more other users from a plurality of other users based upon each of the one or more other users having less than a maximum number of ratings.

64. The apparatus as recited in claim 55, further comprising:

means for selecting the one or more other users from a plurality of other users based upon each of the one or more other users having an average rating within a specified range.

65. The apparatus as recited in claim 55, further comprising:

means for selecting the one or more other users from a plurality of other users based upon each of the one or more other users having a rating distribution within a specified tolerance of a Normal distribution.

66. The apparatus as recited in claim 55, further comprising:

means for selecting the one or more other users from a plurality of other users based upon each of the one or more other users having a rating standard deviation within a specified range.

67. The apparatus as recited in claim 55, wherein the means for identifying one or more other users that have given ratings to the one or more reference items that are substantially similar to ratings given by the particular user to the one or more reference items includes means for consideration of one or more of whether the one or more other users have at least a specified minimum number of co-ratings, whether the one or more other users have less than a specified maximum number of ratings, whether the one or more other users have an average rating within a specified range, whether the one or more other users have a rating distribution within a specified tolerance of a Normal distribution, and whether the one or more other users have a rating standard deviation within a specified range.

68. The apparatus as recited in claim 55, wherein the means for identifying the one or more other users includes means for choosing the one or more other users for whom a statistical correlation with the particular user of co-ratings of the one or more reference items is greater than a specified threshold.

69. The apparatus as recited in claim 55, wherein the means for identifying the one or more other users includes means for choosing the one or more other users for whom a vector cosine distance of co-ratings of the one or more reference items is smaller than a specified threshold.

70. The apparatus as recited in claim 55, wherein the means for identifying the one or more other users includes means for choosing the one or more other users who have co-rated a subset of the one or more reference items exactly the same as the particular user.

71. The apparatus as recited in claim 55, wherein the means for identifying the one or more other users includes means for choosing the one or more other users who have rated co-rated a subset of the one or more reference items within a specified threshold of the ratings by the particular user.

72. The apparatus as recited in claim 55, wherein the means for identifying the one or more other users includes means for consideration of one or more of whether the one or more users have a correlation of co-ratings of the one or more reference items greater than a specified threshold, whether the one or more users have a vector cosine distance of co-ratings of the one or more reference items smaller than a specified threshold, whether the one or more users have co-rated a subset of the one or more reference items exactly the same, and whether the one or more users having co-rated a subset of the one or more reference items within a specified threshold of the ratings by the particular user.

73. The apparatus as recited in claim 55, wherein the means for identifying the one or more other users includes means for choosing a specified number of other users most similar to the particular user.

74. The apparatus as recited in claim 55, wherein the means for identifying the one or more other users includes means for choosing other users whose similarity to the particular user is better than a specified threshold.

75. The apparatus as recited in claim 55, wherein means for identifying the one or more other users includes means for choosing up to a specified number of other users most similar to the particular user, where all the chosen other users have similarity better than a specified threshold.

76. The apparatus as recited in claim 55, wherein the means for generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes means for determining an average rating of the particular item by the one or more other users.

77. The apparatus as recited in claim 55, wherein the means for generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes means for determining an average rating of the particular item by the one or more other users.

78. The apparatus as recited in claim 55, further comprising:

means for generating, for each of the one or more users, a separate estimation of how the particular user would rate the particular item based upon the rating for the particular item given by each of the one or more other users, and

means for calculating a confidence metric based upon disparities between the separate estimations.

79. The apparatus as recited in claim 55, wherein the plurality of items is a plurality of movies, the one or more reference items is one or more movies and the particular item is a particular movie.

80. The apparatus as recited in claim 55, wherein the plurality of items is a plurality of music items, the one or more reference items is one or more music items and the particular item is a particular music item.

81. The apparatus of claim 55, wherein:

the means for generating an estimation of how the particular user would rate the particular item based upon ratings for the particular item given by the one or more other users includes means for determining a similarity between the ratings given by the one or more other users to the one or more reference items and the ratings given by the particular user to the one or more reference items.