Method of evaluating learning rate of recommender systems

• US 20040243604A1
• Filed: 05/28/2004
• Published: 12/02/2004
• Est. Priority Date: 05/28/2003
• Status: Active Grant

• Current Assignee: John Nicholas and Kristin Gross Trust U/A/D April 13 2010
• Original Assignee: John Nicholas and Kristin Gross Trust U/A/D April 13 2010
• Inventors: Gross, John N.
• Granted Patent: US 7,783,512 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/0637 Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: G06Q 30/0631 Item recommendations

Abstract

A recommender system is analyzed to determine various performance characteristics, such as a learning rate for new items, or a learning rate for new subscriber tastes. Comparisons of different recommenders are presented to assist consumers and marketers in selecting appropriate e-commerce sites for purchasing, advertising, etc.

Claims

1. A method of evaluating a recommender system, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:

(a) identifying a first set of reference items to be used in evaluating the recommender system;

(b) generating a plurality of separate recommendations for a second set of items from the recommender system;

(c) correlating said first set of reference items with said second set of items to identify an awareness level exhibited by the recommender system for said first set of reference items;

(d) generating a rating for the recommender system based on the results of step (c) based on an evaluation of said awareness level.

2. The method of claim 1 wherein said awareness level is measured by identifying a first number of said first set of reference items which are appear as selections within a second number of said second set of items.

3. The method of claim 2, wherein said awareness level is measured by comparing said first number to said second number.

4. The method of claim 1, wherein said rating is a numerical value.

5. The method of claim 1, wherein at least some of said first set of references items are selected based on a determination of whether the recommender system is likely to have a relatively low number of existing explicit ratings for such first set of reference items.

6. The method of claim 1, further including a step:

repeating steps (a) through (d) for a second recommender system, and comparing a second rating for said second recommender system with said rating for the recommender system.

7. The method of claim 1, further including a step:

setting up a plurality of proxy accounts and respective demographic profiles prior to step (b) in order to interact with the recommender system.

8. The method of claim 1, further including a step:

generating a plurality of separate ratings for the recommender system based on separate evaluations for a plurality of separate demographic groups.

9. The method of claim 1, wherein said first set of reference items are identified by measuring an expected or measured popularity of a particular item.

10. The method of claim 9, wherein said measuring is performed by analyzing activities of a group of individuals interacting with an online website.

11. The method of claim 1, wherein the first set of reference items represent newly released content and the recommender system is tested to determine an extent of an awareness of such newly released content.

12. The method of claim 1, wherein said plurality of separate recommendations are derived from records generated by the recommender system.

13. The method of claim 1, wherein said plurality of separate recommendations are derived from a software agent extracting recommendations from the recommender system.

14. A method of evaluating a learning rate of a recommender system for new items, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:

(a) identifying a first set of new items to be used in evaluating the recommender system;

wherein at least some of said first set of new items are characterized by relatively few explicit ratings in a recommender system database;

(b) reviewing a plurality of separate recommendations for a second set of items made by the recommender system;

(c) correlating said first set of reference items with said second set of items to identify an awareness level exhibited by the recommender system for said first set of reference items;

(d) generating a rating for the recommender system based on the results of step (c) based on an evaluation of said awareness level.

15. The method of claim 14, wherein at least a subset of said first set of new items are characterized by sparse data in the recommender system database such that fewer than 1% of users identified in said recommender system database have provided an explicit rating for said at subset.

16. The method of claim 14, wherein the recommender system is a collaborative filtering based system.

17. The method of claim 14, wherein the recommender system is a content based filtering system.

18. A method of evaluating a learning rate of a recommender system for new preferences by users, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:

(a) providing a set of proxy accounts to interact with the recommender system;

(b) identifying a first set of recommendations given by the recommender system to said set of proxy accounts;

(c) modifying a profile of said set of proxy accounts to create a set of modified proxy accounts, including explicit ratings for items which can be recommended by the recommender system;

(d) identifying a second set of recommendations given by the recommender system to said set of modified proxy accounts.

19. The method of claim 18, further including a step:

correlating said first set of recommendations with said second set of recommendations to identify an awareness level exhibited by the recommender system for preference changes in said proxy accounts.

20. The method of claim 18, further including a step:

generating a rating for the recommender system based on the results of step (d).