Method of testing online recommender system

• US 20040243527A1
• Filed: 05/28/2004
• Published: 12/02/2004
• Est. Priority Date: 05/28/2003
• Status: Active Grant

• Current Assignee: John Nicholas and Kristin Gross Trust U/A/D April 13 2010
• Original Assignee: John Nicholas and Kristin Gross Trust U/A/D April 13 2010
• Inventors: Gross, John N.
• Granted Patent: US 8,630,960 B2
• Time in Patent Office: Days
• US Class Current: 705/500
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0243 Comparative campaigns
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: G06Q 99/00 Subject matter not provided for in other groups of this subclass

Abstract

A recommender system can be analyzed to determine various performance characteristics of an online content service provider. The recommender system is analyzed using a predetermined policy, to determine if it satisfies such policy, and/or has other measurable intended and/or unintended biases. The policy can include such parameters as whether a particular profile is presented with certain particular types of items by the recommender system. The reliability of search engines can also be tested using a similar approach.

Claims

1. A method of testing a recommender system, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:

(a) setting up a policy to be used in testing the recommender system;

(b) setting up a plurality of separate proxy accounts at the online content service provider;

(c) causing said recommender system to interact with said plurality of separate proxy accounts and generate a plurality of separate recommendations for a plurality of corresponding items;

(d) verifying a compliance level with said policy by examining said separate recommendations.

2. The method of claim 1 wherein said policy includes at least one rule associating a particular subscriber profile with a particular item.

3. The method of claim 2, wherein said particular subscriber profile is defined by reference to a predetermined demographic profile which specifies at least an age and sex of a subscriber.

4. The method of claim 2, wherein each particular item specified in said policy originates from a common content provider source.

5. The method of claim 1, wherein said policy is associated with an expected or measured popularity of a particular item for a particular subscriber.

6. The method of claim 1, wherein said policy is a bias which favors items originating from a particular content provider, so that the recommender system is tested to verify that it behaves with said bias.

7. The method of claim 1, wherein said plurality of separate proxy accounts are set up with separate demographic profiles.

8. The method of claim 1, wherein each recommendation made to a particular proxy account is logged.

9. The method of claim 8, wherein said recommendations are classified and compiled into at least two categories:

(a) recommended items that satisfy said policy;

(b) recommended items that do not satisfy said policy.

10. The method of claim 9, wherein said recommendations are further classified and compiled by reference to a third category:

(a) recommended items that do not satisfy said policy but which originate from a predetermined content provider.

11. The method of claim 1, wherein during step (d) a report is generated which identifies whether first content from a first content provider is recommended as frequently as second content from a second content provider.

12. The method of claim 11, wherein during step (d) said report identifies a list of most frequently recommended items to said separate proxy accounts.

13. The method of claim 11, wherein during step (d) said report identifies a plurality of lists identifying the most frequently recommended items to each of said separate corresponding proxy accounts.

14. The method of claim 1, wherein during step (d) a report is generated which identifies a degree of bias exhibited by the recommender system with respect to items originating from one or more particular content providers.

15. The method of claim 1, wherein the item is a movie title, a book title, a music tide, an article being auctioned, and/or a television program.

16. The method of claim 15, wherein the item includes newly released content, and the recommender system is tested to determine an extent of an awareness of such newly released content by the recommender system.

17. The method of claim 1, wherein the policy is determined by evaluating a popularity of an item as determined from analyzing usage by online subscribers of text descriptors associated with the item.

18. The method of claim 17, wherein a demand for a rental movie tide is determined by reference to a measurement of usage by online subscribers of text descriptors associated with the item during a period in which said movie tide is in active release in public movie theatres.

19. The method of claim 18, wherein a growth in popularity of said movie tide is also determined in setting said policy.

20. The method of claim 1, furthering including a step:

providing a bias to the recommender system in response to a request by a provider of items to the online content provider system.

21. A method of measuring behavior of a recommender system, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:

(a) setting up a target preference to be used by the recommender system;

(b) causing the recommender system to interact with at least one proxy account and so as to generate a plurality of separate recommendations for a plurality of corresponding items;

(c) verifying whether a preference exhibited by the recommender system is within said target preference by examining said separate recommendations.

22. The method of claim 21, wherein said target preference is identified as part of a contractual arrangement between the online content service provider and a content provider which provides items to the online content service provider for distribution.

23. The method of claim 22, wherein said target preference is specified as an absolute number of recommendations, and/or a percentage of recommendations to be provided to subscribers.

24. The method of claim 23 wherein said target preference is limited to a particular time period.

25. The method of claim 21, wherein said preference is measured as part of an electronic audit of the performance of the online content service provider.

26. A method of evaluating recommendations made by a recommender system, which recommender is used for recommending items of interest to subscribers of an online content service provider, the method comprising the steps of:

(a) setting up a policy to be used in testing the recommender system;

(b) reviewing an actual set of recommendations made by the recommender system to a plurality of separate subscribers at the online content service provider;

(c) comparing said actual set of recommendations against a target set of recommendations associated with said policy;

(d) identifying a compliance level with said policy based on said actual set of recommendations.

27. The method of claim 26, wherein said target set of recommendations are based on items recommended by a second separate reference recommender system.

28. The method of claim 26, further including a step of:

tabulating the results of step (c) and posting the same on an Internet website, so that a comparison of recommendations made by a plurality of recommender systems and their associated compliance levels is identified.

29. The method of claim 28, wherein a relative bias rating is identified for each recommender system to indicate a deviation from an expected neutral recommendation.

30. The method of claim 28, wherein a relative ratio of relevant recommendations is identified for each recommender system.