TECHNIQUES FOR EVALUATING RECOMMENDATION SYSTEMS

• US 20090319330A1
• Filed: 06/18/2008
• Published: 12/24/2009
• Est. Priority Date: 06/18/2008
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Yue, Moore, Michael D., Crivat, Ioan Bogdan, MacLennan, C. James
• Application Number:
  US12/141,105
• Publication Number:
  US 20090319330A1
• Time in Patent Office: Days
• US Class Current: 705/10
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0202 Market predictions or forecasting for commercial activities

Abstract

Various technologies and techniques are disclosed for calculating and evaluating the behavior of recommendation systems. Accuracy measures are computed for a plurality of items in a real recommendation system, an ideal recommendation system, and a popularity-based baseline recommendation system. The accuracy measures for the plurality of items are presented to a user so the user can evaluate a performance of the real recommendation system in comparison to the ideal recommendation system and the popularity-based baseline recommendation system. The accuracy measures can be presented in an interactive graph.

Claims

1. A method for computing metrics that can be used for analyzing a performance of a recommendation system comprising the steps of:

computing accuracy measures for a plurality of items in a real recommendation system, an ideal recommendation system, and a popularity-based baseline recommendation system; and

presenting the accuracy measures for the plurality of items to a user so the user can evaluate a performance of the real recommendation system in comparison to the ideal recommendation system and the popularity-based baseline recommendation system.

2. The method of claim 1, wherein the accuracy measures are presented in an interactive graph.

3. The method of claim 2, wherein the interactive graph is sorted in descending order by the accuracy measures.

4. The method of claim 2, wherein an area bounded by a curve and the vertical and horizontal axis on the interactive diagram represents a measure of an overall performance of a respective recommendation system that generated the curve.

5. The method of claim 2, wherein the user can select a respective item in the interactive graph to view additional details regarding the respective item.

6. The method of claim 1, wherein the accuracy measures can be analyzed by the user to determine how well the real recommendation system is performing in comparison to the ideal recommendation system and the comparison recommendation system.

7. The method of claim 1, wherein after the accuracy measures are computed, any multiple measures that result are aggregated for a respective recommendation system.

8. The method of claim 1, wherein the accuracy measures are calculated over a holdout data set that is based upon prior sales across all customers.

9. The method of claim 1, wherein the accuracy measures are calculated over a holdout data set that is based upon new purchases by existing customers.

10. A computer-readable medium having computer-executable instructions for causing a computer to perform steps comprising:

computing accuracy measures for a plurality of items in a real recommendation system, an ideal recommendation system, and a popularity-based baseline recommendation system; and

generating a graph that displays the accuracy measures for the plurality of items in the real recommendation system, the ideal recommendation system, and the popularity-based baseline recommendation system.

11. The computer-readable medium of claim 10, further having computer-executable instructions for causing a computer to perform steps comprising:

displaying the accuracy measures in a descending order.

12. The computer-readable medium of claim 10, further having computer-executable instructions for causing a computer to perform steps comprising:

computing accuracy measures for a plurality of items in a comparison recommendation system and including the accuracy measures for the comparison recommendation system in the graph.

13. The computer-readable medium of claim 10, further having computer-executable instructions operable to cause a computer to perform steps comprising:

computing a lift measure for the ideal recommendation system, for the real recommendation system, and for the popularity-based baseline recommendation system.

14. The computer-readable medium of claim 13, further having computer-executable instructions operable to cause a computer to perform steps comprising:

graphically displaying the coverage measure for the ideal recommendation system, for the real recommendation system, and for the popularity-based baseline recommendation system for a user to analyze.

15. The computer-readable medium of claim 10, further having computer-executable instructions operable to cause a computer to perform steps comprising:

computing a coverage measure for the ideal recommendation system, for the real recommendation system, and for the popularity-based baseline recommendation system.

16. The computer-readable medium of claim 15, further having computer-executable instructions operable to cause a computer to perform steps comprising:

graphically displaying the coverage measure for the ideal recommendation system, for the real recommendation system, and for the popularity-based baseline recommendation system for a user to analyze.

17. A method for generating a graph that can be used by a user to conduct a performance evaluation of a real recommendation system comprising the steps of:

computing accuracy measures for a plurality of items in a real recommendation system, an ideal recommendation system, and a popularity-based baseline recommendation system;

generating a graph in descending order by the accuracy measures for the plurality of items in the real recommendation system, the ideal recommendation system, and the popularity-based baseline recommendation system; and

displaying the graph so a user can interact with the graph to conduct a performance evaluation of the real recommendation system in comparison to the ideal recommendation system and the popularity-based baseline recommendation system

18. The method of claim 17, wherein accuracy measures are also computed for a comparison recommendation system.

19. The method of claim 18, wherein the performance evaluation enables the user to determine that the comparison recommendation system is performing better than the real recommendation system.

20. The method of claim 17, wherein the performance evaluation enables the user to determine that a certain item should be kept in inventory because the item is generating profits based upon recommendations from the real recommendation system.