System, method and article of manufacture for using receiver operating curves to evaluate predictive utility

• US 5,842,199 A
• Filed: 10/18/1996
• Issued: 11/24/1998
• Est. Priority Date: 10/18/1996
• Status: Expired due to Term

• Current Assignee: Regents of The University of Minnesota (University of Minnesota)
• Original Assignee: Regents of The University of Minnesota (University of Minnesota)
• Inventors: Miller, Bradley N., Konstan, Joseph A., Riedl, John T.
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US08/733,806
• Time in Patent Office: 767 Days
• Field of Search: 707/102, 707/2, 707/205, 707/206, 707/10, 707/104, 707/4, 707/5, 707/200, 364/497, 607/5, 600/515, 705/27, 705/10, 705/8, 704/8, 348/1, 395/61
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

An electronic information system is provided for determining predictive utility of prediction techniques in ascertaining which items are valued. The information system includes an input mechanism for obtaining a set of measurements of actual value of items presented to a user. The items presented to the user are derived from a plurality of items by utilizing predictions from a prediction technique. A processor is operatively coupled to the input mechanism to generate a quality metric receiver operating curve (ROC) from the set of measurements which corresponds to a measure of effectiveness of the prediction technique in predicting which items are valued by the user. In addition, a method is provided for operating this electronic information system. Also, a storage device readable by a computer system for implementing this method is provided.

Claims

1. An electronic information system for determining predictive utility of recommendation techniques in ascertaining which items are preferred by a user, comprising:

(a) input means for obtaining a set of measurements of user preference of items recommended to a user, the items presented to the user being derived from a plurality of items by utilizing recommendations from an adaptive recommendation technique; and

(b) processing means, operatively coupled to the input means, generating a quality metric receiver operating curve (ROC) from the set of measurements which corresponds to a measure of effectiveness of the adaptive recommendation technique in predicting which items are preferred by the user.

2. The information system of claim 1 wherein:

(a) the input means comprises means for obtaining a first and a second set of measurements of actual preference of a first and a second set of items recommended to the user, respectively, each set of items presented to the user being derived from the plurality of items by utilizing recommendations from different adaptive recommendation techniques; and

(b) the processing means comprises means for generating a first and a second quality metric ROC from the first and the second set of measurements, respectively, and comparing the first and the second quality metric ROC to compare effectiveness of each different adaptive recommendation technique to each other in recommending items preferred by the user.

3. The information system of claim 2 wherein the processing means comprises means for selecting one of the different adaptive recommendation techniques for subsequent use based on the comparison of the first and the second quality metric ROC.

4. The information system of claim 3 wherein the different adaptive recommendation techniques utilize different underlying recommendation algorithms such that the comparison of the first and the second quality metric ROC selects which underlying recommendation algorithm is subsequently used.

5. The information system of claim 3 wherein the different adaptive recommendation techniques utilize a single underlying recommendation algorithm with different operating parameters such that the comparison of the first and the second quality metric ROC selects which operating parameters are subsequently used with the recommendation algorithm.

6. The information system of claim 5 wherein the operating parameters comprise partitioning schemes for the plurality of items.

7. The information system of claim 1 wherein the input means comprises means for generating a set of implicit measures from user input as the set of measurements of actual user preference of items recommended to the user.

8. The information system of claim 7 wherein the input means comprises means for generating the set of implicit measures based on criteria selected from a group consisting of:

user time spent viewing a particular item, user time spent listening to a particular item, and user time spent sampling a particular item.

9. The information system of claim 7 wherein the input means comprises means for generating the set of implicit measures based on subsequent actions taken, the subsequent actions taken being selected from a group consisting of:

the user viewing details of a particular item after viewing only a title, saving a particular item, forwarding a particular item, saving a return address of a particular item, deleting a particular item, replying to a particular item, and printing a particular item.

10. The information system of claim 1 wherein the input means and processing means together form a quality metric device, the information system further comprising a client device operatively coupled to the quality metric device, the client device comprising:

(a) user interface means for recommending the items to the user;

(b) user input means for receiving data input by the user from which a set of measurements of actual user preference of items recommended to a user is obtained;

(c) client processing means, operatively coupled to the user interface means and user input means, for receiving items to recommend to the user from an output mechanism of the quality metric device, obtaining the set of measurements of actual user preference of items based on the data input by the user, and providing the set of measurements of actual user preference of items to the input means of the quality metric device.

11. The information system of claim 10 wherein the quality metric device and the client device are located within a single apparatus.

12. The information system of claim 10 wherein the quality metric device and the client device each comprise means for communicating with each other through an interactive network such that the quality metric device and the client device can be remotely located from one another in separate apparatuses.

13. A method of operating an electronic information system in order to determine predictive utility of recommendation techniques in ascertaining which items are preferred by a user, comprising:

(a) deriving particular items from a plurality of items by utilizing recommendations from an adaptive recommendation technique;

(b) presenting the particular items to a user on a user interface device;

(c) obtaining a set of measurements of actual value of the particular items presented to a user through an input device; and

(d) generating a quality metric receiver operating curve (ROC) from the set of measurements which corresponds to a measure of effectiveness of the adaptive recommendation technique in predicting which items are preferred by the user.

14. The method of claim 13:

(a) wherein the deriving step comprises deriving a first and a second set of particular items from a plurality of items by utilizing recommendations from different adaptive recommendation techniques;

(b) wherein the presenting step comprises presenting the first and the second set of particular items to the user;

(c) wherein the obtaining step comprises obtaining a first and a second set of measurements of actual user preference of the first and the second set of items presented to the user, respectively;

(d) wherein the generating step comprises generating a first and a second quality metric ROC from the first and the second set of measurements, respectively; and

(e) further comprises a step of comparing the first and the second quality metric ROC to compare effectiveness of each different adaptive recommendation technique to each other in predicting items preferred by the user.

15. The method of claim 14 further comprising a step of selecting one of the different adaptive recommendation techniques for subsequent use based on the comparing step.

16. The method of claim 15 wherein the different adaptive recommendation techniques utilize different underlying recommendation algorithms such that the selecting step comprises selecting which underlying recommendation algorithm is subsequently used.

17. The method of claim 15 wherein the different adaptive recommendation techniques utilize a single underlying recommendation algorithm with different operating parameters such that the selecting step comprises selecting which operating parameters are subsequently used with the recommendation algorithm.

18. The method of claim 17 wherein the operating parameters comprise partitioning schemes for the plurality of items.

19. The method of claim 18 wherein the obtaining step comprises generating a set of implicit measures from user input through the input device as the set of measurements of actual user preference of items presented to the user.

20. The method of claim 19 wherein the obtaining step comprises deriving the implicit measure based on criteria selected from a group consisting of:

user time spent viewing a particular item, user time spent listening to a particular item, and user time spent sampling a particular item.

21. The method of claim 19 wherein the obtaining step comprises deriving the implicit measure based on subsequent actions taken, the subsequent actions taken being selected from a group consisting of:

a user viewing details of a particular item after viewing only a title, saving a particular item, forwarding a particular item, saving a return address of a particular item, deleting a particular item, replying to a particular item, and printing a particular item.

22. A program storage device readable by a computer system tangibly embodying a program of instructions executable by the computer system to perform a method of operating an electronic information system in order to determine predictive utility of recommendation techniques in ascertaining which items are preferred by a user, the method comprising steps of:

(a) deriving particular items from a plurality of items by utilizing recommendations from an adaptive recommendation technique;

(b) presenting the particular items to a user on a user interface device;

(c) obtaining a set of measurements of actual user preference of the particular items presented to a user through an input device; and

(d) generating a quality metric receiver operating curve (ROC) from the set of measurements which corresponds to a measure of effectiveness of the adaptive recommendation technique in predicting which items are preferred by the user.

23. The program storage device of claim 22 wherein:

(a) the method deriving step comprises deriving a first and a second set of particular items from a plurality of items by utilizing recommendations from different adaptive recommendation techniques;

(b) the method presenting step comprises presenting the first and the second set of particular items to the user;

(c) the method obtaining step comprises obtaining a first and a second set of measurements of actual user preference of the first and the second set of items presented to the user, respectively;

(d) the method generating step comprises generating a first and a second quality metric ROC from the first and the second set of measurements, respectively; and

(e) the method further comprises a step of comparing the first and the second quality metric ROC to compare effectiveness of each different recommendation technique to each other in predicting items preferred by the user.

24. The program storage device of claim 23 wherein the method further comprises a step of selecting one of the different recommendation techniques for subsequent use based on the comparing step.