Evaluating recommendations by determining user actions, and performance values pertaining to lists of recommendations

• US 8,250,012 B1
• Filed: 03/28/2008
• Issued: 08/21/2012
• Est. Priority Date: 03/30/2007
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Whitman, Ronald M., Yi, Jin Y., Turner, Wesley M.
• Primary Examiner(s): VINCENT, DAVID ROBERT
• Application Number:
  US12/058,625
• Time in Patent Office: 1,607 Days
• Field of Search: 706/20, 706/45, 706/47, 706/48, 4554561-457, 342/147, 342/450, 342/451, 705/26, 705/14.37, 705/14.38, 705/14.41, 705/14.53, 705/14.58, 705/14.65, 705/14.67, 348/139, 348/143, 348150-159
• US Class Current: 706/45
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

Generally described, embodiments of the present disclosure are directed toward the identification of items for inclusion in a recommendations list that may be displayed concurrently with an item selected by a user or users. The recommended items may be items that are of potential interest to the user and/or may be items that are related to the selected item. More specifically, embodiments of the present disclosure provide a method and system for generating one or more recommendations lists, providing those lists to users, evaluating users'"'"' interactions with those lists, and modifying the engines or techniques used to identify items that are to be included in the recommendations lists.

Claims

1. A system comprising:

a test recommendation engine for processing a selected item as a first input value and generating a first set of recommendations based on the processing of the first input value and a first set of additional input values, the first set of recommendations including a plurality of items distinct from the selected item;

a control recommendation engine for processing the selected item as a second input value and generating a second set of recommendations based on the processing of the second input value and a second set of additional input values, the second set of recommendations including a plurality of items distinct from the selected item, wherein the first set of additional input values and the second set of additional input values are different sets of input values; and

an analyzing module utilized for;

determining a first set of end user actions, wherein each of the first set of end user actions corresponds to an action taken by a first set of end users as a response to being presented with the first set of recommendations;

determining a performance value of the generated first set of recommendations based on the determined first set of end user actions, the performance value corresponding to an assessment of the desirability of each of the first set of end user actions,

determining a second set of end user actions, wherein each of the second set of end user actions corresponds to an action taken by a second set of end users as a response to being presented with the second set of recommendations;

determining a performance value of the generated second set of recommendations based on the determined second set of end user actions, the performance value corresponding to an assessment of the desirability of each of the second set of end user actions,

comparing the determined performance values of the determined first and second sets of recommendations; and

generating feedback based on said comparison, wherein the feedback identifies at least one of a recommendation engine or additional input values corresponding to the higher performance value as more desirable.

2. The system as recited in claim 1, wherein the test recommendation engine and the control recommendation engine are the same.

3. The system as recited in claim 1, wherein each end user action in a set of end user actions is associated with a specific response value, and wherein the performance value associated with the first and second of recommendations corresponds to a sum of each of the response values of each of the end user actions detected with respect to the first and second set of recommendations.

4. The system as recited in claim 1, wherein at least one end user action is associated with a negative value.

5. The system as recited in claim 1, wherein the feedback generated by the analyzing module corresponds to a selection of either the test recommendation engine or the control recommendation engine based on the comparison of the performance values associated with the first and second of recommendations.

6. The system as recited in claim 1, wherein the feedback generated by the analyzing module includes the analyzing module modifying at least one characteristic associated with the test recommendation engine.

7. The system as recited in claim 6, wherein the modifying at least one characteristic associated with the test recommendation engine includes modifying at least one characteristic associated with the test recommendation engine to match a corresponding characteristic associated with the control recommendation engine.

8. The system as recited in claim 1, wherein the analyzing module causes the display of the first set of recommendations and the second set of recommendations and wherein determining at least one of the first set of end user actions or the second set of end users actions comprises monitoring end user activity with regard to the displayed first and second set of recommendations.

9. The system as recited in claim 1, wherein the analyzing module causes the display of an interleaved list including items from the first set of recommendations and the second set of recommendations and wherein determining at least one of the first set of end user actions or the second set of end users actions comprises monitoring end user activity with regard to the displayed interleaved list.

10. The system of claim 1, wherein an additional input value is at least one of:

(1) an end user'"'"'s prior purchase tendencies, (2) what other end users have purchased when purchasing the selected item, (3) items that are complimentary to the selected item, (4) items that have been viewed by other end users that purchased the selected item, (5) the end user'"'"'s profile, (6) the selected item'"'"'s profile, and (7) a wishlist of the end user.

11. The system of claim 1, wherein end users actions elicited by a set of recommendations include at least one of (1) selecting an item within the set of recommendations, (2) purchasing an item within the set of recommendations, and (3) selecting an item not within the set of recommendations.

12. A method comprising:

determining a first set of end user actions corresponding to actions taken by a first set of end users as a response to being presented with a first set of recommendations generated by a first recommendation engine,

wherein the first set of recommendations are generated based on a selected item and a first set of additional input values, and

wherein the first set of recommendations includes a plurality of items distinct from the selected item;

determining a performance value of the generated first set of recommendations based on the determined first set of end user actions, the performance value corresponding to an assessment of the desirability of each of the first set of end user actions,

determining a second set of end user actions corresponding to actions taken by a second set of end users as a response to being presented with a second set of recommendations generated by a second recommendation engine,

determining a second set of end user actions corresponding to actions taken by a second set of end users as a response to being presented with a second set of recommendations generated by a second recommendation engine,

wherein the second set of recommendations are generated based on the selected item and a second set of additional input values,

wherein the first set of additional input values and the second set of additional input values are different sets of input values, and

wherein the first set of recommendations includes a plurality of items distinct from the selected item;

determining a performance value of the generated second set of recommendations based on the determined second set of end user actions, the performance value corresponding to an assessment of the desirability of each of the second set of end user actions,

comparing the determined performance values of the determined first and second sets of recommendations; and

generating feedback based on said comparison, wherein the feedback reflects an assessment of the performance of at least one of a recommendation engine or a set of additional input values according to the specified desirable actions elicited by the at least one of a recommendation engine or a set of additional input values.

13. The method as recited in claim 12, wherein each end user action in a set of end user actions is associated with a specific response value, and wherein the performance value associated with the first and second of recommendations corresponds to a sum of each of the response values of each of the end user actions detected with respect to the first and second set of recommendations.

14. The method as recited in claim 13, wherein at least one end user action is associated with a negative value.

15. The method as recited in claim 12, wherein the generating feedback includes the modifying at least one characteristic associated with the first recommendation engine.

16. The method as recited in claim 15, wherein the modifying at least one characteristic associated with the first recommendation engine includes modifying at least one characteristic associated with the first recommendation engine to match a corresponding characteristic associated with the second recommendation engine.

17. The method as recited in claim 12, wherein determining at least one of the first set of end user actions or the second set of end users actions comprises monitoring end user activity with regard to a display of at least one of the first and second set of recommendations.

18. The method as recited in claim 12 further comprising displaying an interleaved list including items from the first set of recommendations and the second set of recommendations and wherein determining at least one of the first set of end user actions or the second set of end users actions comprises monitoring end user activity with regard to the displayed interleaved list.