RECOMMENDER SYSTEMS AND METHODS USING CASCADED MACHINE LEARNING MODELS

• US 20200134696A1
• Filed: 10/23/2019
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Amadeus (Amadeus Global Travel Distribution S.A.)
• Original Assignee: Amadeus (Amadeus Global Travel Distribution S.A.)
• Inventors: Lardeux, Benoit, Renaudie, David, Acuna Agost, Rodrigo Alejandro, Thomas, Eoin, Boudia, Mourad, Sane, Papa Birame
• Granted Patent: US 11,538,086 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0253 During e-commerce, i.e. online transactions
• CPC Class Codes: G06Q 30/0254 based on statistics
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06Q 30/0633 Lists, e.g. purchase orders, compilation or processing
• CPC Class Codes: H04L 67/306 User profiles

Abstract

Computer-implemented methods of providing personalized recommendations to a user of items available in an online system, and related systems. First-level features including context features are computed based upon context data. A first-level machine learning model is then evaluated using the first-level features to generate predictions of user behavior in relation to a plurality of individual items available via the online system. A list of proposed item recommendations is constructed based upon the predictions. Second-level features are computed based upon the context data and list features based upon the list of proposed item recommendations and the corresponding predictions generated by the first-level machine learning model. A second-level machine learning model is evaluated using the second-level features to generate a prediction of user behavior in relation to the list of proposed item recommendations. A personalized list of item recommendations is provided based upon the prediction generated by the second-level machine learning model.

Claims

1. A computer-implemented method of providing personalized recommendations to a user of items available in an online system, the method comprising:

receiving, via a communications channel, context data comprising user information;

computing a plurality of first-level features comprising context features based upon the context data;

evaluating a first-level machine learning model using the first-level features to generate predictions of user behavior in relation to a plurality of individual items available via the online system;

constructing a list of proposed item recommendations based upon the predictions generated by the first-level machine learning model;

computing a plurality of second-level features comprising context features based upon the context data and list features based upon the list of proposed item recommendations and the corresponding predictions generated by the first-level machine learning model;

evaluating a second-level machine learning model using the second-level features to generate a prediction of user behavior in relation to the list of proposed item recommendations; and

providing, via the communications channel, a personalized list of item recommendations based upon the prediction generated by the second-level machine learning model.

2. The method of claim 1 wherein the steps of constructing the list of proposed item recommendations, computing the plurality of second-level features, and evaluating the second-level machine learning model are repeated iteratively, wherein each iteration further comprises:

prior to computing the plurality of second-level features, swapping a proposed item recommendation in the list of proposed item recommendations based upon the predictions generated by the first-level machine learning model to generate a test list of proposed item recommendations;

after evaluating the second-level machine learning model to generate a prediction of user behavior in relation to the test list of proposed item recommendations, comparing the prediction of user behavior in relation to the test list of proposed item recommendations with a prediction computed on a prior iteration; and

in the event that the comparison determines that the prediction of user behavior in relation to the test list of proposed item recommendations is more favourable than the prediction computed on the prior iteration, replacing the list of proposed item recommendations with the test list of proposed item recommendations.

3. The method of claim 2 wherein the step of swapping a proposed item recommendation in the list of proposed item recommendations comprises selecting a swap that generates a previously untested list of recommendations from the plurality of individual items for which predictions were generated by the first-level machine learning model, and the iteration continues until a stopping condition is satisfied.

4. The method of claim 3 wherein the stopping condition is that all potential lists of recommendations have been exhausted.

5. The method of claim 2 wherein the step of swapping a proposed item recommendation in the list of proposed item recommendations comprises selecting a swap that enhances at least one of the second level features relative to an expected change in the prediction of user behavior in relation to the test list of proposed item recommendations, and the iteration continues until a stopping condition is satisfied.

6. The method of claim 5 wherein the stopping condition is that the prediction of user behavior in relation to the list of proposed item recommendations is not expected to become substantially more favourable by further iteration.

7. The method of claim 1 wherein the first-level machine learning model is trained using item features associated with each available item, and historical data of prior user behavior in response to presentations of personalized recommendations.

8. The method of claim 7 wherein the second-level machine learning model is trained using the historical data of prior user behavior in response to presentations of personalized recommendation and corresponding prediction data generated by evaluation of the first-level machine learning model.

9. The method of claim 1 wherein the list features include one or more aggregated list features selected from the group comprising:

average prediction of user behavior in relation to individual items included in the list of proposed item recommendations;

minimum item cost of individual items included in the list of proposed item recommendations;

number of items in the list of proposed item recommendations;

average value of a rating score associated with individual items in the list of proposed item recommendations;

range of costs of individual items included in the list of proposed item recommendations; and

range of values of a rating score associated with individual items in the list of proposed item recommendations.

10. The method of claim 1 where the prediction of user behavior comprises one or more of:

a probability of the user clicking on a link corresponding with an item presented in the personalized list of item recommendations; and

a probability of the user completing an online purchase of an item presented in the personalized list of item recommendations.

11. The method of claim 1 which is implemented in the context of an online travel service system, wherein the context data comprises information stored in a passenger name record (PNR) associated with an online flight booking, and the items comprise accommodation reservations.

12. A recommender system for providing personalized recommendations to a user of items available in an online system, the system comprising:

a computer-implemented single-item machine learning module comprising a first-level machine learning model configured to receive a plurality of first-level features and to generate a corresponding prediction of user behavior in relation to a plurality of individual items available via the online system;

a computer-implemented item list machine learning module comprising a second-level machine learning model configured to receive a plurality of second-level features and to generate a corresponding prediction of user behavior in relation to a list of proposed item recommendations; and

a computer-implemented list builder module configured to;

receive context data comprising user information;

compute the plurality of first-level features comprising context features based upon the context data;

obtain predictions of user behavior in relation to a plurality of individual items available via the online system from the single-item machine learning module based upon the first-level features;

construct the list of proposed item recommendations based upon the predictions obtained from the single-item machine learning module;

compute the plurality of second-level features comprising context features based upon the context data and list features based upon the list of proposed item recommendations and the corresponding predictions generated by the first-level machine learning model

obtain a prediction of user behavior in relation to the list of proposed item recommendations from the item list machine learning module based upon the second-level features; and

provide a personalized list of item recommendations based upon the prediction obtained from the item list machine learning module.

13. The recommender system of claim 12 wherein the list builder module is configured to iteratively construct the list of proposed item recommendations, compute the plurality of second-level features, and obtain the prediction of user behavior in relation to the list of proposed item recommendations, and is further configured, in each iteration, to:

prior to computing the plurality of second-level features, swap a proposed item recommendation in the list of proposed item recommendations based upon the predictions obtained from the single-item machine learning module, to generate a test list of proposed item recommendations;

after obtaining the prediction of user behavior in relation to the list of proposed item recommendations from the item list machine learning module, compare the prediction of user behavior in relation to the test list of proposed item recommendations with a prediction computed on a prior iteration; and

in the event that the comparison determines that the prediction of user behavior in relation to the test list of proposed item recommendations is more favourable than the prediction computed on the prior iteration, replace the list of proposed item recommendations with the test list of proposed item recommendations.

14. The recommender system of claim 12 wherein the first-level machine learning model is configured by training using item features associated with each available item, and historical data of prior user behavior in response to presentations of personalized recommendations.

15. The recommender system of claim 14 wherein the second-level machine learning model is configured by training using the historical data of prior user behavior in response to presentations of personalized recommendation and corresponding prediction data generated by evaluation of the first-level machine learning model.

16. A computing system for providing personalized recommendations to a user of items available in an online system, the computing system comprising:

a processor;

at least one memory device accessible by the processor; and

a communications interface accessible by the processor,

wherein the memory device contains body of program instructions which, when executed by the processor, cause the computing system to implement a method comprising steps of;

receiving, via the communications interface, context data comprising user information;

computing a plurality of first-level features comprising context features based upon the context data;

evaluating a first-level machine learning model using the first-level features to generate predictions of user behavior in relation to a plurality of individual items available via the online system;

constructing a list of proposed item recommendations based upon the predictions generated by the first-level machine learning model;

computing a plurality of second-level features comprising context features based upon the context data and list features based upon the list of proposed item recommendations and the corresponding predictions generated by the first-level machine learning model;

evaluating a second-level machine learning model using the second-level features to generate a prediction of user behavior in relation to the list of proposed item recommendations; and

providing, via the communications interface, a personalized list of item recommendations based upon the prediction generated by the second-level machine learning model.

17. A computer program product comprising a computer-readable medium having instructions stored thereon which, when executed by a processor implement a method of providing personalized recommendations to a user of items available in an online system, the method comprising:

receiving, via a communications channel, context data comprising user information;

computing a plurality of first-level features comprising context features based upon the context data;

evaluating a first-level machine learning model using the first-level features to generate predictions of user behavior in relation to a plurality of individual items available via the online system;

constructing a list of proposed item recommendations based upon the predictions generated by the first-level machine learning model;

computing a plurality of second-level features comprising context features based upon the context data and list features based upon the list of proposed item recommendations and the corresponding predictions generated by the first-level machine learning model;

evaluating a second-level machine learning model using the second-level features to generate a prediction of user behavior in relation to the list of proposed item recommendations; and

providing, via the communications channel, a personalized list of item recommendations based upon the prediction generated by the second-level machine learning model.