Search result ranking using machine learning

• US 8,924,314 B2
• Filed: 08/04/2011
• Issued: 12/30/2014
• Est. Priority Date: 09/28/2010
• Status: Active Grant

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: Kulkarni, Parashuram, Abbas, Abdelhalim, Mathieson, Michael, Hua, Jingzhou, Degenhardt, Jon, Natarajan, Ramakrishnan
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): FINK, THOMAS ANDREW
• Application Number:
  US13/198,560
• Publication Number:
  US 20120078825A1
• Time in Patent Office: 1,244 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Various embodiments include systems and methods for search result ranking using machine learning. A goal model can be created using machine learning. Responsive to a search query, a plurality of data factors can be inputted into the goal model to create a model output. Search results can be presented to a user based on the model output.

Claims

1. A method comprising:

implementing a goal model for a first goal from a plurality of goals using machine learning, the goal model being at least one of a neural network or an expert system;

factorizing raw data to a set of data factors for the first goal, the raw data including query data from a user search query and at least one of relevance of an item title, temporal data, transaction data, impressions of item listings, item demand, or item supply;

assigning a plurality of impact scores to each of the set of data factors, a first impact score of the plurality of impact scores corresponding to the first goal and a second impact score of the plurality of impact scores corresponding to a second goal in the plurality of goals, the plurality of impact scores respectively measuring the degree to which changes in a data factor influence a corresponding model'"'"'s output, the first impact score corresponding to revenue generation and the second impact score corresponding to cross product marketing;

ranking the set of data factors based on the first impact score;

selecting a plurality of data factors from the set of data factors based on the ranking, the plurality of data factors being a proper subset of the set of data factors;

modifying the values of the plurality of data factors by respective weights prior to being inputted into the goal model;

inputting, responsive to the search query, the plurality of data factors into the goal model to;

select search results from a database using the goal model; and

create a model output, the model output including a ranking for each result in the search results; and

presenting, to a user, an ordered list of search results based on the ranking for each result in the search results from the model output.

2. The method of claim 1, wherein implementing a goal model using machine learning includes training a machine learning system towards a goal, wherein the machine learning system is the at least one of the neural network or the expert system.

3. The method of claim 2, wherein training the machine learning system towards a goal includes:

identifying the goal;

identifying an input dataset relevant to the goal;

inputting the input dataset to the machine learning system;

observing an output of the machine learning system to identify errors in the output based on the goal; and

responsive to receiving the errors in the output, iteratively refining the output until a predetermined condition is met.

4. The method of claim 3, wherein iteratively refining the output includes:

adjusting an internal model of the machine learning system based on the errors in the output;

re-inputting the input dataset to the machine learning system; and

observing intermediate refined output of the machine learning system to identify further errors in the intermediate refined output.

5. The method of claim 3, wherein an input dataset comprises at least one data factor.

6. The method of claim 5, wherein the data factor includes user data from an online publisher.

7. The method of claim 5, wherein the data factor includes transaction data from an online publisher.

8. The method of claim 5, wherein the data factor includes publication data from an online publisher.

9. The method of claim 5, wherein the data factor comprises aggregated data from an online publisher.

10. The method of claim 5, wherein the data factor comprises data from the search query.

11. The method of claim 1, wherein at least one data factor from the plurality of data factors is computed prior to the search query.

12. The method of claim 1, wherein at least one data factor from the plurality of data factors is computed following the search query.

13. A system comprising:

execution hardware;

a machine learning module, performed by the execution hardware, configured to implement a goal model for a first goal from a plurality of goals, the goal model being at least one of a neural network or an expert system;

a factorization module, performed by the execution hardware, configured to;

factorize raw data to a set of data factors for the first goal, the raw data including query data from a user search query and at least one of relevance of an item title, temporal data, transaction data, impressions of item listings, item demand, or item supply;

assign a plurality of impact scores to each of the set of data factors, a first impact score of the plurality of impact scores corresponding to the first goal and a second impact score of the plurality of impact scores corresponding to a second goal in the plurality of goals, the plurality of impact scores respectively measuring the degree to which changes in a data factor influence a corresponding model'"'"'s output, the first impact score corresponding to revenue generation and the second impact score corresponding to cross product marketing;

rank the set of data factors based on the first impact score;

select a plurality of data factors from the set of data factors based on the ranking; and

modify the values of the plurality of data factors by respective weights prior to being inputted into the goal model; and

a goal module, performed by the execution hardware, configured to;

input, responsive to the search query, the plurality of data factors into the goal model to;

select search results from a database using the goal model; and

create a model output, the model output including a ranking for each result in the search results; and

present, to a user, an ordered list of search results based on the ranking for each result in the search results from the model output.

14. The system of claim 13, comprising a training module configured to train a machine learning system towards a goal, wherein the machine learning system is the at least one of the neural network or the expert system.

15. The system of claim 14, wherein the training module if configured to:

receive identification of the goal;

receive identification of an input dataset relevant to the goal;

input the input dataset to the machine learning system;

observe an output of the machine learning system and identify errors in the output based on the goal; and

responsive to receiving the errors in the output, iteratively refine the output until a predetermined condition is met.

16. The system of claim 15, wherein to iteratively refine the output the training module is configured to:

adjust an internal model of the machine learning system based on the errors in the output;

re-input the input dataset to the machine learning system; and

observe an intermediate refined output of the machine learning system and identify further errors in the intermediate refined output.

17. The system of claim 15, wherein an input dataset comprises at least one data factor.

18. The system of claim 17, wherein the data factor includes user data from an online publisher.

19. The system of claim 17, wherein the data factor includes transaction data from an online publisher.

20. The system of claim 17, wherein the data factor includes publication data from an online publisher.

21. The system of claim 17, wherein the data factor comprises aggregated data from an online publisher.

22. The system of claim 17, wherein the data factor comprises data from the search query.

23. The system of claim 13, wherein at least one data factor from the plurality of data factors is computed prior to the search query.

24. The system of claim 13, wherein at least one data factor from the plurality of data factors is computed following the search query.

25. A non-transitory machine readable medium including instructions that, when executed, cause a machine to perform operations including:

implementing a goal model for a first goal from a plurality of goals using machine learning, the goal model being at least one of a neural network or an expert system;

factorizing raw data to a set of data factors for the first goal, the raw data including query data from a user search query and at least one of relevance of an item title, temporal data, transaction data, impressions of item listings, item demand, or item supply;

assigning a plurality of impact scores to each of the set of data factors, a first impact score of the plurality of impact scores corresponding to the first goal and a second impact score of the plurality of impact scores corresponding to a second goal in the plurality of goals, the plurality of impact scores respectively measuring the degree to which changes in a data factor influence a corresponding model'"'"'s output, the first impact score corresponding to revenue generation and the second impact score corresponding to cross product marketing;

ranking the set of data factors based on the first impact score;

selecting a plurality of data factors from the set of data factors based on the ranking, the plurality of data factors being a proper subset of the set of data factors;

modifying the values of the plurality of data factors by respective weights prior to being inputted into the goal model;

inputting, responsive to the search query, the plurality of data factors into the goal model to;

select search results from a database using the goal model; and

create a model output, the model output including a ranking for each result in the search results; and

presenting, to a user, an ordered list of search results based on the ranking for each result in the search results from the model output.