ENTERPRISE RELEVANCY RANKING USING A NEURAL NETWORK

• US 20090106223A1
• Filed: 10/18/2007
• Published: 04/23/2009
• Est. Priority Date: 10/18/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Meyerzon, Dmitriy, Shnitko, Yauhen, Taylor, Michael James, Burges, Chris J.C.
• Granted Patent: US 7,840,569 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 3/02 Neural networks

Abstract

A neural network is used to process a set of ranking features in order to determine the relevancy ranking for a set of documents or other items. The neural network calculates a predicted relevancy score for each document and the documents can then be ordered by that score. Alternate embodiments apply a set of data transformations to the ranking features before they are input to the neural network. Training can be used to adapt both the neural network and certain of the data transformations to target environments.

Claims

1. ) A method of determining a relevancy rank ordering score for a plurality of documents comprising:

(a) identifying a finite set of candidate documents;

(b) for each of the candidate documents;

(i) obtaining a plurality of ranking features associated with the candidate document, at least one of the ranking features selected from the group consisting of BM25, click distance, URL depth, file type, and language; and

(ii) using a neural network to calculate a relevancy score from the ranking features,

2. ) The method of claim 1 further comprising transforming the ranking features prior to calculating the relevancy score.

3. ) The method of claim 2 wherein at least one of the transformations is of the form

4. ) The method of claim 3 wherein at least one of the saturation values is adjusted during training of the neural network.

5. ) The method of claim 2 wherein at least one of the transformations comprises mapping each value of an enumerated data type to a discrete binary value.

6. ) The method of claim 5 wherein the neural network accepts each discrete binary value as a separate input and applies a separate trainable weight to each of the separate inputs.

7. ) The method of claim 1 wherein the BM25 feature comprises the BM25G formula which uses at least one property selected from the group consisting of body, title, author, anchor text, URL, and extracted title.

8. ) A system for generating a relevancy ranking for documents comprising:

(a) a module which identifies a set of candidate documents and makes available a plurality of common ranking features for each of the candidate documents;

(b) a data module comprising weights for each of the ranking features; and

(c) a ranking module comprising at least one input transformation and a neural network, wherein the ranking module accepts the ranking features for each of the candidate documents individually, applies the input transformation to at least one ranking feature, provides the ranking features, including the transformed feature, to the neural network which applies the ranking feature weights and calculates a relevancy score for each of the candidate documents.

9. ) The relevancy ranking system of claim 9 wherein at least one of the ranking features is selected from the group consisting of BM25, click distance, URL depth, file type, and language

11. ) The relevancy ranking system of claim 9 wherein the data module further comprises at least one transformation constant and the input transformation utilizes the transformation constant.

12. ) The relevancy ranking system of claim 9 wherein at least one of the input transformations is of the form

14. ) The relevancy ranking system of claim 9 wherein at least one of the input transformations comprises mapping each value of an enumerated data type to a discrete binary value and the neural network accepts each discrete binary value as a separate ranking feature and applies a separate ranking feature weight to each of the separate ranking features.

10. ) The relevancy ranking system of claim 10 wherein the BM25 feature comprises the BM25G formula which uses at least one property selected from the group consisting of body, title, author, anchor text, URL, and extracted title.

13. ) The relevancy ranking system of claim 13 wherein at least one of the configurable constants is adjusted during training of the neural network.

15. ) A method of rank ordering a plurality of documents by relevancy comprising:

(a) identifying a finite set of candidate documents;

(b) for each of the candidate documents;

(i) obtaining a plurality of ranking features associated with the candidate document, at least one of the ranking features selected from the group consisting of BM25, click distance, URL depth, file type, and language;

(ii) applying a transformation to at least one of the ranking features wherein the transformation comprises a constant which is configurable;

(iii) using a neural network to calculate a relevancy score from the ranking features; and

(c) ordering the candidate documents by the calculated relevancy scores.

16. ) The rank ordering method of claim 16 wherein at least one of the transformations is of the form

17. ) The rank ordering method of claim 17 further comprising at least one ranking feature transformation comprising mapping each value of an enumerated data type to a discrete binary value and wherein the neural network accepts each discrete binary value as a separate ranking feature and applies a separate trainable weight to each of the discrete binary values.

18. ) The rank ordering method of claim 18 wherein the ranking features comprise at least BM25, click distance, URL depth, file type, and language and wherein click distance and URL depth are transformed as specified in claim 17 and wherein file type, and language are transformed as specified in claim 18.

19. ) The method of claim 19 wherein the BM25 feature comprises the BM25G formula which uses the properties of body, title, author, anchor text, URL, and extracted title.