Identifying salient items in documents

• US 9,251,473 B2
• Filed: 03/13/2013
• Issued: 02/02/2016
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Gamon, Michael, Pantel, Patrick, Song, Xinying, Yano, Tae, Apacible, Johnson Tan
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Gonzales, Vincent
• Application Number:
  US13/798,198
• Publication Number:
  US 20140279730A1
• Time in Patent Office: 1,056 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9035 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A set of representations of item-page pairs of items and respective web pages that include the respective items is obtained, each representation including feature function values indicating weights associated with features of associated web pages, the features including page classification features. An annotated set of labeled training data that is annotated with salience annotation values of items for respective web pages that include the items is obtained. The salience annotation values are determined based on a soft function, by determining a first count of a total number of user queries associated with corresponding visits to the respective web pages, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with user queries that include the item, the subset included in the corresponding visits. Models are trained using the annotated set.

Claims

1. A system comprising:

a device that includes at least one processor, and a computer readable storage medium storing instructions for execution by the at least one processor, for implementing a salient item identification engine that;

obtains query data and corresponding click data that indicates web pages visited, in association with respectively corresponding user queries, based on information mined from a web search log; and

determines a salience annotation value of an item for respective ones of the web pages, based on determining a first count of a total number of the user queries that are associated with one or more corresponding visits to the respective ones of the web pages, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with a group of the user queries that include the item, the subset included in the one or more corresponding visits.

2. The system of claim 1, wherein:

determining the ratio includes determining the ratio in accordance with

3. The system of claim 1,

wherein the salient item identification engine obtains a model that is trained with training data that is annotated with labels that are based on a plurality of salience annotation values that include the salience annotation value.

4. The system of claim 1, wherein:

the salient item identification engine determines the salience annotation value of the item for respective ones of the web pages, based on determining a set of the user queries that are associated with one or more corresponding visits to the respective web page, wherein the group of the user queries that include the item includes one or more of;

a group of the user queries that substantially match the item, or exactly match the item, and that are included in the set, or

a group of the user queries that include the item as a proper substring, and that are included in the set.

5. The system of claim 1, wherein:

the salient item identification engine determines a plurality of salience annotation values of a plurality of respective items for respective ones of the web pages,

and obtains labeled training data for model training, based on the determined plurality of salience annotation values.

6. The system of claim 5,

wherein the salient item identification engine obtains a supervised machine-learned model that is trained offline with the labeled training data.

7. The system of claim 5,

wherein the salient item identification engine obtains feature function values for respective item-page pairs of the items and associated web pages, the feature function values indicating weights associated with one or more features of the associated web pages, wherein:

the features include one or more page classification features associated with the associated web pages.

8. The system of claim 7, wherein:

the features include one or more of;

corpus features,

Uniform Resource Locator (URL) features,

length features,

web page structural features,

web graph features associated with the associated web pages,

user profile features,

user interest features,

position features, or

page segmentation features.

9. The system of claim 7,

wherein the salient item identification engine obtains a model that is trained with training data that includes representations of the item-page pairs that are annotated with labels that are based on a plurality of salience annotation values that include the salience annotation value, wherein the representations of the item-page pairs each include a plurality of the feature function values.

10. The system of claim 9,

wherein the salient item identification engine obtains salience scores associated with respective ones of the items and associated ones of the web pages, the salience scores indicating a measure of salience of the respective items to the respective associated web pages, based on the trained model.

11. The system of claim 10, wherein:

the salient item identification engine obtains the salience scores based on data obtained from the supervised machine-learned model and the feature function determination component.

12. A method comprising:

obtaining a first set of representations of a plurality of item-document pairs of items and respective documents that include the respective items, each of the representations including a plurality of feature function values indicating weights associated with one or more features of the associated documents, the features including one or more document classification features associated with the associated documents;

obtaining an annotated set of labeled training data that is annotated with a plurality of salience annotation values of a plurality of respective items for respective documents that include content that includes the respective items, the salience annotation values determined based on a soft function, based on determining a first count of a total number of user queries that are associated with one or more corresponding visits to the respective documents, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with a group of the user queries that include the item, the subset included in the one or more corresponding visits; and

initiating, via a device processor, training of one or more models based on the annotated set.

13. The method of claim 12, further comprising:

obtaining salience scores associated with respective ones of the items and associated ones of the documents, the salience scores indicating a measure of salience of the respective items to the respective associated documents, based on the trained one or more models.

14. The method of claim 12, wherein:

training of the one or more models includes training of the one or more models based on a learning algorithm that includes one or more of;

regression,

ranking,

binary classification,

boosted decision trees,

support vector machines,

linear regression,

logistic regression, or

neural networks.

15. The method of claim 12, wherein:

the documents include web pages, and

the document classification features include one or more of;

page categories of the web pages,

junk page scores of the web pages,

inlink page scores of the web pages, or

spam page confidence values of the web pages.

16. The method of claim 12, wherein:

the documents include web pages, and

the features include one or more of;

corpus features,

Uniform Resource Locator (URL) features,

length features,

web page structural features,

web graph features associated with the associated web pages,

user profile features,

user interest features,

position features, or

page segmentation features.

17. The method of claim 12, wherein:

the documents include web pages, and

the features include one or more of;

corpus features, the corpus features including an indication that a plurality of items included in a first group of the web pages is included in a second group of the web pages in a top domain of web pages;

Uniform Resource Locator (URL) features, the URL features including a top level domain indication;

web page structural features, the web page structural features including one or more of;

indicators of items included in titles of associated web pages,

indicators of items included in meta keywords of associated web pages,

indicators of items included in visual titles of associated web pages,

indicators of items included in bold font of associated web pages,

indicators of items included in emphasized font of associated web pages,

indicators of items included in italicized font of associated web pages,

indicators of items included in URLs of associated web pages,

indicators of items included in domain parts URLs of associated web pages, or

indicators of items included in suffix portions URLs of associated web pages;

or

web graph features associated with the associated web pages, the web graph features including a normalized frequency value indicating a frequency of occurrence of items included in anchors of web pages.

18. A computer program product comprising a hardware computer-readable storage medium storing including executable instructions that, when executed, cause at least one data processing apparatus to:

obtain a first set of representations of a plurality of item-page pairs of items and respective web pages that include the respective items, each of the representations including a plurality of feature function values indicating weights associated with one or more features of the associated web pages, the features including one or more page classification features associated with the associated web pages; and

obtain an annotated set of labeled training data that is annotated with a plurality of salience annotation values of a plurality of the respective items for respective web pages that include content that includes the respective items, the salience annotation values determined based on a soft function, based on determining a first count of a total number of user queries that are associated with one or more corresponding visits to the respective web pages, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with a group of the user queries that include the item, the subset included in the one or more corresponding visits; and

initiate training of one or more models based on the annotated set.

19. The computer program product of claim 18, wherein the executable instructions, when executed, cause the at least one data processing apparatus to:

obtain salience scores associated with respective ones of the items and associated ones of the web pages, the salience scores indicating a measure of salience of the respective items to the respective associated web pages, based on the trained one or more models.

20. The computer program product of claim 19, wherein the executable instructions, when executed, cause the at least one data processing apparatus to:

obtain a ranked list of the salience scores, the ranked list ranked in accordance with an ordering of the salience scores indicating measures of salience of one of the items to one or more of the web pages that include the one of the items.