IDENTIFYING SALIENT ITEMS IN DOCUMENTS

• US 20140279730A1
• Filed: 03/13/2013
• Published: 09/18/2014
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gamon, Michael, Pantel, Patrick, Apacible, Johnson Tan, Song, Xinying, Yano, Tae
• Granted Patent: US 9,251,473 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 16/9035 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A set of representations of item-page pairs of items and respective web pages that include the respective items is obtained, each representation including feature function values indicating weights associated with features of associated web pages, the features including page classification features. An annotated set of labeled training data that is annotated with salience annotation values of items for respective web pages that include the items is obtained. The salience annotation values are determined based on a soft function, by determining a first count of a total number of user queries associated with corresponding visits to the respective web pages, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with user queries that include the item, the subset included in the corresponding visits. Models are trained using the annotated set.

Claims

1. A system comprising:

a device that includes at least one processor, the device including a salient item identification engine comprising instructions tangibly embodied on a computer readable storage medium for execution by the at least one processor, the salient item identification engine including;

a log data acquisition component configured to obtain query data and corresponding click data that indicates web pages visited, in association with respectively corresponding user queries, based on information mined from a web search log; and

a soft labeling component configured to determine a salience annotation value of an item for respective ones of the web pages, based on determining a first count of a total number of the user queries that are associated with one or more corresponding visits to the respective ones of the web pages, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with a group of the user queries that include the item, the subset included in the one or more corresponding visits.

2. The system of claim 1, wherein:

the soft labeling component is configured to determine the ratio in accordance with

3. The system of claim 1, further comprising:

a model determination component configured to obtain a model that is trained with training data that is annotated with labels that are based on a plurality of salience annotation values that include the salience annotation value.

4. The system of claim 1, wherein:

the soft labeling component is configured to determine the salience annotation value of the item for respective ones of the web pages, based on determining a set of the user queries that are associated with one or more corresponding visits to the respective web page, wherein the group of the user queries that include the item includes one or more of;

a group of the user queries that substantially match the item, or exactly match the item, and that are included in the set, or

a group of the user queries that include the item as a proper substring, and that are included in the set.

5. The system of claim 1, wherein:

the soft labeling component is configured to determine a plurality of salience annotation values of a plurality of respective items for respective ones of the web pages, the system further comprising;

a training data acquisition component configured to obtain labeled training data for model training, based on the determined plurality of salience annotation values.

6. The system of claim 5, further comprising:

a model determination component configured to obtain a supervised machine-learned model that is trained offline with the labeled training data.

7. The system of claim 5, further comprising:

a feature function determination component configured to obtain feature function values for respective item-page pairs of the items and associated web pages, the feature function values indicating weights associated with one or more features of the associated web pages, wherein;

the features include one or more page classification features associated with the associated web pages.

8. The system of claim 7, wherein:

the features include one or more of;

corpus features,

Uniform Resource Locator (URL) features,

length features,

web page structural features,

web graph features associated with the associated web pages,

user profile features,

user interest features,

position features, or

page segmentation features.

9. The system of claim 7, further comprising:

a model determination component configured to obtain a model that is trained with training data that includes representations of the item-page pairs that are annotated with labels that are based on a plurality of salience annotation values that include the salience annotation value, wherein the representations of the item-page pairs each include a plurality of the feature function values.

10. The system of claim 9, further comprising:

an item-document salience determination component configured to obtain salience scores associated with respective ones of the items and associated ones of the web pages, the salience scores indicating a measure of salience of the respective items to the respective associated web pages, based on the trained model.

11. The system of claim 10, wherein:

the item-document salience determination component is configured to obtain the salience scores based on data obtained from the supervised machine-learned model and the feature function determination component.

12. A method comprising:

obtaining a first set of representations of a plurality of item-document pairs of items and respective documents that include the respective items, each of the representations including a plurality of feature function values indicating weights associated with one or more features of the associated documents, the features including one or more document classification features associated with the associated documents;

initiating, via a device processor, training of one or more models based on the first set; and

obtaining salience scores associated with respective ones of the items and associated ones of the documents, the salience scores indicating a measure of salience of the respective items to the respective associated documents, based on the trained one or more models.

13. The method of claim 12, further comprising:

obtaining an annotated set of labeled training data that is annotated with a plurality of salience annotation values of a plurality of respective items for respective documents that include content that includes the respective items, the salience annotation values determined based on a soft function, based on determining a first count of a total number of user queries that are associated with one or more corresponding visits to the respective documents, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with a group of the user queries that include the item, the subset included in the one or more corresponding visits, wherein

initiating training of the one or more models includes initiating training of the one or more models based on the annotated set.

14. The method of claim 12, wherein:

training of the one or more models includes training of the one or more models based on a learning algorithm that includes one or more of;

regression,

ranking,

binary classification,

boosted decision trees,

support vector machines,

linear regression,

logistic regression, or

neural networks.

15. The method of claim 12, wherein:

the documents include web pages, and

the document classification features include one or more of;

page categories of the web pages,

junk page scores of the web pages,

inlink page scores of the web pages, or

spam page confidence values of the web pages.

16. The method of claim 12, wherein:

the documents include web pages, and

the features include one or more of;

corpus features,

Uniform Resource Locator (URL) features,

length features,

web page structural features,

web graph features associated with the associated web pages,

user profile features,

user interest features,

position features, or

page segmentation features.

17. The method of claim 16, wherein:

the corpus features include an indication that a plurality of items included in a first group of the web pages is included in a second group of the web pages in a top domain of web pages,

the URL features include a top level domain indication,

the web page structural features include one or more of;

indicators of items included in titles of associated web pages,

indicators of items included in meta keywords of associated web pages,

indicators of items included in visual titles of associated web pages,

indicators of items included in bold font of associated web pages,

indicators of items included in emphasized font of associated web pages,

indicators of items included in italicized font of associated web pages,

indicators of items included in URLs of associated web pages,

indicators of items included in domain parts URLs of associated web pages, or

indicators of items included in suffix portions URLs of associated web pages, and

the web graph features include a normalized frequency value indicating a frequency of occurrence of items included in in anchors of web pages.

18. A computer program product tangibly embodied on a computer-readable storage medium and including executable code that causes at least one data processing apparatus to:

obtain a first set of representations of a plurality of item-page pairs of items and respective web pages that include the respective items, each of the representations including a plurality of feature function values indicating weights associated with one or more features of the associated web pages, the features including one or more page classification features associated with the associated web pages; and

obtain an annotated set of labeled training data that is annotated with a plurality of salience annotation values of a plurality of the respective items for respective web pages that include content that includes the respective items, the salience annotation values determined based on a soft function, based on determining a first count of a total number of user queries that are associated with one or more corresponding visits to the respective web pages, and determining a ratio of a second count to the first count, the second count determined as a cardinality of a subset of the corresponding visits that are associated with a group of the user queries that include the item, the subset included in the one or more corresponding visits; and

initiate training of one or more models based on the annotated set.

19. The computer program product of claim 18, wherein the executable code is configured to cause the at least one data processing apparatus to:

obtain salience scores associated with respective ones of the items and associated ones of the web pages, the salience scores indicating a measure of salience of the respective items to the respective associated web pages, based on the trained one or more models.

20. The computer program product of claim 19, wherein the executable code is configured to cause the at least one data processing apparatus to:

obtain a ranked list of the salience scores, the ranked list ranked in accordance with an ordering of the salience scores indicating measures of salience of one of the items to one or more of the web pages that include the one of the items.