WEB OBJECT RETRIEVAL BASED ON A LANGUAGE MODEL

• US 20080027910A1
• Filed: 07/25/2006
• Published: 01/31/2008
• Est. Priority Date: 07/25/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Nie, Zaiqing, Ma, Yunxiao, Shi, Shuming, Wen, Ji-Rong
• Granted Patent: US 8,001,130 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A method and system is provided for determining relevance of an object to a term based on a language model. The relevance system provides records extracted from web pages that relate to the object. To determine the relevance of the object to a term, the relevance system first determines, for each record of the object, a probability of generating that term using a language model of the record of that object. The relevance system then calculates the relevance of the object to the term by combining the probabilities. The relevance system may also weight the probabilities based on the accuracy or reliability of the extracted information for each data source.

Claims

1. A computing system for determining relevance of an object to a term, comprising:

a component that provides records of terms relating to the object;

a component that determines, for each record, a probability of a language model for that record generating the term; and

a component that calculates the relevance of the object to the term based on the probabilities of the records.

2. The computing system of claim 1 wherein each record is from a data source having an accuracy and wherein the relevance factors in the accuracy of the data sources of the record.

3. The computing system of claim 2 wherein the accuracy represents accuracy of records of the data source.

4. The computing system of claim 2 wherein the accuracy represents accuracy of attributes of the records of the data source.

5. The computing system of claim 1 wherein the component that determines a probability represents each record as a separate document comprising terms of the record.

6. The computing system of claim 1 wherein the component that determines a probability calculates the probability according to the following equation:

$p<br><br>\left(w|o\right)=\underset{k=1}{\overset{K}{\sum <br><br>}}<br><br><br><br>\left({\mathrm{\alpha <br><br>}}_k<br><br>P<br><br>\left(w|R_k\right)\right).$

7. The computing system of claim 1 wherein the component that determines a probability calculates the probability from attributes of each record.

8. The computing system of claim 7 wherein each attribute of a record is represented as a separate document comprising terms of the attribute.

9. The computing system of claim 7 wherein the calculated probability is based on accuracy of records and accuracy of attributes of a data source.

10. The computing system of claim 7 wherein the calculated probability is based on importance of each attribute.

11. The computing system of claim 7 wherein a contribution of each attribute to the calculated probability for a record factors in importance of each attribute to the extent of the accuracy of the attributes.

12. The computing system of claim 1 wherein the component that determines a probability calculates the probability according to the following equation:

$P<br><br>\left(w|o\right)=\underset{k=1}{\overset{K}{\sum <br><br>}}<br><br><br><br>\left({\mathrm{\alpha <br><br>}}_k<br><br>{\mathrm{\gamma <br><br>}}_k<br><br>\underset{j=1}{\overset{M}{\sum <br><br>}}<br><br><br><br>{\mathrm{\beta <br><br>}}_j<br><br>P<br><br>\left(w|O_{\mathrm{jk}}\right)\right).$

13. A computer-readable medium containing instructions for controlling a computer system to determine relevance of an object to a term, by a method comprising:

providing records of terms relating to the object, each record from a data source having an accuracy;

for each record, calculating a likelihood that the object is relevant to the term; and

calculating the relevance of the object to the term by combining the likelihoods based on the accuracies of the records.

14. The computer-readable medium of claim 13 wherein each record has a record accuracy and attribute accuracy and each attribute of a record is represented as a separate document comprising terms of the attribute.

15. The computer-readable medium of claim 13 wherein the calculated probability is based on importance of each attribute of a record.

16. The computer-readable medium of claim 13 wherein a contribution of each attribute to the calculated probability for a record factors in importance of each attribute to the extent of the accuracy of the attributes.

17. The computer-readable medium of claim 13 wherein the likelihood is derived from a probability for each attribute of the record of a language model for the attribute to generate the term.

18. The computer-readable medium of claim 17 wherein the calculating of a probability calculates the probability according to the following equation:

$P<br><br>\left(w|O\right)=\underset{k=1}{\overset{K}{\sum <br><br>}}<br><br><br><br>\left({\mathrm{\alpha <br><br>}}_k<br><br>{\mathrm{\gamma <br><br>}}_k<br><br>\underset{j=1}{\overset{M}{\sum <br><br>}}<br><br><br><br>{\mathrm{\beta <br><br>}}_j<br><br>P<br><br>\left(w|O_{\mathrm{jk}}\right)\right).$

19. A method in a computing system for controlling a computer system to determine relevance of an object to a term, the method comprising:

providing records of terms relating to the object, each record from a data source having a record accuracy and an attribute accuracy;

for each record, calculating a probability for each attribute of the record of a language model for the attribute to generate the term, the probability being estimated based on frequency of the term within an attribute of a record and frequency of the term within a collection of that attribute of all the records; and

calculating the relevance of the object to the term by combining the probabilities of the attributes based on the record accuracies and attribute accuracy of the data sources of the records and based on importance of an attribute to the object.

20. The method of claim 19 including calculating the relevance of the object to each term of a query and combining the calculated relevances to provide a relevance of the object to the query.