Web object retrieval based on a language model

• US 8,001,130 B2
• Filed: 07/25/2006
• Issued: 08/16/2011
• Est. Priority Date: 07/25/2006
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Nie, Zaiqing, Ma, Yunxiao, Shi, Shuming, Wen, Ji-Rong
• Primary Examiner(s): FILIPCZYK, MARCIN R
• Application Number:
  US11/459,857
• Publication Number:
  US 20080027910A1
• Time in Patent Office: 1,848 Days
• Field of Search: 707 1- 4, 707/10, 707/100, 707/750, 707/999.003, 707/999.006, 707/999.007, 707/730, 715/500
• US Class Current: 707/750
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A method and system is provided for determining relevance of an object to a term based on a language model. The relevance system provides records extracted from web pages that relate to the object. To determine the relevance of the object to a term, the relevance system first determines, for each record of the object, a probability of generating that term using a language model of the record of that object. The relevance system then calculates the relevance of the object to the term by combining the probabilities. The relevance system may also weight the probabilities based on the accuracy or reliability of the extracted information for each data source.

Claims

1. A computing system with a processor and a memory for determining relevance of an object to a target term, the object being a product, comprising:

a component that retrieves a plurality of web pages containing a description of the object, each description including terms used to describe the object;

a component that generates a collection of records of terms relating to the object, the collection including a record generated from each of the plurality of retrieved web pages, each record including a plurality of terms, each record of the collection being generated by extracting from a retrieved web page terms used to describe the object, such that the terms extracted from that retrieved web page compose the record of terms for that retrieved web page;

a component that, for each record of the collection of records of terms relating to the object,

determines a language model probability for that record generating the target term, the language model probability for that record generating the target term being a weighted summation of a number of occurrences of the target term in the record divided by a number of occurrences of all terms in the record and a number of occurrences of the target term in the collection of the records divided by a number of occurrences of all terms in the collection of the records as represented by the following equation;

2. The computing system of claim 1 wherein each record is from a data source having an accuracy and wherein the relevance factors in the accuracy of the data sources of the record.

3. The computing system of claim 2 wherein the accuracy represents accuracy of records of the data source.

4. The computing system of claim 2 wherein the accuracy represents accuracy of attributes of the records of the data source.

5. The computing system of claim 1 wherein the component that determines the relevance generates a summation of the language model probability according to the following equation:

6. The computing system of claim 1 wherein the component that determines a language model probability factors in probability from attributes of each record.

7. The computing system of claim 6 wherein each attribute of a record is represented as a separate document comprising terms of the attribute.

8. The computing system of claim 6 wherein the determined language model probability is based on accuracy of records and accuracy of attributes of a data source.

9. The computing system of claim 6 wherein the determined language model probability is based on importance of each attribute.

10. The computing system of claim 6 wherein a contribution of each attribute to the determined language model probability for a record factors in importance of each attribute based on the accuracy of the attributes.

11. The computing system of claim 1 wherein the component that determines the relevance factors in a probability of each record generated according to the following equation: