Identifying the unifying subject of a set of facts

• US 7,831,545 B1
• Filed: 05/31/2005
• Issued: 11/09/2010
• Est. Priority Date: 05/31/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Betz, Jonathan T.
• Primary Examiner(s): Vital; Pierre M
• Assistant Examiner(s): Nofal; Christopher P
• Application Number:
  US11/142,765
• Time in Patent Office: 1,988 Days
• Field of Search: 707/1, 707/2, 707/5, 707/6, 707/7, 707/100, 709/218, 715/500.1, 715/501.1, 715/513
• US Class Current: 707/603
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: Y10S 707/96 Object-relational

Abstract

A method and system for identifying a subject of a document and facts included within are described. A source document that includes facts and linking documents that include hyperlinks to the source document are identified. The anchor texts of the hyperlinks are identified and candidate labels are generated based on the anchor texts. One of the candidate labels is selected based on first predefined criteria and associated with the source document and/or the facts included within the source document.

Claims

1. A computer-implemented method of processing a set of documents for generating a facts database, comprising:

at a system having one or more processors and memory storing one or more modules to be executed by the one or more processors;

accessing a source document from a document host;

extracting one or more facts from the source document, each fact including an attribute-value pair and a list of documents that include the fact;

identifying a set of linking documents that have one or more links to the source document, wherein a respective link contains anchor text;

generating a set of candidate labels from the anchor text of the linking documents;

assigning a score to each candidate label based on a number of linking documents having the candidate label in the anchor text;

selecting the candidate label with a highest score as a unifying subject of the one or more facts; and

for the unifying subject, storing in the facts database an object distinct from the source document, wherein the object includes the unifying subject, one or more entries corresponding to the one or more facts extracted from the source document, and information associating the source document with at least one of the facts extracted from the source document and included in the object.

2. The method of claim 1, further comprising:

selecting one or more second labels of the candidate labels according to second predefined criteria; and

associating the selected second labels with the source document and the one or more facts extracted from the source document.

3. The method of claim 1, wherein selecting the candidate label comprises:

for each of the set of candidate labels;

determining a set of frequencies of one or more substrings of the respective candidate label;

generating a frequency vector associated with the respective candidate label based on the set of frequencies;

determining a centroid vector based on the frequency vectors of the candidate labels, wherein the selected candidate label is associated with the respective frequency vector having a shortest distance to the centroid vector.

4. A server system for processing a set of documents for generating a facts database, comprising:

one or more processors;

memory storing one or more programs to be executed by the one or more processors, the one or more programs including;

a document identification module to access a source document from a document host;

an extraction module to extract one or more facts from the source document, each fact including an attribute-value pair and a list of documents that include the fact;

a linking document module to identify a set of linking documents, that have one or more links to the source document, wherein a respective link contains anchor text; and

a label module having instructions to;

generate a set of candidate labels from the anchor text of the linking documents;

assign a score to each candidate label based on a number of linking documents having the candidate label in the anchor text;

select the candidate label with a highest score as a unifying subject of the one or more facts; and

for the unifying subject, store in the facts database an object distinct from the source document, wherein the object includes the unifying subject, one or more entries corresponding to the one or more facts extracted from the source document, and information associating the source document with at least one of the facts extracted from the source document and included in the object.

5. The system of claim 4, wherein the label selection instructions further include instructions to select one or more second labels of the candidate labels according to second predefined criteria;

and

wherein the label association instructions further include instructions to associate the selected second labels with the source document and the one or more facts extracted from the source document.

6. The system of claim 4, wherein the label selection instructions include instructions to:

for each of the set of candidate labels;

determine a set of frequencies of one or more substrings of the respective candidate label;

generate a frequency vector associated with the respective candidate label based on the set of frequencies; and

determine a centroid vector based on the frequency vectors of the candidate labels, wherein the selected candidate label is associated with the respective frequency vector having a shortest distance to the centroid vector.

7. A computer readable storage medium storing one or more computer programs executed by a computerized server system, the one or more computer programs comprising instructions to generate a facts database, the instructions including:

instructions to access a source document from a document host;

instructions to extract one or more facts from the source document, each fact including an attribute-value pair and a list of documents that include the fact;

instructions to identify a set of linking documents that have one or more links to the source document, wherein a respective link contains anchor text;

instructions to generate a set of candidate labels from the anchor text of the linking documents;

instructions to select the candidate label with a highest score as a unifying subject of the one or more facts; and

instructions to, for the unifying subject, store in the facts database an object distinct from the source document, wherein the object includes the unifying subject, one or more entries corresponding to the one or more facts extracted from the source document, and information associating the source document with at least one of the facts extracted from the source document and included in the object.

8. The computer readable storage medium of claim 7, further comprising instructions to:

select one or more second labels of the candidate labels according to second predefined criteria; and

associate the selected second labels with the source document and the one or more facts extracted from the source document.

9. The computer readable storage medium of claim 7, wherein the instructions for selecting the label comprise instructions to:

for each of the set of candidate labels;

determine a set of frequencies of one or more substrings of the respective candidate label;

generate a frequency vector associated with the respective candidate label based on the set of frequencies; and

determine a centroid vector based on the frequency vectors of the candidate labels, wherein the selected label is associated with the respective frequency.

10. A server system for processing a set of documents for generating a facts database, comprising:

one or more processors;

memory storing one or more programs to be executed by the one or more processors, the system including;

means for accessing a source document from a document host;

means for extracting one or more facts from the source document each fact including an attribute-value pair and a list of documents that include the fact;

means for identifying a set of linking documents that have one or more links to the source document, wherein a respective link contains anchor text;

means for generating a set of candidate labels from the anchor text of the linking documents;

means for assigning a score to each candidate label based on a number of linking documents having the candidate label in the anchor text;

means for selecting the candidate label with a highest score as a unifying subject of the one or more facts; and

means for storing data in the memory, separate from the source document, the stored data comprising a representation of a three-way association among the one or more facts, the unifying subject of the one or more facts, and the source document;

means, for the unifying subject, for storing in the facts database an object distinct from the source document, wherein the object includes the unifying subject, one or more entries corresponding to the one or more facts extracted from the source document, and information associating the source document with at least one of the facts extracted from the source document and included in the object.