Automatically Linking Documents With Relevant Structured Information

• US 20080065655A1
• Filed: 09/08/2006
• Published: 03/13/2008
• Est. Priority Date: 09/08/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Roy, Prasan, Mohania, Mukesh Kumar, Gupta, Himanshu, Chakravarthy, Venkat
• Granted Patent: US 7,899,822 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A method of associating a given text document with relevant structured data is disclosed. The method receives as inputs a text document, and structured data in the form of a relational database. The method then identifies terms in the text document, and searches and queries the structured data using the terms to identify fragments of the structured data that are relevant to the document. Finally, the text document and the identified fragments of structured data are output to a user.

Claims

1. A method of associating a text document with relevant structured data, said method comprising the steps of:

(a) receiving a text document;

(b) receiving structured data in the form of a relational database;

(c) identifying fragments of the structured data that are relevant to the text document; and

(d) outputting the text document and the identified fragments of structured data.

2. A method according to claim 1 wherein step (c) comprises the sub-steps of:

(ci) receiving a set of predefined entities in the relational database;

(cii) receiving context information for the entities;

(ciii) parsing the document and using a filtering mechanism to identify a set of terms, where said set of terms is a subset of word phrases found in said text document; and

(civ) searching and querying the structured data based on the identified set of terms to find an annotation that, based upon a predefined criterion, best matches the document among all annotations, where an annotation consists of a set of entities and for each entity, a set of segments within the text document that are related to the entity.

3. A method according to claim 2 wherein step (civ) comprises:

calculating a score according to said predefined criterion for each annotation, wherein said score is based on the terms contained in the context information for the entities present in the annotation; and

finding the annotation with the maximum score.

4. A method according to claim 2 wherein steps (ci) and (cii) comprise:

receiving one or more templates, where each template is a rooted tree with a designated root node, each node in the tree is labelled with a table from said relational database, and an edge associating two nodes exists when the tables labelling each of the two nodes have a foreign-key relationship in the relational schema of said relational database, each tuple in the relation labelling the root node of any template constitutes an entity, and for any entity in a template, the context information of the entity consists of the tuples that have a path to said entity through one or more foreign-keys covered by said template.

5. A method according to claim 3 wherein the step of finding the annotation with the maximum score comprises the sub-steps of:

(A) maintaining a context cache comprising containment relationships between entities and terms, where each containment relationship is an entity-term pair such that the term is contained in the context of the entity;

(B) based on the containment relationships present in the context cache, issuing a query to the relational database based on the contents of the cache to identify further containment relationships between entities and terms;

(C) updating the context cache with the results from the query;

(D) finding a current best annotation based on the context cache; and

(E) repeating steps (B) through (D) until the annotation having the maximum score is found.

6. A method of partitioning a given text document into coherent segments with respect to structured data, said method comprising the steps of:

(a) receiving a text document;

(b) receiving structured data in the form of a relational database;

(c) identifying terms in the text document;

(d) receiving a set of predefined entities in the relational database;

(e) receiving context information for the entities;

(f) identifying entities that are most relevant to the document by using the context information and the terms in the text document;

(g) finding an annotation that is most relevant to the document among all annotations, where an annotation consists of a set of entities and related segments that partition the text document; and

(h) outputting the text document and the identified segments that partition the text document.

7. An apparatus for associating a text document with relevant structured data, said apparatus comprising:

means for receiving a text document;

means for receiving structured data in the form of a relational database;

means for identifying fragments of the structured data that are relevant to the text document; and

means for outputting the text document and the identified fragments of structured data.

8. An apparatus according to claim 7 wherein said means for identifying fragments comprises:

means for receiving a set of predefined entities in the relational database;

means for receiving context information for the entities;

means for parsing the document and using a filtering mechanism to identify a set of terms, where said set of terms is a subset of word phrases found in said text document; and

means for searching and querying the structured data based on the identified set of terms to find an annotation that, based upon a predefined criterion, best matches the document among all annotations, where an annotation consists of a set of entities and for each entity, a set of segments within the text document that are related to the entity.

9. An apparatus for partitioning a given text document into coherent segments with respect to structured data, said apparatus comprising:

means for receiving a text document;

means for receiving structured data in the form of a relational database;

means for identifying terms in the text document;

means for receiving a set of predefined entities in the relational database;

means for receiving context information for the entities;

means for identifying entities that are most relevant to the document by using the context information and the terms in the text document;

means for finding an annotation that is most relevant to the document among all annotations, where an annotation consists of a set of entities and related segments that partition the text document; and

means for outputting the text document and the identified segments that partition the text document.

10. A computer readable medium comprising a program for associating a text document with relevant structured data, said program controlling the operation of a data processing apparatus on which the program executes to perform the steps of:

(a) receiving a text document;

(b) receiving structured data in the form of a relational database;

(c) identifying fragments of the structured data that are relevant to the text document; and

(d) outputting the text document and the identified fragments of structured data.

11. A computer readable medium according to claim 10 wherein step (c) comprises the sub-steps of:

(ci) receiving a set of predefined entities in the relational database;

(cii) receiving context information for the entities;

(ciii) parsing the document and using a filtering mechanism to identify a set of terms, where said set of terms is a subset of word phrases found in said text document; and

(civ) searching and querying the structured data based on the identified set of terms to find an annotation that, based upon a predefined criterion, best matches the document among all annotations, where an annotation consists of a set of entities and for each entity, a set of segments within the text document that are related to the entity.

12. A computer readable medium according to claim 11 wherein step (civ) comprises:

calculating a score according to said predefined criterion for each annotation, wherein said score is based on the terms contained in the context information for the entities present in the annotation; and

finding the annotation with the maximum score.

13. A computer readable medium according to claim 12 wherein the step of finding the annotation with the maximum score comprises the sub-steps of:

(A) maintaining a context cache comprising containment relationships between entities and terms, where each containment relationship is an entity-term pair such that the term is contained in the context of the entity;

(B) based on the containment relationships present in the context cache, issuing a query to the relational database based on the contents of the cache to identify further containment relationships between entities and terms;

(C) updating the context cache with the results from the query;

(D) finding a current best annotation based on the context cache; and

(E) repeating steps (B) through (D) until the annotation having the maximum score is found.

14. A computer readable medium comprising a program for partitioning a given text document into coherent segments with respect to structured data, said program controlling the operation of a data processing apparatus on which the program executes to perform the steps of:

(a) receiving a text document;

(b) receiving structured data in the form of a relational database;

(c) identifying terms in the text document;

(d) receiving a set of predefined entities in the relational database;

(e) receiving context information for the entities;

(f) identifying entities that are most relevant to the document by using the context information and the terms in the text document;

(g) finding an annotation that is most relevant to the document among all annotations, where an annotation consists of a set of entities and related segments that partition the text document; and

(h) outputting the text document and the identified segments that partition the text document.