System and method for the automatic mining of new relationships

• US 6,539,376 B1
• Filed: 11/15/1999
• Issued: 03/25/2003
• Est. Priority Date: 11/15/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Sundaresan, Neelakantan, Yi, Jeonghee
• Primary Examiner(s): Homere, Jean R.
• Assistant Examiner(s): Wassum, Luke S
• Application Number:
  US09/440,626
• Time in Patent Office: 1,226 Days
• Field of Search: 707/1-5, 707/10, 704/9
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

An automatic mining system that identifies a set of relevant terms from a large text database of unstructured information, such as the World Wide Web with a high degree of confidence. The automatic mining system includes a software program that enables the discovery of new relationships by association mining and refinement of co-occurrences, using automatic and iterative recognition of new binary relations through phrases that embody related pairs, by applying lexicographic and statistical techniques to classify the relations, and further by applying a minimal amount of domain knowledge of the relevance of the terms and relations. The automatic mining system includes a knowledge module and a statistics module. The knowledge module is comprised of a stemming unit, a synonym check unit, and a domain knowledge check unit. The stemming unit determines if the relation being analyzed shares a common root with a previously mined relation. The synonym check unit identifies the synonyms of the relation, and the domain knowledge check unit considers extrinsic factors for indications that would further clarify the relationship being mined. The statistics module optimizes the confidence level in the relationship.

Claims

1. A system of automatically mining a new relationship for an extracted pair p_i and a relation r_i within a document d_i, comprising:

a relations database for storing previously identified sets of pairs P<sub>i−

1</sub> and relations R<sub>i−

1</sub>;

a relations database for storing previously identified sets of pairs P<sub>i−

1</sub> and relations R<sub>i−

1</sub>;

a relationship database for storing the relation r_i under a corresponding relationship category;

wherein if there is no relationship category corresponding to the relation r_i in the relationship database, a knowledge module receives the extracted pair p_i and the relation r_i and applies a linguistic technique to classify the relation r_i;

wherein if the linguistic technique fails to identify a corresponding relationship category in the relationship database, a statistics module receives the relation r_i and applies a statistical technique to classify the relation r_i; and

wherein if the statistical technique fails to identity a corresponding relationship category in the relationship database, defining a new relationship category and storing the relation r_i under the new relationship category in the relationship database.

2. The system according to claim 1, wherein the knowledge module includes a stemming unit that determines if the relation r_i shares a common root with a previously mined relation r_i−

1 in the relations database.

3. The system according to claim 2, wherein the knowledge module includes a synonym check unit that identifies one or more synonyms of the relation r_i.

4. The system according to claim 2, wherein the knowledge module includes a domain knowledge check unit that considers one or more extrinsic indications to clarify the new relationship.

5. The system according to claim 1, wherein the knowledge module includes:

a stemming unit that determines if the relation r_i shares a common root with a previously mined relation r<sub>i−

1</sub> in the database;

a synonym check unit that identifies one or more synonyms of the relation r_i;

a domain knowledge check unit that considers one or more extrinsic indications to clarify the new relationship.

6. The system according to claim 1, wherein the knowledge module includes a stemming unit that determines if the relation r_i shares a common root with a previously mined relation r_i−

1 in the database, and removes common suffixes, if any.

7. The system according to claim 1, wherein the linguistic technique comprises stemming.

8. The system according to claim 7, wherein the linguistic technique further comprises synonym checking.

9. The system according to claim 1, wherein the knowledge module further applies domain knowledge to classify the relation r_i.

10. A computer program product of automatically mining a new relationship for an extracted pair p_i and a relation r_i within a document d_i, comprising:

a relations database for storing previously identified sets of pairs P<sub>i−

1</sub> and relations R<sub>i−

1</sub>;

a relations database for storing previously identified sets of pairs P<sub>i−

1</sub> and relations R<sub>i−

1</sub>;

a relationship database for storing the relation r_i under a corresponding relationship category;

wherein if there is no relationship category corresponding to the relation r_i in the relationship database, a knowledge module receives the extracted pair p_i and the relation r_i and applies a linguistic technique to classify the relation r_i;

wherein if the linguistic technique fails to identify a corresponding relationship category in the relationship database, a statistics module receives the relation r_i and applies a statistical technique to classify the relation r_i; and

wherein if the statistical technique fails to identify a corresponding relationship category in the relationship database, defining a new relationship category and storing the relation r_i under the new relationship category in the relationship database.

11. The computer program product according to claim 10, wherein the knowledge module includes a stemming unit that determines if the relation r_i shares a common root with a previously mined relation r_i−

1 in the relations database.

12. The computer program product according to claim 11, wherein the knowledge module includes a synonym check unit that identifies one or more synonyms of the relation r_i.

13. The computer program product according to claim 11, wherein the knowledge module includes a domain knowledge check unit that considers one or more extrinsic indications to clarify the new relationship.

14. The computer program product according to claim 10, wherein the knowledge module includes:

a stemming unit that determines if the relation r_i shares a common root with a previously mined relation r<sub>i−

1</sub> in the database;

a synonym check unit that identifies one or more synonyms of the relation r_i;

a domain knowledge check unit that considers one or more extrinsic indications to clarify the new relationship.

15. The computer program product according to claim 10, wherein the knowledge module includes a stemming unit that determines if the relation r_i shares a common root with a previously mined relation r_i−

1 in the database, and removes common suffixes, if any.

16. A method of automatically mining a new relationship for an extracted pair p_i and a relation r_i within a document d_i, comprising:

a relations database storing previously identified sets of pairs P<sub>i−

1</sub> and relations R<sub>i−

1</sub>;

a relationship database storing the relation r_i under a corresponding relationship category;

wherein if there is no relationship category corresponding to the relation r_i in the relationship database, a knowledge module receiving the extracted pair p_i and the relation r_i and applying a linguistic technique to classify the relation r_i;

wherein if the linguistic technique fails to identify a corresponding relationship category in the relationship database, a statistics module receiving the relation r_i and applies a statistical technique to classify the relation r_i; and

wherein if the statistical technique fails to identify a corresponding relationship category in the relationship database, defining a new relationship category and storing the relation r_i under the new relationship category in the relationship database.

17. The method according to claim 16, further including determining if the relation r_i shares a common root with a previously mined relation r_i−

1 in the relations database.

18. The method according to claim 17, further including identifying one or more synonyms of the relation r_i.

19. The method according to claim 17, further including considering one or more extrinsic indications to clarify the new relationship.

20. The method according to claim 16, further including:

determining if the relation r_i shares a common root with a previously mined relation r<sub>i−

1</sub> in the relations database;

identifying one or more synonyms of the relation r_i; and

considering one or more extrinsic indications to clarify the new relationship.

21. The method according to claim 16, further including determining if the relation r_i shares a common root with a previously mined