System and method of automatic discovery of terms in a document that are relevant to a given target topic

• US 6,651,058 B1
• Filed: 11/15/1999
• Issued: 11/18/2003
• Est. Priority Date: 11/15/1999
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Sundaresan, Neelakantan, Yi, Jeonghee
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US09/439,758
• Time in Patent Office: 1,464 Days
• Field of Search: 707/1-10, 707/100-104, 707/200-205, 707/501.1, 707/512-515, 709/200-235, 706/45-50
• US Class Current: 1/1
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 2216/03 Data mining
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A computer program product is provided as an automatic mining system to discover terms that are relevant to a given target topic from a large databases of unstructured information such as the World Wide Web. The operation of the automatic mining system is performed in three stages: The first stage is carried out by a new terms discoverer for discovering the terms in a document, the second stage is carried out by a candidate terms discoverer for discovering potentially relevant terms, and the third stage is carried out by a relevant terms discoverer for refining or testing the discovered relevance to filter false relevance. The new terms discoverer includes a system for the automatic mining of patterns and relations, a system for the automatic mining of new relationships, and a system for selecting new terms from relations. In one embodiment, the system for the automatic mining of patterns and relations identifies a set of related terms on the WWW with a high degree of confidence, using a duality concept, and includes a terms database and two identifiers: a relation identifier and a pattern identifier. The system for the automatic mining of new relationships includes a database a knowledge module and a statistics module. The knowledge module includes a stemming unit, a synonym check unit, and a domain knowledge check unit. The candidate terms discoverer includes a metadata extractor, a document vector module, an association module, a filtering module, and a database. The relevant terms discoverer includes a stop word filter and a system for the automatic construction of generalization—specialization hierarchy of terms comprised of a terms database, an augmentation module, a generalization detection module, and a hierarchy database.

Claims

1. A system of automatically discovering terms in a document that are relevant to a given target topic, comprising:

a new terms discoverer for identifying the terms in the document;

a candidate terms discoverer for identifying potentially relevant terms from the terms identified by the new terms discoverer;

the candidate terms discoverer comprising;

an association module that performs a statistical analysis, regardless of an occurrence frequency of the terms within a single document; and

a filtering module that filters association rules with relevance that surpasses a user specified threshold; and

a relevant terms discoverer that identifies relevant terms by applying the association rules to the potentially relevant terms identified by the candidate terms discoverer, to refine a relevance of the potentially relevant terms by filtering false relevance.

2. The system according to claim 1, wherein the new terms discoverer includes a system for the automatic mining of patterns and relations.

3. The system according to claim 1, wherein the new terms discoverer includes a system for the automatic mining of new relationships.

4. The system according to claim 1, wherein the new terms discoverer includes a system for selecting new terms from relations.

5. The system according to claim 1, wherein the new terms discoverer includes a system for the automatic mining of patterns and relations, a system for the automatic mining of new relationships, and a system for selecting new terms from relations.

6. The system according to claim 2, wherein the system for the automatic mining of new relationships includes a knowledge module and a statistics module.

7. The system according to claim 6, wherein the knowledge module includes one or more of:

a stemming unit, a synonym check unit, or a domain knowledge check unit.

8. The system according to claim 1, wherein the candidate terms discoverer includes a metadata extractor.

9. The system according to claim 8, wherein the candidate terms discoverer includes a document vector module, an association module, and a filtering module.

10. The system according to claim 9, wherein the candidate terms discoverer further includes a database for storing relevant terms.

11. The system according to claim 9, wherein the metadata extractor identifies hypertext link metadata in the document, and the document vector module creates a document vector for each document.

12. The system according to claim 1, wherein the relevant terms discoverer includes a stop word filter and a system for the automatic construction of a generalization hierarchy of terms.

13. The system according to claim 9, wherein the system for the automatic construction of the generalization hierarchy of terms includes an augmentation module, a generalization detection module, and a hierarchy database.

14. A system of automatically discovering terms in a document that are relevant to a given target topic, comprising:

a new terms discoverer for identifying the terms in the document;

a candidate terms discoverer for identifying potentially relevant terms from the terms identified by the new terms discoverer;

the candidate terms discoverer comprising;

an association module that performs a statistical analysis, regardless of an occurrence frequency of the terms within a single document; and

a filtering module that filters association rules with relevance that surpasses a user specified threshold; and

a relevant terms discoverer that identifies relevant terms by applying the association rules to the potentially relevant terms identified by the candidate terms discoverer, to refine a relevance of the potentially relevant terms by filtering false relevance.

15. The system according to claim 14, wherein the new terms discoverer includes a system for the automatic mining of patterns and relations, a system for the automatic mining of new relationships, and a system for selecting new terms from relations.

16. The system according to claim 15, wherein the system for the automatic mining of new relationships includes a knowledge module and a statistics module;

and

wherein the knowledge module includes one or more of;

a stemming unit, a synonym check unit, or a domain knowledge check unit.

17. The system according to claim 15, wherein the candidate terms discoverer includes a metadata extractor, a document vector module, an association module, a filtering module, and a database for storing relevant terms;

and

wherein the system for the automatic construction of the generalization hierarchy of terms includes an augmentation module, a generalization detection module, and a hierarchy database.

18. A method of automatically discovering terms in a document that are relevant to a given target topic, comprising:

identifying the terms in the document by means of a new terms discoverer;

identifying potentially relevant terms from the terms identified by the new terms discoverer by means of a candidate terms discoverer that performs a statistical analysis, regardless of an occurrence frequency of the terms within a single document and that filters association rules with relevance that surpasses a user specified threshold; and

identifying relevant terms by applying the association rules to the potentially relevant terms identified by the candidate terms discoverer, to refine a relevance of the potentially relevant terms by filtering false relevance, by means of a relevant terms discoverer.

19. The method according to claim 18, wherein identifying the terms includes automatically mining patterns and relations, automatically mining new relationships, and selecting new terms from relations.

20. The system according to claim 19, wherein automatically mining new relationships includes applying knowledge by means of a knowledge module, and applying a statistical analysis by means of a statistics module.

21. A computer usable medium having instruction codes for automatically discovering terms in a document that are relevant to a given target topic, comprising:

a first set of instruction codes for identifying the terms in the document;

a second set of instruction codes for identifying potentially relevant terms from the terms identified by the first set of instruction codes;

for performing a statistical analysis regardless of an occurrence frequency of the terms within a single document; and

for filtering association rules with relevance that surpasses a user specified threshold; and

a third set of instruction codes that identifies relevant terms by applying the association rules to the potentially relevant terms identified by the second set of instruction codes, to refine a relevance of the potentially relevant terms by filtering false relevance.

22. The computer usable medium according to claim 21, wherein the first set of instruction codes comprises program codes for the automatic mining of patterns and relations, for the automatic mining of new relationships, and for selecting new terms from relations.