Method and system for indentifying significant topics of a document

• US 6,167,368 A
• Filed: 08/14/1998
• Issued: 12/26/2000
• Est. Priority Date: 08/14/1998
• Status: Expired due to Term

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Wacholder, Faye (Nina) P.
• Primary Examiner(s): EDOUARD, PATRICK NESTOR
• Application Number:
  US09/134,193
• Time in Patent Office: 865 Days
• Field of Search: 704/1, 704/8-10, 707/530, 707/531, 707/532
• US Class Current: 704/9
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking

Abstract

A "domain-general" method for representing the "sense" of a document includes the steps of extracting a list of simplex noun phrases representing candidate significant topics in the document, clustering the simplex noun phrases by head, and ranking the simplex noun phrases according to a significance measure to indicate the relative importance of the simplex noun phrases as significant topics of the document. Furthermore, the output can be filtered in a variety of ways, both for automatic processing and for presentation to users.

Claims

1. A method for identifying significant topics in a document comprising the steps of:

extracting from said document a complete list of simplex noun phrases and corresponding heads that represent candidate significant topics of said document;

clustering said simplex noun phrases into groups by said heads; and

ranking said clustered simplex noun phrases by said heads in accordance with a significance measure to identify said significant topics of said document from said candidate significant topics.

3. The method according to claim 1 or 2, further comprising the step of pre-processing said document to provide parsing information for extracting said simplex noun phrases.

4. The method according to claim 1 or 2, further comprising the step of post-processing said ranked simplex noun phrases to remove duplicates of said identified significant topics.

5. The method according to claim 1 or 2, further comprising the step of post-processing said ranked simplex noun phrases to remove near-duplicates of said identified significant topics.

6. The method according to claim 1 or 2, further comprising the step of post-processing said ranked simplex noun phrases to remove pronouns from said identified significant topics.

7. The method according to claim 1 or 2, further comprising the step of post-processing said ranked simplex noun phrases to exclude all of said ranked simplex noun phrases except those having a content-bearing modifier.

8. The method according to claim 1 or 2, wherein said significance measure is based at least in part on a frequency of said heads in said simplex noun phrases.

9. The method according to claim 1 or 2, wherein said significance measure is based at least in part on a frequency of said heads occurring as modifiers within said simplex noun phrases.

10. The method according to claim 1 or 2, wherein said significance measure is based at least in part on the location of said heads within said document.

11. The method according to claim 1, wherein said simplex noun phrases are described by a user-specified pattern.

12. The method according to claim 10, wherein said user-specified pattern includes a determiner preceded by an adjective preceded by a noun.

2. A method for identifying significant topics in a document comprising the steps of:

extracting from said document a complete list of simplex noun phrases and corresponding heads that represent candidate significant topics of said document, said simplex noun phrases being described by a user-specified pattern having a determiner preceding an adjective preceding a noun;

clustering said simplex noun phrases into groups by said heads; and

ranking said clustered simplex noun phrases by said heads in accordance with a significance measure to identify said significant topics of said document from said candidate significant topics.

13. A system for identifying significant topics in a document comprising:

a general purpose computer having a computer usable media;

computer readable program code means embodied in said computer usable media, said program code means comprising;

means for extracting from said document a complete list of simplex noun phrases and corresponding heads that represent candidate significant topics of said document;

means for clustering said simplex noun phrases into groups by said heads; and

means for ranking said clustered simplex noun phrases by said heads in accordance with a significance measure to identify significant topics of said document from said candidate significant topics.

14. The system according to claim 13, further comprising means for pre-processing said document to provide parsing information for extracting said simplex noun phrases.

15. The system according to claim 13, further comprising means for post-processing said ranked simplex noun phrases to remove duplicates of said identified significant topics.

16. The system according to claim 13, further comprising means for post-processing said ranked simplex noun phrases to remove near-duplicatcs of said identified significant topics.

17. The system according to claim 13, further comprising means for post-processing said ranked simplex noun phrases to exclude all of said ranked simplex noun phrases except those having a content-bearing modifier.

18. The system according to claim 13, wherein said significance measure is based at least in part on a frequency of said heads in said simplex noun phrases.

19. The system according to claim 13, wherein said significance measure is based at least in part on a frequency of said heads occurring as modifiers within said simplex noun phrases.

20. The system according to claim 13, wherein said significance measure is based at least in part on the location of said heads within said document.

21. The system according to claim 13, wherein said simplex noun phrases comprise a user-specified pattern.

22. The system according to claim 13, wherein said user-specified pattern comprises a determiner preceded by an adjective preceded by a noun.