METHOD AND APPARATUS FOR GENERATING A LANGUAGE INDEPENDENT DOCUMENT ABSTRACT

• US 20090182553A1
• Filed: 09/26/2008
• Published: 07/16/2009
• Est. Priority Date: 09/28/1998
• Status: Active Grant

• Current Assignee: Schukhaus Group GmbH LLC (Intellectual Ventures LLC), Prompt Software, Incorporated
• Original Assignee: Udico Holdings
• Inventors: Chaney, Garnet R., Rubinstein, Seymour I., Richardson, Robert F.
• Granted Patent: US 7,792,667 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93

Abstract

A method of extracting significant phrases from one or more documents stored in a computer-readable medium. A sequence of words is read from the one or more documents and a score is determined for each word in the sequence based on the length of the word. The score for each word in the sequence is compared against a threshold score. The sequence of words is indicated to be a significant phrase if the number of words in the sequences that have a score greater than the threshold score equals or exceeds a predetermined number. A sentence containing the sequence of words is retrieved from the document, if the sequence of words is a significant phrase. An abstract of the document is searched to determine if the sentence has been previously included in the abstract. If not, the sentence is added to the abstract.

Claims

1. A method of automatic, computer based identification of a significant phrase in a document, the method comprising:

storing the document, a threshold score, a verbosity setting, and a significant phrases data structure in a memory;

accessing the memory to read a sequence of words from the document;

determining by a processing unit a score for each word in the sequence based on the length of each word;

operating the processing unit to compare the score for each word in the sequence against the threshold score;

adding the sequence of words as a significant phrase to the significant phrase data structure if;

the number of words in the sequence that have the score greater than the threshold score equals or exceeds a predetermined number and

the number of words in the sequence satisfies the verbosity setting;

retrieving a sentence from the document in the memory if the sentence contains a significant phrase stored in the significant phrases data structure; and

operating the processing unit to search an abstract of the document to determine whether the sentence is included in the abstract.

2. The method of claim 1, wherein reading the sequence of words in the document further comprises reading the sequence of words in the document until a phrase delimiter is detected.

3. The method of claim 1, wherein the score for the word is proportional to the number of characters in the word.

4. The method of claim 3, wherein the number of characters in the word includes the number of numeric digits in the word.

5. The method of claim 1, wherein the score for the word is increased for each capitalized letter in the word.

6. The method of claim 1, wherein the score for the word is the length of the word plus the number of capitalized letters in the word.

7. The method of claim 1, wherein determining the score for each word in the sequence further includes:

determining whether the word exists in a predetermined table; and

retrieving the score for the word from the predetermined table.

8. The method of claim 1, wherein the threshold score is based on an average of the scores of words in the sequence of words.

9. The method of claim 8, wherein the threshold score is adjusted by a predetermined value to be different from the average of the scores of words in the sequence of words.

10. The method according to claim 1, further comprising including the sentence in the abstract, if the sentence is not included in the abstract.

11. The method according to claim 1, further comprising discarding the sentence if the sentence is included in the abstract.

12. The method according to claim 1, wherein the abstract is language independent.

35. The method of claim 1, wherein a significant phrase is indicated based on adjoining patterns within the sequences of words.

36. The method of claim 1, wherein a significant phrase in a document is identified with a single pass through the words of the document and the recognition of adjacent pattern comparisons of long words surrounded by short words.

37. The method of claim 1, in which the verbosity setting is self tuning to the complexity of the document if there is no verbosity setting received from an input device.

13. A method of identifying a significant phrase in a document, the method comprising:

storing the document, a threshold score, a verbosity setting, and a significant phrases data structure in a memory;

accessing the memory to read a sequence of words from the document;

determining by a processing unit a score for each word in the sequence based on the length of each word;

operating the processing unit to compare the score for each word in the sequence against a threshold score;

adding the sequence of words as a significant phrase to the significant phrase data structure if the number of words in the sequence that have the score greater than the threshold score equals or exceeds a predetermined number and the number of words in the sequence satisfies the verbosity setting;

storing the sequence of words in the significant phrases data structure and storing the number of words in the sequence in the memory, if the sequence of words is a significant phrase.

14. The method according to claim 13, further comprising:

retrieving a sentence from the document, the sentence containing the sequence of words, if the sequence of words is a significant phrase; and

searching an abstract of the document to determine whether the sentence is included in the abstract.

15. The method according to claim 14, further comprising including the sentence in the abstract, if the sentence is not included in the abstract.

16. The method according to claim 14, further comprising discarding the sentence if the sentence is included in the abstract.

17. The method according to claim 14, wherein the abstract is language independent.

18. A tangible computer readable storage medium containing executable instructions which, if executed in a processing system, cause the system to perform a method for identifying a significant phrase in a document, the method comprising:

reading a sequence of words from the document;

determining a score for each word in the sequence based on the length of each word;

comparing the score for each word in the sequence against a threshold score;

indicating that the sequence of words is a significant phrase if the number of words in the sequence that have the score greater than the threshold score equals or exceeds a predetermined number and the number of words in the sequence satisfies the verbosity setting;

retrieving a sentence from the document, the sentence containing the sequence of words, if the sequence of words is a significant phrase; and

searching an abstract of the document to determine whether the sentence is included in the abstract.

19. The tangible computer readable storage medium according to claim 18, wherein reading the sequence of words in the document further comprises reading the sequence of words in the document until a phrase delimiter is detected.

20. The tangible computer readable storage medium according to claim 18, wherein the score for the word is proportional to the number of characters in the word.

21. The tangible computer readable storage medium according to claim 20, wherein the number of characters in the word includes the number of numeric digits in the word.

22. The tangible computer readable storage medium according to claim 21, wherein the score for the word is increased for each capitalized letter in the word.

23. The tangible computer readable storage medium according to claim 18, wherein the score for the word is the length of the word plus the number of capitalized letters in the word.

24. The tangible computer readable storage medium according to claim 18, wherein determining the score for each word in the sequence further includes:

determining whether the word exists in a predetermined table; and

retrieving the score for the word from the predetermined table.

25. The tangible computer readable storage medium according to claim 18, wherein the threshold score is based on an average of the scores of words in the sequence of words.

26. The tangible computer readable storage medium according to claim 25, wherein the threshold score is adjusted by a predetermined value to be different from the average of the scores of words in the sequence of words.

27. The tangible computer readable storage medium according to claim 18, further comprising including the sentence in the abstract, if the sentence is not included in the abstract.

28. The tangible computer readable storage medium according to claim 18, further comprising discarding the sentence if the sentence is included in the abstract.

29. The tangible computer readable storage medium according to claim 18, wherein the abstract is language independent.

30. A tangible computer readable storage medium containing executable instructions which, if executed in a processing system, cause the system to perform a method for identifying a significant phrase in a document, the method comprising:

reading a sequence of words from the document;

determining a score for each word in the sequence based on the length of each word;

comparing the score for each word in the sequence against a threshold score;

indicating that the sequence of words is a significant phrase if the number of words in the sequence that have the score greater than the threshold score equals or exceeds a predetermined number and the number of words in the sequence satisfies a verbosity setting;

storing the sequence of words and the number of words in the sequence, if the sequence of words is a significant phrase.

31. The tangible computer readable storage medium according to claim 30, wherein the method further comprises:

retrieving a sentence from the document, the sentence containing the sequence of words, if the sequence of words is a significant phrase; and

searching an abstract of the document to determine whether the sentence is included in the abstract.

32. The tangible computer readable storage medium according to claim 31, wherein the method further comprises including the sentence in the abstract, if the sentence is not included in the abstract.

33. The tangible computer readable storage medium according to claim 31, wherein the method further comprises discarding the sentence if the sentence is included in the abstract.

34. The tangible computer readable storage medium according to claim 31, wherein the abstract is language independent.

38. A processing system configured to create an abstract for a document comprising:

a memory storing the document, a threshold score, a verbosity setting, and a significant phrases data structure;

a processing unit;

a bus operably coupling the processing unit to the memory;

the processing unit configured to;

read a sequence of words from the document from the memory via the bus and a document pointer indicating a portion of the memory to read;

determine a score for each word in the sequence based at least in part on the length of each word;

compare the score for each word in the sequence against the threshold score stored in the memory;

add the sequence of words as a significant phrase to the significant phrases data structure if the number of words in the sequence that have the score greater than the threshold score equals or exceeds a predetermined number and the number of words in the sequence satisfies the verbosity setting;

retrieve a sentence from the document from the memory via the bus if the sentence contains a significant phrase stored in the significant phrases data structure; and

search the abstract of the document to determine whether the sentence is included in the abstract.

39. The processing system of claim 38, wherein the processing unit reads the sequence of words in the document until the processing unit detects a phrase delimiter.

40. The processing system of claim 38, wherein the processing unit, as part of determining the score for each word in the sequence, is further configured to:

determine whether the word exists in a table of fixed length words stored in the memory; and

retrieve the score for the word from the table of fixed length words.

41. The processing system of claim 40 wherein the table of fixed length words stored in the memory is responsive to inputs from an input device to modify to the table of fixed length words.

42. The processing system of claim 38, wherein the threshold score stored in the memory is based on an average of the scores of words in the sequence of words.

43. The processing system of claim 42, wherein the threshold score stored in the memory is adjusted by a predetermined value to be different from the average of the scores of words in the sequence of words.

44. The processing system of claim 38, wherein the threshold score stored in the memory is set by an input received from an input device.

45. The processing system according to claim 38, wherein the processing unit is configured to add the sentence to the abstract, if the sentence is not included in the abstract.

46. The processing system according to claim 38, wherein the processing unit is configured to discard the sentence if the sentence is included in the abstract.

47. The processing system according to claim 38, further comprising a phrase buffer stored in the memory, the phrase buffer storing the sequence of words read by the processing unit prior to adding the sequence of words to the significant phrases data structure.

48. The processing system according to claim 38, wherein the processing unit is further configured to provide a display of sequences of words stored in the significant phrases data structure;

the processing system further comprising an input device providing inputs to control the of sequences of words stored in the significant phrases data structure.