Text summarization using part-of-speech

• US 6,289,304 B1
• Filed: 03/17/1999
• Issued: 09/11/2001
• Est. Priority Date: 03/23/1998
• Status: Expired due to Term

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Grefenstette, Gregory
• Primary Examiner(s): EDOUARD, PATRICK NESTOR
• Application Number:
  US09/270,457
• Time in Patent Office: 909 Days
• Field of Search: 345/333, 704/1, 704/9, 704/10, 707/530, 707/531, 707/1, 707/4, 707/101, 707/500
• US Class Current: 704/9
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems

Abstract

Text is summarized using part-of-speech (POS) data indicating parts of speech for tokens in the text. The POS data can be obtained using input text data defining the text, such as by POS tagging. The POS data can be used to obtain group data indicating groups of tokens of the text, such as verb groups and noun groups. The group data can also indicate, within each group, any tokens that meet a POS based removal criterion. The group data can be used to obtain summarized text data by removing tokens that meet the removal criterion. The original text may be obtained via scanner or video camera from a user'"'"'s document, and may be recognized to obtain input text data. The summarized text may output as text or as audio pronunciation using a speech synthesizer.

Claims

1. A method for automatically summarizing text, comprising:

(a) obtaining input text data defining a text that includes two or more tokens;

(b1) using the input text data to tokenize the text, the tokenized text including one or more tokenized sentences;

(b2) obtaining part-of-speech (POS) data indicating parts of speech for tokens in the text of each of the tokenized sentences from (b1);

(c) using the POS data for each tokenized sentence to obtain group data for the sentence indicating one or more groups of consecutive tokens of the text and indicating, within each group, any tokens that meet a POS-based removal criterion; and

(d) using the group data for each sentence to obtain summarized text data defining a summarized version of the text for the sentence in which tokens in each group that are indicated as meeting the removal criterion are removed so that the number of tokens in the summarized version of the text for the sentence is less than the number of tokens in the text.

2. The method of claim 1, wherein each tokenized sentence includes, in sequence, two or more word tokens and a sentence-final token, the word tokens each indicating a word of text and the sentence-final token indicating the end of a sentence.

3. The method of claim 1, wherein (a) comprises

(a1) using an image capture device directed upon an image bearing portable medium containing text matter to generate image data representative of the text matter; and

(a2) converting the image data to machine readable text data, the text data being a representation of the text matter, the text data being said input text data.

4. The method of claim 1, further comprising:

(e) converting the summarized text data to audio data, the audio data being a representation of the pronunciation of the words in the summarized text data, and emitting sounds corresponding to said audio data.

5. The method of claim 1, wherein (b), (c), and (d) are performed in one pass through the input text data.

6. A method for automatically summarizing text, comprising:

(a) obtaining input text data defining a text that includes two or more tokens;

(b) using the input text data to obtain part-of-speech (POS) data indicating parts of speech for tokens in the text;

(c) using the POS data to obtain group data indicating one or more groups of consecutive tokens of the text and indicating, within each group, any tokens that meet a POS-based removal criterion; and

(d) using the group data to obtain summarized text data defining a summarized version of the text in which tokens in each group that are indicated as meeting the removal criterion are removed so that the number of tokens in the summarized version of the text is less than the number of tokens in the text;

wherein (c) comprises;

(c1) obtaining first group type data indicating one or more groups of consecutive tokens that have a first word group type, and, within each group having the first word group type, any tokens that meet a first POS-based removal criterion applicable to groups of the first word group type;

(c2) obtaining second group type data indicating one or more groups of consecutive tokens that have a second word group type, and, within each group having the second word group type, any tokens that meet a second POS-based removal criterion applicable to groups of the second word group type.

7. The method of claim 6, wherein said first word group type is a verb group type, and said first group type data include, for each group having the first word group type, a verb group start element preceding the group and a verb group end element following the group.

8. The method of claim 6, wherein said second word group type is a noun group type, and said second group type data include, for each group having the second word group type, a noun group start element preceding the group and a noun group end element following the group.

9. A system for automatically summarizing text, the system comprising:

input text data defining a text that includes two or more tokens; and

a processor connected for accessing the input text data;

the processor automatically summarizing the text;

in automatically summarizing, the processor operating to;

use the input text data to tokenize the text, the tokenized text including one or more tokenized sentences;

obtain part-of-speech (POS) data indicating parts of speech for tokens in the text of each of the tokenized sentences;

use the POS data for each tokenized sentence to obtain group data for the sentence indicating one or more groups of consecutive tokens of the text and indicating, within each group, any tokens that meet a POS-based removal criterion; and

use the group data for each sentence to obtain summarized text data defining a summarized version of the text for the sentence in which tokens in each group that are indicated as meeting the removal criterion are removed so that the number of tokens in the summarized version of the text for the sentence is less than the number of tokens in the text.

10. The system of claim 9, wherein the processor operates to automatically summarize the text in one pass through the input text data.

11. An article of manufacture for use in a system for automatically summarizing text;

the system including;

input text data defining a text that includes two or more tokens;

a storage medium access device; and

a processor connected for receiving data accessed on a storage medium by the storage medium access device and for accessing the input text data;

the article of manufacture comprising;

a storage medium; and

instruction data stored by the storage medium;

the instruction data indicating instructions the processor can execute;

the processor, in executing the instructions, automatically summarizing the text;

in automatically summarizing, the processor operating to;

use the input text data to tokenize the text, the tokenized text including one or more tokenized sentences;

obtain part-of-speech (POS) data indicating parts of speech for tokens in the text of each of the tokenized sentences;

use the POS data for each tokenized sentence to obtain group data for the sentence indicating one or more groups of consecutive tokens of the text and indicating, within each group, any tokens that meet a POS-based removal criterion; and

use the group data for each sentence to obtain summarized text data defining a summarized version of the text for the sentence in which tokens in each group that are indicated as meeting the removal criterion are removed so that the number of tokens in the summarized version of the text for the sentence is less than the number of tokens in the text.

12. The article of manufacture of claim 11, wherein the processor operates to automatically summarize the text in one pass through the input text data.

13. A method for automatically summarizing text, comprising:

(A) receiving a signal from a user input device selecting one of a set of part-of-speech (POS) based removal criteria and obtaining input text data defining a text that includes two or more tokens;

(B1) using the input text data to tokenize the text, the tokenized text including one or more tokenized sentences;

(B2) obtaining POS data indicating parts of speech for tokens in the text of each of the tokenized sentences from (B1); and

(C) using the POS data for each tokenized sentence to obtain summarized text data defining a summarized version of the text for the sentence in which tokens are removed in accordance with the selected POS based criterion so that the number of tokens in the summarized version of the text for the sentence is less than the number of tokens in the text.

14. The method of claim 13 in which (A) comprises:

(A1) displaying on a display device an image showing the set of POS based removal criteria; and

(A2) receiving the signal from the user input device, the signal selecting the selected POS based removal criterion.

15. The method of claim 13, further comprising:

(D) converting the summarized text data to audio data, the audio data being a representation of the pronunciation of the words in the summarized text data, and emitting sounds corresponding to said audio data.

16. The method of claim 13, wherein (A), (B), and (C) are performed in one pass through the input text data.

17. A system for automatically summarizing text, the system comprising:

input text data defining a text that includes two or more tokens; and

a processor connected for accessing the input text data;

the processor automatically summarizing the text;

in automatically summarizing, the processor operating to;

receive a signal from a user input device selecting one of a set of part-of-speech (POS) based removal criteria;

use the input text data to tokenize the text, the tokenized text including one or more tokenized sentences;

obtain POS data indicating parts of speech for tokens in the text of each of the tokenized sentences; and

use the POS data for each tokenized sentence to obtain summarized text data defining a summarized version of the text for the sentence in which tokens are removed in accordance with the selected POS based criterion so that the number of tokens in the summarized version of the text for the sentence is less than the number of tokens in the text.

18. The system of claim 17, wherein the processor operates to automatically summarize the text in one pass through the input text data.

19. An article of manufacture for use in a system for automatically summarizing text;

the system including;

input text data defining a text that includes two or more tokens;

a storage medium access device; and

a processor connected for receiving data accessed on a storage medium by the storage medium access device and for accessing the input text data;

the article of manufacture comprising;

a storage medium; and

instruction data stored by the storage medium;

the instruction data indicating instructions the processor can execute;

the processor, in executing the instructions, automatically summarizing the text;

in automatically summarizing, the processor operating to;

receive a signal from a user input device selecting one of a set of part-of-speech (POS) based removal criteria;

use the input text data to use the input text data to tokenize the text, the tokenized text including one or more tokenized sentences;

obtain POS data indicating parts of speech for tokens in the text of each of the tokenized sentences; and

use the POS data for each tokenized sentence to obtain summarized text data defining a summarized version of the text for the sentence in which tokens are removed in accordance with the selected POS based criterion so that the number of tokens in the summarized version of the text for the sentence is less than the number of tokens in the text.

20. The article of manufacture of claim 19, wherein the processor operates to automatically summarize the text in one pass through the input text data.