Method and apparatus for text analysis

• US 4,773,009 A
• Filed: 06/06/1986
• Issued: 09/20/1988
• Est. Priority Date: 06/06/1986
• Status: Expired due to Term

• Current Assignee: Vantage Technology Holdings LLC
• Original Assignee: Houghton Mifflin Harcourt Publishing Company (Houghton Mifflin Harcourt Co. (Private Equity))
• Inventors: Russom, Jacqueline, Sokolowski, Rachael, Kucera, Henry
• Primary Examiner(s): Smith, Jerry
• Assistant Examiner(s): Hayes, Gail O.
• Application Number:
  US06/872,094
• Time in Patent Office: 837 Days
• Field of Search: 364/419, 364/900, 364/200, 340/39, 340/40, 400/63, 400/83
• US Class Current: 715/260
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique

Abstract

An electronic text analyzer operates on an ordered block of digitally coded text by analyzing sequential strings thereof to determine paragraph and sentence boundaries. Each string is broken down into component words. Possible abbreviations are identified and checked against a table of common abbreviations to identify abbreviations which cannot end a sentence. End punctuation and the following string are analyzed to identify the terminal word of a sentence. When sentence boundaries have been determined, the test may be further processed by a grammar checker, a readability analyzer, or other higher-level text processing system.

A preferred embodiment includes a readability analyzer having a syllable counter for determining the number of syllables in each word. The system includes a modified common-word table having an empirical syllable-count field. A checker first determines if a word is in the table and, if so, returns its syllable count. An exception table identifies words not conforming to a syllable counting algorithm. Each word not in the common-word or exception tables is modified, and the modified word is processed to derive its syllable count. In a preferred embodiment, tallies are kept of words per sentence, syllable count, sentences per paragraph, and similar data, and readability scores based on the tallies are displayed.

Claims

1. An electronic system for the analysis of digitally encoded text, such text including sequential strings of characters separated by spaces, such system comprising

buffer means for receiving and storing coded text,

means for examining each successive character sequence of the text to determine word and punctuation portions thereof and identify each character sequence having a terminal period,

abbreviation table means for storing a table of common abbreviations with an indication if an abbreviation is of a non-terminating type,

look-up means for checking a character sequence having a terminal period against the abbreviation table and determining its type, and sentence boundary determining means in communication with the look-up means and operative on successive character sequences for determining whether a sentence boundary occurs at an occurrence of a terminal period and wherein a discrimination means identifies and diverts from the sentence boundary determining means said non-terminating abbreviations.

2. A system according to claim 1, further including

paragraph boundary determining means operative solely on white-space code, for determining a paragraph boundary of the text.

3. A system according to claim 2, further including

comparator means responsive to the paragraph means and to the sentence boundary determining means, for identifying as a text fragment a block of coded text having a paragraph boundary preceding a sentence boundary.

4. A system according to claim 3, further including

means for generating counts of basic text units from the coded text excluding text fragments, and for developing a readability output representative thereof.

5. A system according to claim 4, further including

word table means, for storing a Dale-Chall table of common words together with an indication of the syllable count of a word, and wherein the counts of basic text units include at least one of word length in syllables and occurrences of familiar words in the text.

6. A system according to claim 5, further including

means for displaying an interpretive message with the readability output.

7. A system according to claim 6, wherein the readability output includes a linear function of plural basic text units.

8. A system according to claim 7, wherein a readability output includes a Bormuth Reading Power Score.

9. A system according to claim 1, further including

means for generating counts of basic text units from coded text and for developing a readability output representative thereof.

10. A system according to claim 9, further including

word table means, for storing a Dale-Chall table of common words together with an indication of the syllable count of a word, and wherein the basic text units include at least one of word length in syllables and occurrences of familiar words in the text.

11. A system according to claim 10, further including

means for displaying an interpretive message with the readability output.

12. A system according to claim 11 wherein the readability output includes a linear function of plural basic text units.

13. A system according to claim 11, wherein a readability output includes a Bormuth Reading Power Score.

14. A system for the analysis of digitally encoded text including sequential strings of characters separated by white space, such system comprising

means for receiving and storing digitally encoded text

word table means for storing a table of common words together with a syllable count indication for a stored word

word identifier means for examining successive strings of coded text and identifying each word therein, and

syllable counter means for determining the syllable count of each identified word, wherein the syllable counter means includes means for checking an identified word against the word table and returning, for a word stored therein, the syllable count indication.

15. A system according to claim 14, wherein the syllable counter means further includes

means for performing an ordered sequence of context-dependent substitutions of coded characters in an identified word to determine a modified word, and

means for counting occurrences of a discrete number of coded characters in the modified word to obtain the said syllable count.

16. A system according to claim 15, further including

readability means for measuring the readability of the digitally encoded text material.

17. A system according to claim 16, wherein a syllable count indication for a word in the table includes an indication that the word is an exception to a context-dependent substitution.

18. A system according to claim 17, further including

means operative on the digitally encoded text for separating text body from text fragments, and wherein the readability means measures the readability of the text body.

19. A system according to claim 16, further including

sentence splitter means for determining sentence boundaries in the text information, and wherein the readability means includes means for measuring sentence data including at least one of sentence length in words, and number of sentences per paragraph.

20. A system according to claim 16, further including

means for displaying an output indicative of the measured readability.

21. A system according to claim 20, wherein the output indicative of measured readability includes an interpretive message.

22. A method for analyzing digitally encoded text, such text including sequential strings of characters separated by spaces, the method comprising the steps of:

receiving and storing coded text,

storing a table of abbreviations together with an indication of non-terminating abbreviations,

examining in sequence character strings of the text to recognize a string including a period,

checking a recognized character string against the abbreviation table to identify non-terminating abbreviations,

operating on successive character strings not so identified to determine a sentence boundary,

identifying successive white space codes in the text, and

comparing the identified codes to a nominal document spacing to determine paragraph boundaries of the text.

23. A method according to claim 22 further including the step of

identifying as a text fragment text having a paragraph boundary as determined by said comparing step occurring before a sentence boundary as determined by said operating step.

24. A method according to claim 23, further including the steps of

generating measures of the text between sentence boundaries, and

developing a readability output as a function of the generated measures.

25. A method of claim 24, further including the step of

storing a table of common words together with an indication of the syllable count of a word, and wherein a measure of the text includes at least one of word length in syllables and a measure of familiar words occurring in the text.

26. A method according to claim 25, further including the step of

displaying an interpretive message with the readability output.

27. A method according to claim 26, wherein the readability output includes a linear function of at least one of the measures.

28. A method according to claim 26, wherein a readability output includes a Bormuth reading power score.

29. A method according to claim 22, further including the steps of

generating measures of the text between sentence boundaries, and

developing a readability output as a function of the generated measures.

30. A method of claim 29, further including the step of

storing a table of common words together with an indication of the syllable count of a word, and wherein a measure of the text includes at least one of word length in syllables and a measure of familiar words occurring in the text.

31. A method according to claim 30, further including the step of

displaying an interpretive message with the readability output.

32. A method according to claim 31, wherein the readability output includes a linear function of at least one of the measures.

33. A method according to claim 31, wherein a readability output includes a Bormuth reading power score.

34. A method for analyzing digitally encoded text including sequential strings of characters separated by white space, the method comprising the steps of

receiving and storing digitally encoded text

storing a table of words together with a syllable count indication for a stored word

examining successive strings of coded text and identifying each word therein, and

checking an identified word against the table to determine its syllable count indication thereby determining the syllable count of the identified word.

35. A method according to claim 34, further including the steps of

performing an ordered sequence of context-dependent substitutions of coded characters in the identified word to determine a modified word, and

counting occurrences of specific coded characters in the modified word to obtain a syllable count of the identified word.

36. A method according to claim 35, further including the step of

computing a function of the syllable count of words in the text to determine the readability of the digitally encode material.

37. A method according to claim 36, wherein a syllable count indication for a word in the table includes an indication of a context-dependent substitution applicable to the word.

38. A method according to claim 36, further including the steps of

distinguishing a text fragment from text body, and wherein the step of computing a function includes computing such function of the syllable count of the words in the text body.

39. A method according to claim 36, further including the steps of

determining sentence boundaries in the coded text, and

determining the readability as a function of sentence data, such data including at least one of sentence length in words, and number of sentences per paragraph.

40. A method according to claim 36, further including the step of

displaying an output indicative of the readability.

41. A method according to claim 40, wherein the step of displaying includes displaying an interpretive message.

42. An electronic system for the analysis of digitally encoded text, such encoded text including white-space code and character code, such system comprising

sentence splitting means operating on successive strings of encoded text for determining sentence boundaries of the text

paragraph identifier means operative solely on white-space code of the text for determining paragraph boundaries

text fragment identifier means for identifying text fragments by comparing the sentence boundaries and the paragraph boundaries, and identifying as a text fragment text having a paragraph boundary preceding a sentence boundary.