Sentence level analysis in a reading tutor

• US 9,520,068 B2
• Filed: 09/10/2004
• Issued: 12/13/2016
• Est. Priority Date: 09/10/2004
• Status: Active Grant

• Current Assignee: Scientific Learning Corp. (Carnegie Learning, Inc.)
• Original Assignee: JTT Holdings Inc.
• Inventors: Beattie, Valerie L., Adams, Marilyn Jager
• Primary Examiner(s): Serrou, Abdelali
• Application Number:
  US10/938,762
• Publication Number:
  US 20060069558A1
• Time in Patent Office: 4,477 Days
• Field of Search: 704/251, 704/231, 704/235, 704/257
• US Class Current: 1/1
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A method and related system, computer program product and device for interactively tracking oral reading of text from a document includes recording audio for a sentence read by a user and determining when the user has reached the last word of the sentence. The method also includes providing visual feedback to the user reading on a sentence by sentence level to indicate a current location in the passage.

Claims

1. A computer implemented method comprising:

interactively tracking by a computer oral reading of text from an electronic representation of a document;

causing by the computer text of the electronic representation of the document to be displayed on a display device associated with the computer, with the text of the document having a first appearance that includes a first color and first style, and with at least first and second sentences of the document displayed on the display device in the first appearance;

generating by the computer, a first visual feedback indicium over all text of the first sentence displayed on the display device to indicate a current first sentence to be read aloud, while the second sentence is displayed having the first appearance;

receiving by the computer audio corresponding to the first sentence as read aloud by the user;

applying by the computer speech recognition to convert the received audio into a text file;

analyzing by the computer the user'"'"'s fluency in reading of the first sentence using results of speech recognition processing of the audio to determine when the user has read aloud the first sentence with sufficient fluency to advance to the second sentence;

determining by the computer whether an intervention is required for a word in the first sentence;

rendering on the display device by the computer a visual intervention as a highlight on the word in the first sentence that requires the intervention, and with words in the first sentence, which are prior to the word that requires the intervention, in a second color different from the first visual feedback indicium, the word that required the intervention rendered with the highlight and with the highlighted word and words in the first sentence subsequent to the word rendered in the same color that is different from the second color; and

concurrently causing by the computer all of the displayed text in the first sentence to return to the first appearance and the first visual feedback indicium to be rendered over all text of the second sentence displayed on the display device to indicate a next current sentence to be read aloud by the user.

2. The computer implemented method of claim 1 wherein the same color is the first color.

3. The computer implemented method of claim 2 wherein analyzing by the computer further comprises:

determining by the computer correctness of the pronunciation of the determined word using an acoustic model that represents a speech pattern suited to the user.

4. The computer implemented method of claim 1 wherein analyzing by the computer further comprises:

assessing by the computer the quality of the user'"'"'s reading on a word-by-word basis.

5. The computer implemented method of claim 2 wherein when analyzing by the computer indicates an error in pronunciation or timing of the word that requires the intervention, the method further comprises:

providing by the computer pronunciation and timing indications on the word that requires the intervention.

6. The computer implemented method of claim 1 further comprises:

generating by the computer an intervention that assists the user in reading the text in the sentence when a correct reading of the sentence is not received.

7. The computer implemented method of claim 2 wherein determining where the user is within the first sentence includes recognizing by the computer, the user re-starting reading of the sentence.

8. The computer implemented method of claim 2 wherein determining where the user is within the first sentence includes recognizing by the computer, the user repeating words.

9. The computer implemented method of claim 2 wherein determining where the user is within the first sentence includes recognizing by the computer, the user skipping words.

10. The computer implemented method of claim 1 further comprising:

determining by the computer that the user is skipping reading of sentences.

11. The computer implemented method of claim 1 wherein the first visual feedback indicium is provided by displaying text in the first sentence in a first color and surrounding text of surrounding sentences in a second different color.

12. The computer implemented method of claim 1 further comprising:

rendering on the display a third sentence from the document, which third sentence precedes the first sentence that has the first visual feedback indicium, and with the third sentence and the second sentence displayed in the first appearance while the first sentence is displayed with the first visual feedback indicium.

13. The computer implemented method of claim 1 wherein sufficient fluency is the first sentence being read aloud correctly according to the analyzing.

14. The computer implemented method of claim 1 wherein a first language model is used for the first sentence, the method further comprising switching the first language model used for a subsequent sentence to a second, different language model when the end of the first sentence is reached.

15. A hardware device encoded with a computer program product, the computer program product comprising instructions for interactively tracking oral reading of text from an electronic representation of a document, with the computer program product that when executed by an electrical device cause the electrical device to:

cause text of the electronic representation of the document to be displayed on a display device associated with the computer, with the text of the document having a first appearance that includes a first color and first style, and with at least first and second sentences of the document displayed on the display device in the first appearance;

generate a first visual feedback indicium to apply on all text in first the sentence when the document is displayed on a display device to indicate a current first sentence to be read;

receive a text file from speech recognition processing that converted the oral reading of the first sentence into text;

analyze the user'"'"'s fluency in reading of the first sentence using the text file to determine when the user has read the first sentence with sufficient fluency to advance to the second sentence;

determine whether an intervention is required for a word in the first sentence based on a length of time elapsed from a correct pronunciation of a previous word in the first sentence;

render on the display device a visual intervention indicium on the word in the first sentence that required the intervention, with words in the first sentence that precede the word that required the intervention rendered in a second color, and the word that required the intervention rendered with the first visual feedback indicium that is different than the second color, and with a highlight overlaid over the word that required the intervention, and with words in the first sentence subsequent to the word, rendered in the same color that is different from the second color; and

concurrently cause all of the displayed text in the first sentence to return to the first appearance and cause the first visual feedback indicium to be rendered over all text of the second sentence displayed on the display device to indicate a next current sentence to be read by the user.

16. The device of claim 15 further comprising instructions for causing an electrical device to:

remove the visual intervention indicium after removal of the intervention.

17. The device of claim 15 further comprising instructions for causing an electrical device to provide pronunciation and timing indications on a word-by-word basis.

18. The device of claim 15 further comprising instructions for causing an electrical device to assess the quality of the user'"'"'s reading on a word-by-word basis.

19. The device of claim 15 further comprising instructions for causing an electrical device to:

render on the display device a third sentence from the document, which third sentence precedes the first sentence that has the first visual feedback indicium, and with the third sentence and the second sentence displayed in the first appearance while the first sentence is displayed with the first visual feedback indicium.

20. A device comprising:

a processor device;

memory in communication with the processor device;

a display device operatively coupled to the processor and memory; and

a storage medium storing a computer program product to configure the processor to;

render text of an electronic representation of a document on the display device, with the text of the document having a first appearance that includes a first color and first style, and with at least first and second sentences of the document displayed on the display device in the first appearance;

apply a first visual feedback indicium over all text of the first sentence displayed on the display device to indicate a current first sentence to be read, while having the second sentence displayed in the first appearance;

receive audio from a user reading the first sentence aloud;

determine, using speech recognition processing that converts the received audio for the first sentence a text file, when the user has reached a last word of the sentence;

concurrently remove the first visual feedback indicium from the first sentence, return all of the displayed text of the first sentence to the first appearance and apply the first visual feedback indicium over all words in the second sentence after the user completes the last word of the first sentence;

apply the first visual feedback indicium on the second sentence displayed on the display device, while the first sentence is displayed on the display device without the first visual feedback indicium; and

generate in the presence of an intervention a visual intervention indicium for the word that required the intervention in the second sentence, by rendering the words in the second sentence that are prior to the word that required the intervention in a second color, the word that required the intervention rendered in a different color from the second color and with a highlight on the word that required the intervention, and with words in the second sentence subsequent to the word rendered in the different color.

21. The device of claim 20 further configured to determine during reading of the first sentence what word the user is reading within the first sentence.

22. The device of claim 20 further configured to assess the quality of the user'"'"'s reading on a word-by-word basis.

23. The device of claim 20 further configured to:

analyze the user'"'"'s fluency in reading of the first sentence using results of speech recognition processing of the audio, which results include the text file; and

provide pronunciation and timing indications on a word-by-word basis when the processor indicates an error in pronunciation or timing of words in the first sentence.

24. The device of claim 20 further configured to:

render on the display a third sentence of the document that after the first sentence and the second sentence, with the second sentence having the first visual feedback indicium, and the first and third sentences displayed with the first appearance and without the first visual feedback indicium.

25. A device comprising:

a processor device;

memory in communication with the processor device;

a display device operatively coupled to the processor and memory; and

a hardware storage device storing a computer program, the computer program comprising instructions for interactively tracking oral reading of text from an electronic representation of a document, with the computer program product that when executed by an electrical device cause the electrical device to;

cause text of the electronic representation of the document to be displayed on a display device associated with the computer, with the text of the document having a first appearance that includes a first color and first style, and with at least first and second sentences of the document displayed on the display device in the first appearance;

generate a first visual feedback indicium to apply on all text in first the sentence when the document is displayed on a display device to indicate a current first sentence to be read;

record audio for the first sentence as read by the user;

receive a text file from speech recognition processing that converts the recorded audio of the first sentence into text;

analyze the user'"'"'s fluency in reading of the first sentence using the text file to determine when the user has read the first sentence with sufficient fluency to advance to the second sentence;

determine whether an intervention is required for a word in the first sentence based on a length of time elapsed from a correct pronunciation of a previous word in the first sentence;

render on the display a visual intervention indicium on the word in the first sentence that required the intervention, with words in the first sentence that precede the word that required the intervention rendered in the first color, the word that required the intervention rendered in a second, different color and with a highlight overlaid over the word that required the intervention, and with words in the first sentence subsequent to the word, rendered in the same color that is different from the second color; and

concurrently cause all of the displayed text in the first sentence to return to the first appearance and cause the first visual feedback indicium to be rendered over all text of the second sentence displayed on the display device to indicate a next current sentence to be read by the user.

26. The device of claim 25 wherein the computer program further comprises instructions for causing an electrical device to:

apply the second, different color to words in the first sentence that are subsequent to the word that required the intervention.