Aphasia therapy system

• US 7,203,649 B1
• Filed: 04/15/1998
• Issued: 04/10/2007
• Est. Priority Date: 04/15/1998
• Status: Expired due to Fees

• Current Assignee: Unisys Corporation
• Original Assignee: Unisys Corporation
• Inventors: Romania, John F., Linebarger, Marcia C.
• Primary Examiner(s): ARMSTRONG, ANGELA A
• Application Number:
  US09/060,313
• Time in Patent Office: 3,282 Days
• Field of Search: 704/9, 704/257, 704/270, 704/271, 704/272, 704/275, 434/167, 434/176
• US Class Current: 704/270
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A computer-operated speech therapy system is provided that includes speech input, speech recognition and natural language understanding, and audio and visual outputs to enable an aphasic patient to conduct self-paced speech therapy autonomously. The system of the invention conducts a therapy exercise by displaying a picture; generating a speech prompt asking the patient for information about the picture; receiving the patient'"'"'s speech response and processing it to determine its semantic content; determining whether the patient'"'"'s response was correct; and outputting feedback to the patient. Preferably the system includes a touch screen as a graphical input/output device by which the patient controls the therapy exercise.

Claims

1. A computer-implemented method for conducting speech therapy comprising:

displaying a picture, wherein the picture comprises a plurality of aspects;

generating speech prompts for information describing each of the plurality of aspects of the picture;

inputting speech responses, including user-identification of each of the plurality of aspects,

performing speech recognition on the input speech responses, including the user-identification of each of the plurality of aspects to recognize words comprising the responses;

performing natural language analysis of the recognized words to determine whether the user-identification of each of the plurality of aspects accurately describes the plurality of aspects,

generating a prompt to use each of the plurality of aspects in a sentence if the user-identification of each of the plurality of aspects accurately describes the plurality of aspects,

receiving a sentence from said user responsive to said prompt to use each of the plurality of aspects,

performing natural language analysis, to analyze the semantic content of the sentence for appropriate sentence correctness, and

providing feedback to the user regarding the correctness of the semantic content of the sentence.

2. The method of claim 1, wherein the step of performing natural language analysis includes analyzing a semantic content of the recognized words.

3. The method of claim 1, wherein the step of performing natural language analysis includes judging that a plurality of possible recognized words are synonyms.

4. The method of claim 3, wherein the step of performing natural language analysis includes accepting synonyms for the recognized words.

5. The method of claim 1, wherein the step of performing natural language analysis includes analyzing a grammatical structure of the recognized words.

6. The method of claim 1, wherein the step of performing natural language analysis includes altering a visual characteristic of at least one of the aspects of the picture in response to the input speech response.

7. The method of claim 6, wherein the step of performing natural language analysis includes colorizing at least one of the aspects in response to the input speech response.

8. The method of claim 6, wherein the step of performing natural language analysis includes de-colorizing at least one of the aspects in response to the input speech response.

9. The method of claim 1, further comprising the step of replaying the speech response.

10. The method of claim 1, wherein the step of generating a speech prompt includes providing an auditory cue to the user that is activated by interacting with an icon.

11. The method of claim 10, wherein the step of generating a speech prompt includes providing an auditory cue for one of a verb and a preposition to the user that is activated by interacting with an icon.

12. A system for conducting speech therapy comprising:

a visual display for displaying a picture, the picture comprising a plurality of aspects;

a microphone adapted to capture sounds spoken by a user to describe the plurality of aspects of the picture;

a speaker adapted to output sound in response to the sounds spoken by the user;

a processor including memory coupled to the visual display device and the speaker and receiving the sounds from the microphone, the processor being programmed to;

display the picture, including the plurality of aspects;

generate speech prompts for information describing each of the plurality of aspects of the picture;

receive as inputs, speech responses, including user-identification of each of the plurality of aspects;

perform speech recognition on the input speech responses to recognize words comprising the response, including the user-identification of each of the plurality of aspects;

perform natural language analysis of the recognized words to determine whether the user-identification of each of the plurality of aspect accurately describes the plurality of aspects; and

,

generate a prompt to use each of the plurality of aspects in a sentence if the user-identification of each of the plurality of aspects accurately describes the plurality of aspects,

receive a sentence from said user responsive to said prompt to use each of the plurality of aspects,

perform natural language analysis, to analyze the semantic content of the sentence for appropriate sentence correctness, and

provide feedback to the user regarding the correctness of the semantic content of the sentence.

13. The system of claim 12, wherein the processor is programmed to perform natural language analysis including analyzing a semantic content of the recognized words.

14. The system of claim 12, wherein the processor is programmed to perform natural language analysis including judging that a plurality of possible recognized words are synonyms.

15. The system of claim 14, wherein the processor is programmed to perform natural language analysis including accepting synonyms for the recognized words.

16. The system of claim 12, wherein the processor is programmed to perform natural language analysis including analyzing a grammatical structure of the recognized words.

17. The system of claim 12, wherein the processor is programmed to perform natural language analysis including altering a visual characteristic of at least one of the aspects in response to the input speech response.

18. The system of claim 17, wherein the processor is programmed to perform natural language analysis including colorizing at least one of the aspects in response to the input speech response.

19. The system of claim 17, wherein the processor is programmed to perform natural language analysis including de-colorizing at least one of the aspects in response to the input speech response.

20. The system of claim 12, wherein the processor is programmed to generate a speech prompt that provides an auditory cue to the user that is activated by interacting with an icon.

21. A computer assisted method for conducting speech therapy comprising:

displaying a picture, wherein the picture comprises a plurality of aspects;

generating a prompt for first information describing a first of the plurality of aspects of the picture;

inputting a speech response, wherein the input speech response includes a user-identification of the first aspect;

performing speech recognition on the input speech response, including the user-identified first aspect in order to recognize words comprising the input speech response;

performing natural language analysis of the recognized words to determine whether the user-identified first aspect accurately describes the first aspect;

repeating the steps of prompt generating, speech response inputting, speech recognition performing and natural language analysis performing for each of the remaining plurality of aspects,

generating a prompt for second information if the user-identification of each of the plurality of aspects accurately describes the plurality of aspects, wherein the second information includes a sentence describing the entire picture,

receiving a sentence from said user responsive to said prompt for second information,

performing natural language analysis, to analyze the semantic content of the sentence for appropriate sentence correctness, and

providing feedback to the user regarding the correctness of the semantic content of the sentence.

22. The method of claim 21, wherein said step of generating a prompt for second information is carried out only if each of the user-identifications of the plurality of aspects is determined to accurately describe the plurality of aspects of the picture.

23. The method of claim 22, wherein the step of prompting the second information is carried out a predetermined number of times, and further comprising the step of:

generating a prompt for a different sentence, the prompt being generated if, after the predetermined number of times, at least one of the user-identified aspects is determined not to accurately describe the respective aspect of the picture.

24. The system of claim 23, wherein the different sentence prompt further includes an indication of the user-identified aspects determined to accurately describe the plurality of aspects of the picture.

25. The method of claim 21, wherein the step of generating a prompt for first information and the step of generating a prompt for second information each include a speech prompt.

26. The method of claim 24, wherein the steps of generating a prompt for first information, generating a prompt for second information, and generating a different sentence prompt each include a visual prompt.

27. The method of claim 21, further comprising the step of:

generating a first tone if the user-identification of the first aspect is accurate; and

,

generating a second tone if the user-identification of the first aspect is not accurate.

28. A system for conducting speech therapy, comprising:

a display for displaying a picture, the picture comprising a plurality of aspects,

input means for receiving a spoken description of each of the plurality of aspects of the picture by a user;

output means adapted to output a response to the spoken user description; and

,

processing unit means coupled to the display means, the input means, and the output means, the processing unit means being programmed to;

analyze the spoken user description of each of the plurality of aspects, to determine whether the spoken user description of each of the aspects accurately describes the respective aspect of the picture, and,

generate a prompt for the user to use each of the plurality of aspects in a sentence describing the picture if the spoken user description of each of the plurality of aspects accurately describes the respective aspect of the picture,

receive a sentence from said user responsive to said prompt to use each of the plurality of aspects,

perform natural language analysis, to analyze the semantic content of the sentence for appropriate sentence correctness, and

provide feedback to the user regarding the correctness of the semantic content of the sentence.

29. The system of claim 28, wherein said processing unit means includes:

a processor; and

,

a memory coupled to the processor, the memory storing;

a speech recognizer for performing speech recognition on the spoken user description in order to recognize words included in the spoken user description; and

,

a natural language analyzer for receiving the recognized words from the speech recognizer and for comparing the recognized words with pre-defined acceptable words describing the respective aspect of the picture, wherein the spoken user description of the aspect accurately describes the respective aspect of the picture if the recognized words match any of the pre-defined acceptable words.

30. The system of claim 29, wherein the prompt to use each of the plurality of aspects in a sentence describing the picture is a speech prompt.

31. The system of claim 29, wherein the system is implemented in a computer and the display is a computer monitor.

32. The system of claim 28, wherein the input means includes a microphone.

33. The system of claim 28, wherein the input means includes a speaker.