Method and device for enhancing the recognition of speech among speech-impaired individuals

• US 6,123,548 A
• Filed: 04/09/1997
• Issued: 09/26/2000
• Est. Priority Date: 12/08/1994
• Status: Expired due to Term

• Current Assignee: Rutgers University, Regents of the University of California (University of California)
• Original Assignee: Rutgers University, Regents of the University of California (University of California)
• Inventors: Miller, Steven Lamont, Jenkins, William Michael, Schreiner, Christoph E., Tallal, Paula Anne, Merzenich, Mathias
• Primary Examiner(s): Cheng, Joe H.
• Application Number:
  US08/773,691
• Time in Patent Office: 1,266 Days
• Field of Search: 434/118, 434/156, 434/169, 434/185, 434/307 R, 434/308, 434/309, 434/323, 434/362, 434/365, 704/1, 704/9, 704/200, 704/211, 704/241-261, 704/213, 381/34
• US Class Current: 434/185
• CPC Class Codes: G09B 19/04 Speaking with audible presentation of the material to be studied G09B5/04
• CPC Class Codes: G09B 5/065 Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems
• CPC Class Codes: G10L 21/0364 for improving intelligibility
• CPC Class Codes: G11B 31/00 Arrangements for the associated working of recording or reproducing apparatus with related apparatus with cameras or projectors G03B31/00 ; recording/reproducing of music for electrophonic musical instruments G10H1/0033; automatic arrangements for answering calls or for recording messages for absent subscribers H04M1/64; telephonic communication systems adapted for combination with dictation recording and playback systems H04M11/10; connection of TV recorder with other related apparatus, e.g. TV camera or receiver, in which the TV signal is significantly involved H04N, e.g. H04N23/00, H04N5/765; combination of radio or TV with other apparatus, e.g. with vehicles H05K11/00
• CPC Class Codes: H04R 2225/43 Signal processing in hearing aids to enhance the speech intelligibility

Abstract

A method and a system is disclosed that provide means to enable individuals with speech, language and reading based communication disabilities, due to a temporal processing problem, to improve their temporal processing abilities as well as their communication abilities. The method and system include provisions to elongate portions of phonemes that have brief and/or rapidly changing acoustic spectra, such as occur in the stop consonants b and d in the phonemes /ba/ and /da/, as well as reduce the duration of the steady state portion of the syllable. In addition, some emphasis is added to the rapidly changing segments of these phonemes. Additionally, the disclosure includes method for and computer software to modify fluent speech to make the modified speech better recognizable by communicatively impaired individuals. Finally, the disclosure includes method for and computer software to train temporal processing abilities, specifically speed and precision of temporal integration, sequencing and serial memory.

Claims

1. In an interactive computer system including a processor, an input device, an output and a sound generator, a method for training an individual user over the course of more than one session to discriminate between two sound stimuli, the method comprising:

providing, by the computer system output to the individual user, a series of presentations of the two sound stimuli separated from each other by an interstimulus interval (ISI);

receiving by the computer input device, as the series of presentations is provided, user inputs wherein each received user input is responsive to at least one presentation in the series and wherein each such received user input provides an indication of the individual user'"'"'s ability to discriminate between the two stimuli; and

decreasing, by the computer system, the ISI between the two sound stimuli in the random presentations as the individual'"'"'s ability to discriminate between the two sound stimuli improves.

2. The method of claim 1 wherein each such received user input represents the individual user'"'"'s attempt to discriminate between the two stimuli of the at least one presentation to which such received user input is responsive.

3. The method of claim 2 and further including:

immediately providing, by the computer system to the individual user, a sound associated with a correct discrimination when the individual user correctly discriminates between the two sound stimuli so as to reinforce the correct discrimination for the individual user.

4. The method of claim 2 and further including:

immediately providing, by the computer system to the individual user, an animation associated with a correct discrimination when the individual user correctly discriminates between the two sound stimuli so as to reinforce the correct discrimination for the individual user.

5. The method of claim 1 and further including:

decreasing, by the computer system, during at least one training session the ISI between the two sound stimuli in the random presentations as the series of presentations is provided if the individual user'"'"'s ability to discriminate between the two sound stimuli improves during the at least one training session.

6. The method of claim 1 and further including:

decreasing, by the computer system, the ISI between the two sound stimuli in the random presentations from one session to the next if the individual'"'"'s ability to discriminate between the two sound stimuli improves from one session to the next.

7. The method of claim 1 wherein the individual user'"'"'s ability to discriminate between the two different sound stimuli is measured in terms of the individual user'"'"'s ability to discriminate the order in which the two different stimuli are provided.

8. The method of claim 1, wherein the two sound stimuli include a first sound stimulus and a second sound stimulus;

and

wherein a first type presentation in the series includes the first sound stimulus followed by the second sound stimulus in which the first and second sound stimuli are separated by the ISI;

wherein a second type of presentation in the series includes the second sound stimulus followed by the first sound stimulus in which the second and first sound stimuli are separated by the ISI;

wherein a third type of presentation in the series includes two occurrences of the first sound stimulus separated from each other by the ISI; and

wherein a fourth type of presentation in the series includes two occurrences of the second sound stimulus separated from each other by the ISI.

9. The method of claim 1 wherein each sound stimulus includes a frequency modulated sweep.

10. The method of claim 1 wherein each sound stimulus includes a tonal stimulus.

11. The method of claim 1 wherein each sound stimulus includes a complex tonal stimulus.

12. The method of claim 11 wherein the at least two speech portions include a consonant and a vowel.

13. The method of claim 1 wherein each sound stimulus includes a frequency modulated sweep plus a tonal stimulus.

14. The method of claim 1 wherein the each sound stimulus includes a frequency modulated sweep plus a complex tonal stimulus.

15. The method of claim 1, wherein each sound stimulus includes a speech sound.

16. The method of claim 1, wherein each sound stimulus includes a portion of a speech sound.

17. The method of claim 1 wherein the two sound stimuli are presented in random order in the series of presentations.

18. In an interactive computer system including a processor, an input device, an output and a sound generator, a method for training an individual user over the course of more than one training session to recognize a speech sound which includes at least two speech sound portions, the method comprising:

providing, by the computer system output to the individual user, a series of presentations that include the at least two speech sound portions separated from each other by an interstimulus interval (ISI);

receiving by the computer input device, as the series of presentations is provided, a series of user inputs wherein each user input is responsive to at least one presentation in the series and wherein each such received user input provides an indication of the individual user'"'"'s ability to discriminate between the two different speech sound portions; and

decreasing, by the computer system, the ISI between the at least two speech sound portions in the presentations as the individual user'"'"'s ability to discriminate between the two speech sound portions improves.

19. The method of claim 18 wherein each such received user input represents the individual user'"'"'s attempt to discriminate between the two speech sounds of the at least one presentation to which such received user input is responsive.

20. The method of claim 19 and further including:

immediately providing, by the computer system to the individual user, a sound associated with a correct discrimination when the individual user correctly discriminates between the two speech sound portions so as to reinforce the correct discrimination for the individual user.

21. The method of claim 19 and further including:

immediately providing, by the computer system to the individual user, an animation associated with a correct discrimination when the individual user correctly discriminates between the two sound portions so as to reinforce the correct discrimination for the individual user.

22. The method of claim 18 and further including:

decreasing by the computer system, during at least one training session, the ISI between the two speech sounds in the presentations as the presentations are provided if the individual user'"'"'s ability to discriminate between the two speech sound portions improves during the at least one given session.

23. The method of claim 18 and further including:

decreasing, by the computer system, the ISI between the two speech sound portions in the presentations from one session to the next if the individual user'"'"'s ability to discriminate between the two speech sound portions improves from one session to the next.

24. The method of claim 18 wherein the at least two sound portions are presented in random order in the series of presentations.

25. In an interactive computer system including a processor, an input device, an output, and a sound generator, a method for training an individual user over the course of more than one training sessions to recognize a speech sound, the method comprising:

repeatedly providing, by the computer system output, the speech sound to the individual user;

emphasizing, by the computer system, the energy level of one portion of the provided speech sound relative to another portion of the provided speech sound;

as the speech sound is repeatedly provided, receiving by the computer input device, a series of user inputs wherein each user input is responsive to at least one occurrence of a provided speech sound and wherein each such user input provides an indication of the individual user'"'"'s ability to identify the provided speech sound; and

decreasing, by the computer system, the relative energy level emphasis of the one portion of the provided speech sound as the individual user'"'"'s ability to identify the speech sound improves.

26. The method of claim 25 wherein each such received user input represents the individual user'"'"'s attempt to identify the provided speech sound of the at least one occurrence to which such received user input is responsive.

27. The method of claim 26 and further including:

immediately providing, by the computer system to the individual user, a sound associated with a correct identification when the individual user correctly identifies the speech sound so as to reinforce the correct identification for the individual user.

28. The method of claim 26 and further including:

immediately providing, by the computer system to the individual user, an animation associated with a correct identification when the individual user correctly identifies the speech sound so as to reinforce the correct identification for the individual user.

29. The method of claim 25 wherein the emphasized portion of the speech sound has a rapid temporal change in frequency content.

30. The method of claim 25 wherein the emphasized portion of the speech sound is a consonant.

31. The method of claim 25 wherein the emphasized portion of the speech sound is a consonant and the other portion of the speech sound is a vowel.

32. The method of claim 25 and further including:

decreasing by the computer system, during at least one training session, the energy level emphasis of the portion of the repeatedly provided speech sound as the speech sound is repeatedly provided, if the individual user'"'"'s ability to identify the speech sound improves during the at least one session.

33. The method of claim 25 and further including:

decreasing, by the computer system, the energy level emphasis of the emphasized portion of the repeatedly provided speech sound from one session to the next if the individual user'"'"'s ability to identify the repeatedly provided speech sound improves from one session to the next.

34. The method of claim 25 wherein the emphasis of the emphasized portion of the repeatedly provided speech sound increases the relative salience of the emphasized speech sound portion to promote learning the speech sound by the individual user.

35. In an interactive computer system including a processor, an input device, an output and a sound generator, a method for training an individual user over the course of more than one training session to recognize a speech sound, the method comprising:

repeatedly providing, by the computer system output, the speech sound to the individual user;

emphasizing, by the computer system, the energy level of one portion of the provided speech sound relative to another portion of the provided speech sound;

detecting, by the computer system using the input device, whether the individual user is able to consistently recognize the provided speech sound; and

if the individual user is able to consistently recognize the provided speech sound, decreasing, by the computer system, the relative emphasis of the one portion of the provided speech sound and repeating the provided speech sound with the decreased relative emphasis at least until the individual user can consistently recognize the provided speech sound with the decreased emphasis.

36. In an interactive computer system including a processor, an input device, an output and a sound generator, a method for training an individual user over the course of more than one training session to recognize a speech sound which includes at least two speech sound portions, the method comprising:

repeatedly providing, by the computer system output, the speech sound to the individual user;

prolonging, by the computer system, at least one of the speech sound portions of the repeatedly provided speech sound;

receiving by the computer input device, as the speech sound is repeatedly provided, a series of user inputs wherein each user input is responsive to at least one occurrence of the speech sound and wherein each such received user input provides an indication of the individual user'"'"'s ability to identify the provided speech sound and

decreasing, by the computer system, the length of the prolonged speech sound portion of the repeatedly provided speech sound as the individual user'"'"'s ability to identify the speech sound improves.

37. The method of claim 36 wherein each such received user input represents the individual user'"'"'s attempt to identify the provided speech sound of the at least one occurrence to which such received user input is responsive.

38. The method of claim 37 and further including:

immediately providing, by the computer system to the individual user, a sound associated with a correct identification of the speech sound when the individual user correctly identifies the speech sound so as to reinforce the correct identification for the individual user.

39. The method of claim 37 and further including:

immediately providing, by the computer system to the individual user, an animation associated with a correct identification when the individual user correctly identifies the speech sound so as to reinforce the correct identification for the individual user.

40. The method of claim 36 and further including:

decreasing by the computer system, during at least one training session, the length of the prolonged portion of the repeatedly provided speech sound as the speech sound is repeatedly provided if the individual user'"'"'s ability to identify the repeatedly provided speech sound improves during the at least one session.

41. The method of claim 36 and further including:

decreasing, by the computer system, the length of the prolonged portion of the repeatedly provided speech sound from one session to the next if the individual user'"'"'s ability to identify the repeatedly provided speech sound improves from one session to the next.

42. The method of claim 36 wherein the prolonged portion of the speech sound has a rapid temporal change in frequency content.

43. The method of claim 36 wherein the prolonged portion of the speech sound is a consonant.

44. In an interactive computer system including a processor, an input device, an output and a sound generator, a method for training an individual user over the course of more than one training session to recognize a speech sound which includes at least two portions, the method comprising:

repeatedly providing, by the computer system output, the speech sound to the individual user;

prolonging, by the computer system, at least one portion of the repeatedly provided speech sound;

detecting by the computer system using the input device, whether the individual user is able to consistently identify the provided speech sound; and

if the individual user is able to consistently identify the provided speech sound, decreasing, by the computer system, the length of the prolonged portion of the provided speech sound and repeating the provided speech sound with the shortened length of the prolonged portion of the provided speech sound at least until the individual user can consistently identify the provided speech sound with the shortened length.

45. In an interactive computer system including a processor, an input device, an output and a sound generator, a method for training an individual user to recognize a speech sound, the method comprising:

repeatedly providing, by the computer system output, the speech sound to the individual user;

differentially emphasizing, by the computer system, the energy level of a portion of the provided speech sound relative to another portion of the provided speech sound; and

receiving by the computer input device, as the speech sound is repeatedly provided, a series of user inputs wherein each received user input is responsive to at least one occurrence of the provided speech sound and wherein each such received user input provides an indication of the individual user'"'"'s ability to identify the provided speech sound and

repeatedly providing the speech sound to the individual user at least until the individual user learns to identify it.

46. The method of claim 45 wherein each such received user input represents the individual user'"'"'s attempt to identify the provided speech sound of the at least one occurrence to which such received user input corresponds.

47. The method of claim 45 wherein the differentially emphasized portion of the speech sound includes a rapid temporal change in frequency content.

48. The method of claim 45 wherein the differentially emphasized portion of the speech sound is a consonant.

49. The method of claim 45 wherein the differentially emphasized portion of the speech sound is a consonant and the other portion of the speech sound is a vowel.

50. The method of claim 45 wherein the differentially emphasized portion of the speech sound is a consonant and the differential emphasis increases the relative salience of the emphasized consonant to the individual user to promote learning the emphasized consonant by the individual user.