SYSTEM AND METHOD FOR EXPRESSIVE LANGUAGE, DEVELOPMENTAL DISORDER, AND EMOTION ASSESSMENT

• US 20090191521A1
• Filed: 01/23/2009
• Published: 07/30/2009
• Est. Priority Date: 09/16/2004
• Status: Active Grant

• Current Assignee: LENA Foundation
• Original Assignee: Infoture, Inc.
• Inventors: Paul, Terrance D., Yapanel, Umit, Xu, Dongxin D., Gray, Sharmistha S., Gilkerson, Jill S., Richards, Jeffrey A.
• Granted Patent: US 9,240,188 B2
• Time in Patent Office: Days
• US Class Current: 434/169
• CPC Class Codes: A61B 2503/06 Children, e.g. for attention deficit diagnosis
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/075 supervised, i.e. under machine guidance
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 17/16 Hidden Markov models [HMM]
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2015/022 Demisyllables, biphones or triphones being the recognition units
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/66 for extracting parameters related to health condition detecting or measuring for diagnostic purposes A61B5/00

Abstract

In one embodiment, the system and method for expressive language development; a method for detecting autism in a natural language environment using a microphone, sound recorder, and a computer programmed with software for the specialized purpose of processing recordings captured by the microphone and sound recorder combination; and the computer programmed to execute a method that includes segmenting an audio signal captured by the microphone and sound recorder combination using the computer programmed for the specialized purpose into a plurality recording segments. The method further includes determining which of the plurality of recording segments correspond to a key child. The method also includes extracting acoustic parameters of the key child recordings and comparing the acoustic parameters of the key child recordings to known acoustic parameters for children. The method returns a determination of a likelihood of autism.

Claims

1. A method for detecting autism in a natural language environment using a microphone, sound recorder, and a computer programmed with software for the specialized purpose of processing recordings captured by the microphone and sound recorder combination, the computer programmed to execute a method comprising:

(a) segmenting an audio signal captured by the microphone and sound recorder combination using the computer programmed for the specialized purpose into a plurality recording segments;

(b) determining which of the plurality of recording segments correspond to a key child;

(c) determining which of the plurality of recording segments that correspond to the key child are classified as key child recordings;

(d) extracting acoustic parameters of the key child recordings;

(e) comparing the acoustic parameters of the key child recordings to known acoustic parameters for children; and

(f) determining a likelihood of autism.

2. The method of claim 1 wherein the acoustic parameters include canonical syllables.

3. The method of claim 2 wherein the acoustic parameters are divided into 4 categories, including a rhythm/syllabicity category, a low spectral tilt and high pitch control category, a wide formant bandwidth and low pitch control category, and a duration of islands within utterances category.

4. The method of claim 3 wherein the rhythm/syllabicity category includes a voiced parameter, a canonical syllable transitions category, and a spectral entropy category.

5. The method of claim 3 wherein the low spectral tilt and high pitch control category includes a mean pitch high parameter, a low tilt of spectrum parameter, and a high frequency energy concentration parameter.

6. The method of claim 3 wherein the wide formant bandwidth and low pitch control category includes a Mean pitch low parameter and Wide Bandwidth of first two formants parameter.

7. The method of claim 3 wherein the duration of islands within utterances category includes a short parameter, a medium parameter, a long parameter, and an extra long parameter.

8. The method of claim 1 wherein the comparing includes Logical Regression Analysis.

9. The method of claim 1 wherein the key child recordings are broken down into cry, vegetative-sound, and vocalization categories;

and the extracting of (d) is applied only to the vocalization category of the key child recordings.

10. The method of claim 1, further comprising:

(g) performing acoustic analysis on the key child recordings.

11. The method of claim 10 wherein the performing of (g) includes transforming the recordings to reveal formant contours, formant bandwidths, pitch contours, spectral analysis, and duration analysis.

12. The method of claim 11 wherein the acoustic analysis is used to perform the extracting of (e).

13. The method of claim 1, further comprising:

(h) transforming a display of a user to display the likelihood of autism.

14. The method of claim 1, further comprising:

(h) transforming an information storage device to store the likelihood of autism.

15. A method for detecting autism, comprising:

transforming an audio recording to output an indication of autism on an output mechanism selected from the group consisting of a display, a printing device, an electronic storage device, and an audio output device; and

the transforming of the audio recording performed by comparing it to a model developed by analyzing the transparent parameters of a plurality of sound recordings captured in a natural language environment.

16. The method of claim 15 wherein, in the case of each of the plurality of sound recordings, the analyzing includes:

(a) segmenting the sound recording into a plurality of recording segments, wherein the sound recording is captured by a microphone and sound recorder combination;

(b) determining which of the plurality of recording segments correspond to a key child;

(c) determining which of the plurality of recording segments that correspond to the key child are classified as key child recordings; and

(d) extracting acoustic parameters of the key child recordings.

17. The method of claim 15 wherein the comparing includes Logical Regression Analysis.

18. The method of claim 16 wherein the key child recordings are broken down into cry, vegetative-sound, and vocalization categories;

and the extracting of (d) is applied only to the vocalization category of the key child recordings.

19. The method of claim 16, further comprising:

(e) performing acoustic analysis on the key child recordings.

20. The method of claim 19 wherein the performing of (e) includes transforming the recordings to reveal formant contours, formant bandwidths, pitch contours, spectral analysis, and duration analysis.

21. The method of claim 20 wherein the acoustic analysis is used to perform the extracting of (e).

22. The method of claim 16 wherein the acoustic parameters include canonical syllables.

23. A method for detecting a disorder in a natural language environment using a microphone, sound recorder, and a computer programmed with software for the specialized purpose of processing recordings captured by the microphone and sound recorder combination, the computer programmed to execute a method comprising:

(a) segmenting an audio signal captured by the microphone and sound recorder combination using the computer programmed for the specialized purpose into a plurality of recording segments;

(b) determining which of the plurality of recording segments correspond to a key subject;

(c) determining which of the plurality of recording segments that correspond to the key subject are classified as key subject recordings;

(d) extracting acoustic parameters of the key subject recordings;

(e) comparing the acoustic parameters of the key subject recordings to known acoustic parameters for subjects; and

(f) determining a likelihood of the disorder.

24. The method of claim 23, further comprising:

(g) transforming an electronic storage device to record the likelihood of the disorder.

25. A method for detecting a disorder, comprising:

transforming an audio recording to output an indication of autism on an output mechanism selected from the group consisting of a display, a printing device, an electronic storage device, and an audio output device, the transforming of the audio recording performed by comparing it to a model developed by analyzing the transparent parameters of a plurality of sound recordings captured in a natural language environment, wherein in the case of each of the plurality of sound recordings, the analyzing includes;

(a) segmenting the sound recording into a plurality of recording segments, wherein the sound recording is captured by a microphone and sound recorder combination;

(b) determining which of the plurality of recording segments correspond to a key subject;

(c) determining which of the plurality of recording segments that correspond to the key subject are classified as key subject recordings; and

(d) extracting acoustic parameters of the key subject recordings.

26. A method of creating an automatic language characteristic recognition system, the method comprising:

(a) receiving a plurality of audio recordings;

(b) segmenting each of the plurality of audio recordings to create a plurality of audio segments for each audio recording; and

(c) clustering each audio segment of the plurality of audio recordings according to audio characteristics of each audio segment to form a plurality of audio segment clusters.

27. The method of claim 26, further comprising:

(d) transforming an information storage device to store the plurality of audio segment clusters.

28. The method of claim 26 wherein the clustering of (c) includes:

(i) initiating a plurality of center points;

(ii) for each audio segment, measuring the distance from the plurality of center points to the audio segment and assigning the audio segment to the center point of the plurality of center points closest to the audio segment;

(iii) for each center point, averaging the audio segments assigned to the center point and reassigning the center point to the average; and

(iv) repeating (ii) and (iii) until a convergence criteria is met.

29. A method of decoding speech using an automatic language characteristic recognition system, the method comprising:

(a) receiving a plurality of audio recordings;

(b) segmenting each of the plurality of audio recordings to create a first plurality of audio segments for each audio recording;

(c) clustering each audio segment of the plurality of audio recordings according to audio characteristics of each audio segment to form a plurality of audio segment clusters;

(d) receiving a new audio recording;

(e) segmenting the new audio recording to create a second plurality of audio segments for the new audio recording; and

(f) determining to which cluster of the plurality of audio segment clusters each segment of the second plurality of audio segments corresponds.

30. The method of claim 29, further comprising:

(g) determining a distribution for the second plurality of audio segments based on the cluster that each audio segment of the second plurality of audio segments corresponds; and

(h) comparing the distribution to an age-based model to access the language development of a key child corresponding to the second plurality of audio segments.

31. The method of claim 30, further comprising:

(i) outputting a language development assessment based on the comparing of (h).

32. A method of assessing a key child'"'"'s expressive language development, comprising:

(a) processing an audio recording taken in the key child'"'"'s language environment to identify segments of the recording that correspond to the key child'"'"'s vocalizations;

(b) applying an adult automatic speech recognition phone decoder to the segments to identify each occurrence of each of a plurality of bi-phone categories, wherein each of the bi-phone categories corresponds to a pre-defined speech sound sequence;

(c) determining a distribution for the bi-phone categories; and

(d) using the distribution in an age-based model to assess the key child'"'"'s expressive language development.

33. The method of claim 32 wherein determining a distribution for the bi-phone categories comprises determining a frequency distribution.

34. The method of claim 32 wherein determining a distribution for the bi-phone categories comprises determining a duration distribution.

35. The method of claim 32 wherein the age-based model is selected based on the key child'"'"'s chronological age, and the age-based model includes a weight associated with each of the bi-phone categories.

36. A system for of assessing a key child'"'"'s language development, comprising:

(a) a processor-based device comprising an application having an audio engine for processing an audio recording taken in the key child'"'"'s language environment to identify segments of the recording that correspond to the key child'"'"'s vocalizations;

(b) an adult automatic speech recognition phone decoder for processing the segments that correspond to the key child'"'"'s vocalizations to identify each occurrence of each of a plurality of bi-phone categories, wherein each of the bi-phone categories corresponds to a pre-defined speech sound sequence; and

(c) an expressive language assessment component for determining a distribution for the bi-phone categories and using the distribution in an age-based model to assess the key child'"'"'s expressive language development, wherein the age-based model is selected based on the key child'"'"'s chronological age, and the age-based model includes a weight associated with each of the bi-phone categories.

37. The system of claim 36 wherein the expressive language assessment component computes a standard score and a developmental age for the key child, further comprising:

an output component for receiving the standard score and the developmental age for the key child and generating an output including one or more of the following;

the standard score, the developmental age, a message, or a report.

38. The system of claim 36 wherein the expressive language assessment component determines a frequency distribution.

39. The system of claim 36 wherein the expressive language assessment component determines a duration distribution.

40. A method of determining the emotion of an utterance, the method comprising:

(a) receiving the utterance at a processor-based device comprising an application having an audio engine;

(b) extracting emotion-related acoustic features from the utterance;

(c) comparing the emotion-related acoustic features to a plurality of models representative of emotions;

(d) selecting a model from the plurality of models based on the comparing of (c); and

(e) outputting the emotion corresponding to the selected model.

41. The method of claim 40 wherein the utterance is segmented from a sound recording.

42. The method of claim 41 wherein a speaker of the utterance is identified.