System and method for expressive language assessment

• US 8,744,847 B2
• Filed: 04/25/2008
• Issued: 06/03/2014
• Est. Priority Date: 01/23/2007
• Status: Active Grant

• Current Assignee: LENA Foundation
• Original Assignee: LENA Foundation
• Inventors: Paul, Terrance, Xu, Dongxin, Richards, Jeffrey A.
• Primary Examiner(s): Shah, Paras D
• Assistant Examiner(s): KOVACEK, DAVID M
• Application Number:
  US12/109,785
• Publication Number:
  US 20090155751A1
• Time in Patent Office: 2,230 Days
• Field of Search: 704 1- 10, 704/200, 704205-218, 704/230, 704/231, 704/271, 704E17001-E17016, 704E15001-E15013, 704E11001-E11007, 434/112, 434/116, 434/156, 434/167, 434/169, 434178-179, 434319-321
• US Class Current: 704/233
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 17/16 Hidden Markov models [HMM]

Abstract

Certain aspects and embodiments of the present invention are directed to systems and methods for monitoring and analyzing the language environment and the development of a key child. A key child'"'"'s language environment and language development can be monitored without placing artificial limitations on the key child'"'"'s activities or requiring a third party observer. The language environment can be analyzed to identify phones or speech sounds spoken by the key child, independent of content. The number and type of phones is analyzed to automatically assess the key child'"'"'s expressive language development. The assessment can result in a standard score, an estimated developmental age, or an estimated mean length of utterance.

Claims

1. A method of assessing a key child'"'"'s expressive language development, comprising:

processing an audio recording taken in the key child'"'"'s language environment to identify segments of the recording that correspond to the key child'"'"'s vocalizations, wherein a computing device configured to perform the processing is used and the processing includes categorizing a plurality of segments of the audio recording into a plurality of categories, the plurality of categories including categories selected from the group consisting of vocalizations, cries, vegetative sounds, and fixed sounds, and determining which of the plurality of segments characterized as vocalizations are segments of the recording that correspond to the key child'"'"'s vocalizations by comparing the plurality of segments characterized as vocalizations to a plurality of models;

applying an adult automatic speech recognition phone decoder to segments of the key child'"'"'s vocalizations to identify each occurrence of each of a plurality of phone categories, wherein each of the phone categories corresponds to a pre-defined speech sound;

determining a distribution for the phone categories; and

using the distribution in an age-based model to assess the key child'"'"'s expressive language development.

2. The method of claim 1, wherein determining a distribution for the phone categories comprises determining a frequency distribution.

3. The method of claim 1, wherein determining a distribution for the phone categories comprises determining a duration distribution.

4. The method of claim 1, wherein the age-based model is selected based on the key child'"'"'s chronological age and the age-based model includes a weight associated with each of the phone categories.

5. The method of claim 1, wherein there is a correlation between the frequency of each of the phone categories to chronological age.

6. The method of claim 1, wherein the age-based model is based on an age in months.

7. The method of claim 1, wherein the age-based model is an adjustment of the key child'"'"'s chronological age.

8. The method of claim 1, wherein using the distribution in an age-based model to assess the child'"'"'s language development comprises computing a developmental age for the key child.

9. The method of claim 8, further comprising using the developmental age for the key child to determine an estimated mean length of utterance for the child.

10. The method of claim 1, wherein using the distribution in an age-based model to assess the key child'"'"'s expressive language development results in an estimated developmental age further comprises:

receiving results from a questionnaire that includes questions about the key child'"'"'s use of expressive language; and

averaging the results from the questionnaire with the estimated developmental age.

11. The method of claim 1, wherein using the distribution in an age-based model to assess the key child'"'"'s expressive language development results in an estimated developmental age, further comprising:

applying the adult automatic speech recognition phone decoder to additional segments to identify each occurrence of each of the phone categories;

determining an additional distribution for the phone categories;

using the additional distribution in an age-based model to assess the key child'"'"'s expressive language development and generate an additional estimated developmental age; and

averaging the estimated developmental age and the additional estimated developmental age.

12. The method of claim 11, further comprising processing at least one additional audio recording taken in the key child'"'"'s language environment to identify the additional segments that correspond to the key child'"'"'s vocalizations.

13. The method of claim 1, wherein the vocalizations include words, phrases, marginal syllables, consonant-vowel sequences, utterances, phonemes, sequence phonemes, phoneme-like sounds, protophones, lip-trilling sounds, canonical syllables, repetitive babbles, and pitch variations.

14. The method of claim 1, wherein the vegetative sounds include non-vocal sounds related to respiration and digestions.

15. The method of claim 1, wherein the fixed sounds include laughing, moaning, signing, and lip smacking.

16. A method of assessing a key child'"'"'s expressive language development, comprising:

processing an audio recording using a computing device configured to determine a plurality of key child audio segments that correspond to the key child'"'"'s vocalizations, wherein the processing includes segmenting and assigning a segment ID indicating a source to a first plurality of audio segments derived from the audio recording, the segmenting and assigning performed using a Minimum Duration Gaussian Mixture Model (MD-GMM), wherein the MD-GMM includes a plurality of models used for matching to the first plurality of audio segments, the plurality of models including a noise model that includes characteristics of sound attributable to noise, a key child model that includes characteristics of sounds from a hypothetical key child, and an adult model that includes characteristics of sound from an adult, wherein the segmenting and assigning assigns a portion of the first plurality of audio segments to the plurality of key child audio segments;

receiving the plurality of key child audio segments that correspond to the key child'"'"'s vocalizations;

determining a distribution for each of a plurality of phone categories for the key child audio segments, wherein each of the phone categories corresponds to a pre-defined speech sound;

selecting an age-based model, wherein the selected age-based model corresponds to the key child'"'"'s chronological age and the age-based model includes a weight associated with each of the phone categories; and

using the distribution in the selected age-based model to assess the key child'"'"'s language development.

17. The method of claim 16, wherein the distribution is a frequency distribution.

18. The method of claim 16, wherein the distribution is a duration distribution.

19. The method of claim 16, wherein the distribution for each of a plurality of phone categories for the audio segments is determined independent of a meaning of content of the audio segments.

20. The method of claim 16, wherein using the distribution in the selected age-based model to assess the key child'"'"'s language development comprises computing a developmental score for the key child.

21. The method of claim 16, wherein using the distribution in the selected age-based model to assess the key child'"'"'s language development comprises computing a developmental age for the key child by computing and an adjustment to the key child'"'"'s chronological age.

22. The method of claim 21, further comprising using the developmental age for the key child to determine an estimated mean length of utterance for the key child.

23. The method of claim 16, wherein determining a distribution for each of a plurality of phone categories for the key child audio segments comprises applying an adult automatic speech recognition phone decoder to the audio segments.

24. The method of claim 16, wherein there is a correlation between the frequency of each of the phone categories to chronological age.

25. A system for of assessing a key child'"'"'s language development, comprising:

a processor-based device executing software comprising;

an application having an audio engine configured to process an audio recording taken in the key child'"'"'s language environment to identify segments of the recording that correspond to the key child'"'"'s vocalizations;

an adult automatic speech recognition phone decoder configured to process segments that correspond to a key child'"'"'s vocalizations to identify each occurrence of each of a plurality of phone categories, wherein each of the phone categories corresponds to a pre-defined speech sound; and

an expressive language assessment component configured to determine a distribution for the phone categories and using the distributions in an age-based model to assess the key child'"'"'s expressive language development, wherein the age-based model is selected based on the key child'"'"'s chronological age and the age-based model includes a weight associated with each of the phone categories.

26. The system of claim 25, wherein the expressive language assessment component computes a standard score and a developmental age for the key child, further comprising:

an output component for receiving the standard score and the developmental age for the key child and generating an output including one or more of the following;

the standard score, the developmental age, a message, or a report.

27. The system of claim 25, wherein the expressive language assessment component determines a frequency distribution.

28. The system of claim 25, wherein the expressive language assessment component determines a duration distribution.

29. The system of claim 25, wherein the weight is numerical.

30. The system of claim 25, wherein the audio engine is configured to segment and assign a segment ID indicating a source to a first plurality of audio segments derived from the audio recording, the segmenting and assigning performed using a Minimum Duration Gaussian Mixture Model (MD-GMM), wherein the MD-GMM includes a plurality of models used for matching to the first plurality of audio segments, the plurality of models including a noise model that includes characteristics of sound attributable to noise, a key child model that includes characteristics of sounds from a hypothetical key child, and an adult model that includes characteristics of sound from an adult, wherein the segmenting and assigning assigns a portion of the first plurality of audio segments to the key child'"'"'s vocalizations.

31. A method of assessing a key child'"'"'s expressive language development, comprising:

processing an audio recording taken in the key child'"'"'s language environment to identify segments of the recording that correspond to the key child'"'"'s vocalizations;

applying an adult automatic speech recognition phone decoder to segments of the key child'"'"'s vocalizations to identify each occurrence of each of a plurality of phone categories, wherein each of the phone categories corresponds to a pre-defined speech sound, wherein applying an adult automatic speech recognition phone decoder comprises identifying occurrences of a plurality of non-phone categories, wherein each of the non-phone categories corresponds to a pre-defined non-speech sound;

determining a distribution for the phone categories; and

using the distribution in an age-based model to assess the key child'"'"'s expressive language development.

32. The method of claim 31, wherein the pre-defined non-speech sounds include vocalizations, cries, vegetative sounds, and fixed sounds.

33. A method of assessing a key child'"'"'s expressive language development, comprising:

processing an audio recording taken in the key child'"'"'s language environment to identify segments of the recording that correspond to the key child'"'"'s vocalizations;

applying an adult automatic speech recognition phone decoder to segments of the key child'"'"'s vocalizations to identify each occurrence of each of a plurality of phone categories, wherein each of the phone categories corresponds to a pre-defined speech sound;

determining a distribution for the phone categories; and

using the distribution in an age-based model to assess the key child'"'"'s expressive language development, wherein the age-based model is selected based on the key child'"'"'s chronological age and the age-based model includes a weight associated with each of the phone categories.

34. The method of claim 33, wherein the weight associated with each of the phone categories is numerical.

35. The method of claim 33, wherein the phone categories include individual phones.

36. The method of claim 35, wherein the individual phones include the AA phone and the AE phone.