Systems and methods for an automatic language characteristic recognition system

• US 9,799,348 B2
• Filed: 01/15/2016
• Issued: 10/24/2017
• Est. Priority Date: 09/16/2004
• Status: Active Grant

• Current Assignee: LENA Foundation
• Original Assignee: LENA Foundation
• Inventors: Paul, Terrance D., Xu, Dongxin D., Gray, Sharmistha Sarkar, Yapanel, Umit, Gilkerson, Jill S., Richards, Jeffrey A.
• Primary Examiner(s): Guerra-Erazo, Edgar
• Application Number:
  US14/997,172
• Publication Number:
  US 20160203832A1
• Time in Patent Office: 648 Days
• CPC Class Codes: A61B 2503/06 Children, e.g. for attention deficit diagnosis
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/075 supervised, i.e. under machine guidance
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 17/16 Hidden Markov models [HMM]
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2015/022 Demisyllables, biphones or triphones being the recognition units
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/66 for extracting parameters related to health condition detecting or measuring for diagnostic purposes A61B5/00

Abstract

In some embodiments, a method of creating an automatic language characteristic recognition system. The method can include receiving a plurality of audio recordings. The method also can include segmenting each of the plurality of audio recordings to create a plurality of audio segments for each audio recording. The method additionally can include clustering each audio segment of the plurality of audio segments according to audio characteristics of each audio segment to form a plurality of audio segment clusters. Other embodiments are provided.

Claims

1. A method of creating an automatic language characteristic recognition system, the method being implemented via execution of computer instructions configured to run at one or more processors and configured to be stored at one or more non-transitory computer-readable media, the method comprising:

receiving a plurality of audio recordings;

segmenting each of the plurality of audio recordings to create a plurality of audio segments for each audio recording;

clustering each audio segment of the plurality of audio segments according to audio characteristics of the each audio segment to form a plurality of audio segment clusters, wherein the audio characteristics of the each audio segment used in the clustering comprise at least one of a pitch of sound in the each audio segment, a duration of the sound in the each audio segment, a rhythm of the sound in the each audio segment, or an organization of the sound in the each audio segment; and

generating an age-based model in a data store that associates the plurality of audio segment clusters to weightings for specific ages of those represented in the plurality of audio recordings.

2. The method of claim 1 further comprising:

transforming the data store to store the plurality of audio segment clusters.

3. The method of claim 1, wherein:

clustering the each audio segment of the plurality of audio segments comprises using an unsupervised clustering method.

4. The method of claim 1, wherein:

clustering the each audio segment of the plurality of audio segments comprises using K-means clustering.

5. The method of claim 1, wherein:

a quantity of the plurality of audio segment clusters is at least 64.

6. The method of claim 1, wherein clustering the each audio segment of the plurality of audio segments comprises:

initiating a plurality of center points; and

until a convergence criterion is met, repeating;

for each audio segment of the plurality of audio segments;

measuring a distance from the plurality of center points to the audio segment; and

assigning the audio segment to a center point of the plurality of center points that is closest to the audio segment; and

for each center point of the plurality of center points;

averaging the audio segments assigned to the center point to obtain an average; and

reassigning the center point to the average.

7. The method of claim 1 further comprising:

transforming the data store to store the plurality of audio segment clusters,

wherein;

clustering the each audio segment of the plurality of audio segments comprises using an unsupervised clustering method.

8. The method of claim 1 further comprising:

transforming the data store to store the plurality of audio segment clusters,

wherein;

clustering the each audio segment of the plurality of audio segments comprises using K-means clustering.

9. The method of claim 1 further comprising:

transforming the data store to store the plurality of audio segment clusters,

wherein;

clustering the each audio segment of the plurality of audio segments comprises;

initiating a plurality of center points; and

until a convergence criterion is met, repeating;

for each audio segment of the plurality of audio segments;

 

measuring a distance from the plurality of center points to the audio segment; and

 

assigning the audio segment to a center point of the plurality of center points that is closest to the audio segment; and

for each center point of the plurality of center points;

 

averaging the audio segments assigned to the center point to obtain an average; and

 

reassigning the center point to the average.

10. The method of claim 9, wherein:

a quantity of the plurality of audio segment clusters is at least 64.

11. A method of decoding speech using an automatic language characteristic recognition system, the method being implemented via execution of computer instructions configured to run at one or more processors and configured to be stored at one or more non-transitory computer-readable media, the method comprising:

receiving a plurality of audio recordings;

segmenting each of the plurality of audio recordings to create a first plurality of audio segments for each audio recording;

clustering each audio segment of the first plurality of audio segments across all of the plurality of audio recordings according to audio characteristics of the each audio segment to form a plurality of audio segment clusters, wherein the audio characteristics of the each audio segment used in the clustering comprise at least one of a pitch of sound in the each audio segment, a duration of the sound in the each audio segment, a rhythm of the sound in the each audio segment, or an organization of the sound in the each audio segment

generating an age-based model in a data store that associates the plurality of audio segment clusters to weightings for specific ages of those represented in the plurality of audio recordings;

receiving a new audio recording from a key child;

segmenting the new audio recording to create a second plurality of audio segments for the new audio recording;

determining a corresponding cluster of the plurality of audio segment clusters for each audio segment of the second plurality of audio segments; and

applying the age-based model from the data store to the corresponding clusters for the second plurality of audio segments to determine a language development assessment of the key child.

12. The method of claim 11, wherein applying the age-based model further comprises:

determining a distribution for the second plurality of audio segments based on the corresponding cluster to which the each audio segment of the second plurality of audio segments corresponds.

13. The method of claim 12, wherein applying the age-based model further comprises:

comparing the distribution to the age-based model to determine the language development assessment of the key child corresponding to the second plurality of audio segments.

14. The method of claim 13 further comprising:

outputting the language development assessment based on the comparing the distribution to the age-based model.

15. The method of claim 11, wherein:

clustering the each audio segment of the first plurality of audio segments comprises using an unsupervised clustering method.

16. The method of claim 11, wherein:

clustering the each audio segment of the first plurality of audio segments comprises using K-means clustering.

17. The method of claim 11, wherein:

a quantity of the plurality of audio segment clusters is at least 64.

18. The method of claim 11 further comprising:

transforming the data store to store the plurality of audio segment clusters.

19. The method of claim 11, wherein clustering the each audio segment of the first plurality of audio segments comprises:

initiating a plurality of center points; and

until a convergence criterion is met, repeating;

for each audio segment of the first plurality of audio segments;

measuring a distance from the plurality of center points to the audio segment; and

assigning the audio segment to a center point of the plurality of center points that is closest to the audio segment; and

for each center point of the plurality of center points;

averaging the audio segments assigned to the center point to obtain an average; and

reassigning the center point to the average.

20. The method of claim 11 further comprising:

adding each audio segment of the second plurality of audio segments to the corresponding cluster of the plurality of audio segment clusters; and

updating the age-based model in the data store.