Audio analysis learning with video data

• US 10,573,313 B2
• Filed: 02/11/2019
• Issued: 02/25/2020
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Mishra, Taniya, el Kaliouby, Rana
• Primary Examiner(s): Alizada, Omeed
• Application Number:
  US16/272,054
• Publication Number:
  US 20190172462A1
• Time in Patent Office: 379 Days
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/215 Selection or confirmation of options
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/0356 for synchronising with other signals, e.g. video signals
• CPC Class Codes: G10L 21/055 for synchronising with other signals, e.g. video signals
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

Audio analysis learning is performed using video data. Video data is obtained, on a first computing device, wherein the video data includes images of one or more people. Audio data is obtained, on a second computing device, which corresponds to the video data. A face within the video data is identified. A first voice, from the audio data, is associated with the face within the video data. The face within the video data is analyzed for cognitive content. Audio features corresponding to the cognitive content of the video data are extracted. The audio data is segmented to correspond to an analyzed cognitive state. An audio classifier is learned, on a third computing device, based on the analyzing of the face within the video data. Further audio data is analyzed using the audio classifier.

Claims

1. A computer-implemented method for audio analysis comprising:

obtaining video data, on a first computing device, wherein the video data includes images of one or more people;

obtaining audio data, on a second computing device, corresponding to the video data;

identifying a face within the video data;

associating a first voice, from the audio data, with the face within the video data;

analyzing the face within the video data for cognitive content;

learning an audio classifier, on a third computing device, based on the analyzing of the face within the video data, wherein the learning the audio classifier is based on analyzing a plurality of faces within the video data; and

analyzing further audio data using the audio classifier.

2. The method of claim 1 further comprising extracting audio features corresponding to the cognitive content of the video data.

3. The method of claim 1 further comprising segmenting the audio data to correspond to an analyzed cognitive state.

4. The method of claim 3 wherein the segmenting the audio data is for a human sensorially detectable unit of time.

5. The method of claim 4 wherein the segmenting the audio data includes noticeable differences in intensity, duration, or pitch.

6. The method of claim 3 wherein the segmenting the audio data is for less than thirty seconds.

7. The method of claim 1 further comprising synchronizing the audio data with the video data.

8. The method of claim 1 further comprising analyzing a first voice for features.

9. The method of claim 8 wherein the analyzing the first voice for features includes evaluation of timbre.

10. The method of claim 8 wherein the analyzing the first voice for features includes evaluation of prosody.

11. The method of claim 8 wherein the analyzing the first voice for features includes analysis of vocal register and vocal resonance, pitch, speech volumes, or speech rate.

12. The method of claim 8 wherein the analyzing the first voice for features includes language analysis.

13. The method of claim 12 wherein the language analysis is dependent on language content.

14. The method of claim 1 wherein the learning is independent of language content.

15. The method of claim 1 wherein the learning further encompasses learning a second audio classifier.

16. The method of claim 1 wherein the cognitive content includes detection of one or more of sadness, stress, happiness, anger, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, curiosity, humor, depression, envy, sympathy, embarrassment, poignancy, fatigue, drowsiness, or mirth.

17. The method of claim 1 wherein the identifying of the face includes detection of facial expressions.

18. The method of claim 1 further comprising determining a temporal audio signature for use with the further audio data.

19. The method of claim 1 further comprising identifying and separating a second voice from the obtained audio data corresponding to the video data but not associated with the face associated with a first voice, wherein the second voice is included in the learning and wherein the second voice corresponds to a second person.

20. The method of claim 19 further comprising associating the second voice, from the second person, with a second face within the video data.

21. The method of claim 1 further comprising manipulating a vehicle based on the analyzing of the further audio data.

22. The method of claim 21 wherein the manipulating the vehicle includes transferring into autonomous mode;

transferring out of autonomous mode;

locking out operation;

recommending a break for an occupant;

recommending a different route;

recommending how far to drive;

responding to traffic;

adjusting seats, mirrors, climate control, lighting, music, audio stimuli, or interior temperature;

activating brakes; and

assuming steering control.

23. The method of claim 1 wherein the cognitive content includes emotional content or mental state content.

24. The method of claim 1 wherein the first computing device and the second computing device, or the second computing device and the third computing device, are a common device.

25. The method of claim 1 wherein the learning further comprises:

synchronizing the audio data and the video data;

extracting an audio feature associated with the cognitive content that was analyzed from the face; and

abstracting an audio classifier based on the extracted audio feature.

26. A computer program product embodied in a non-transitory computer readable medium for audio analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining video data wherein the video data includes images of one or more people;

obtaining audio data corresponding to the video data;

identifying a face within the video data;

associating a first voice, from the audio data, with the face within the video data;

analyzing the face within the video data for cognitive content;

learning an audio classifier, on a third computing device, based on the analyzing of the face within the video data, wherein the learning the audio classifier is based on analyzing a plurality of faces within the video data; and

analyzing further audio data using the audio classifier.

27. A computer system for audio analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain video data, on a first computing device, wherein the video data includes images of one or more people;

obtain audio data, on a second computing device, corresponding to the video data;

identify a face within the video data;

associate a first voice, from the audio data, with the face within the video data;

analyze the face within the video data for cognitive content;

learn an audio classifier, on a third computing device, based on the analyzing of the face within the video data, wherein the learning the audio classifier is based on analyzing a plurality of faces within the video data; and

analyze further audio data using the audio classifier.