Methods and vehicles for capturing emotion of a human driver and moderating vehicle response

• US 10,453,453 B2
• Filed: 10/18/2017
• Issued: 10/22/2019
• Est. Priority Date: 04/22/2011
• Status: Active Grant

• Current Assignee: Emerging Automotive, LLC
• Original Assignee: Emerging Automotive, LLC
• Inventors: Penilla, Angel A., Penilla, Albert S.
• Primary Examiner(s): McFadden, Susan I
• Application Number:
  US15/787,677
• Publication Number:
  US 20180061415A1
• Time in Patent Office: 734 Days
• Field of Search: 704275
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: G01C 21/3641 Personalized guidance, e.g. limited guidance on previously travelled routes
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/04 : Training, enrolment or model building
• CPC Class Codes: G10L 17/06 : Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 : of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 : of application context
• CPC Class Codes: G10L 21/0208 : Noise filtering
• CPC Class Codes: G10L 25/45 : characterised by the type of analysis window
• CPC Class Codes: G10L 25/57 : for processing of video signals
• CPC Class Codes: G10L 25/63 : for estimating an emotional state
• CPC Class Codes: G10L 25/84 : for discriminating voice from noise
• CPC Class Codes: G10L 25/90 : Pitch determination of speech signals
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/306 : User profiles

Abstract

Methods and systems for determining an emotion of a human driver of a vehicle and using the emotion for generating a vehicle response, is provided. One example method includes capturing, by a camera of the vehicle, a face of the human driver. The capturing is configured to capture a plurality of images over a period of time, and the plurality of images are analyzed to identify a facial expression and changes in the facial expression of the human driver over the period of time. The method further includes capturing, by a microphone of the vehicle, voice input of the human driver. The voice input is captured over the period of time. The voice input is analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time. The method processes, by a processor of the vehicle, a combination of the facial expression and the voice profile captured during the period of time to predict the emotion of the human driver. The method generates the vehicle response that is responsive to the emotion of the human driver. The vehicle response is configured to make at least one adjustment to a setting of the vehicle. The adjustment is selected based on the emotion of the human driver. The vehicle response can be used to make the driver more calm and/or assist in reducing distracted driving. The prediction of the emotion may be additionally increased by capturing and analyzing touch and/or gesture characteristic of the human driver when interfacing with a graphical user interface or surfaces of the vehicle or systems of the vehicle.

Claims

1. A method for determining an emotion of a human driver of a vehicle and using the emotion for generating a vehicle response, comprising,

capturing, by a camera of the vehicle, a face of the human driver, the capturing is configured to capture a plurality of images over a period of time, the plurality of images being analyzed to identify a facial expression and changes in the facial expression of the human driver over the period of time;

capturing, by a microphone of the vehicle, voice input of the human driver, the voice input is captured over the period of time, the voice input being analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time;

processing, by a processor of the vehicle, a combination of the facial expression and the voice profile captured during the period of time to predict the emotion of the human driver; and

generating the vehicle response that is responsive to the emotion of the human driver, the vehicle response is configured to make at least one adjustment to a setting of the vehicle, the adjustment is selected based on the emotion of the human driver.

2. The method of claim 1, wherein combining data obtained from facial expressions and the voice profile improve the prediction of the emotion of the human driver and wherein prediction of the emotion is increased by capturing touch or gesture characteristic of the human driver when interfacing with a graphical user interface of the vehicle or systems of the vehicle.

3. The method of claim 1, wherein facial characteristics of the human driver is are related to a plurality of facial expressions that have been previously associated with specific emotions, the facial characteristics include one or more of placement of the human driver'"'"'s mouth, a way eyes are opened or closed by the human driver, cheekbone characteristics of the human driver, or biometric determining of features of the human driver, or a combination of two or more thereof.

4. The method of claim 1, wherein processing the voice profile includes processing an audio sample of the voice input and identifying frequency and/or magnitude characteristics in the audio sample, the voice profile is associated with a tone of voice that is correlated to a possible emotion among a plurality of emotions.

5. The method of claim 1, wherein facial characteristics of the human driver is are related to a plurality of facial expressions that have been previously associated with specific emotions, the facial characteristics include one or more of placement of the human driver'"'"'s mouth, a way eyes are opened or closed by the human driver, cheekbone characteristics of the human driver, or biometric determining of features of the human driver, or a combination of two or more thereof and the processing the voice profile includes processing an audio sample of the voice input and identifying frequency and/or magnitude characteristics in the audio sample, the voice profile is associated with a tone of voice that is correlated to a possible emotion among a plurality of emotions.

6. The method of claim 5, wherein the plurality of facial expressions are associated to a plurality of templates of facial characteristics, the plurality of templates being generated during a training or calibration session or generated over time based on learned facial characteristics, wherein learned facial characteristics are identified using one or more other detected physical or physiological conditions.

7. The method of claim 6, wherein said physiological conditions include one or more of heart rate, or body heat, or blood pressure, or skin temperature, or skin galvanic resistance, or temperature of skin, or blood rush to a face of the human driver, or eye iris or retina changes, or odor/scent changes, or combinations of two or more thereof, and the physical conditions include one or more of a changes in the face of the human driver, or changes in a typing rhythm, or changes in gait, or changes in voice, changes in voice tone, or voice inflections, or voice speed or slowness, or sharp voice harmonics, or voice identifiers or patterns, or elevated heat patterns, or palm veins, or eye iris or retina changes, heat rate changes, or eye gaze patterns or motions, or combinations of two or more thereof.

8. The method of claim 1, wherein a user account with a profile for the human driver of the vehicle is provided, the profile includes a plurality of learned facial expressions and learned voice profiles that are indicative of an emotion, the learned plurality of facial expressions and learned voice profiles are updated from time to time to improve the prediction of the emotion of the human driver, wherein the user profile is synchronized from time to time with a server that manages the user account for the human driver.

9. The method of claim 1, wherein the vehicle response to the emotion includes one of waking up the human driver with a sound or air if the emotion is a sleepy emotion, generating warnings and/or signals to alert the human driver, generating recommendations to calm the human driver from an angry emotion or agitated emotion, sending a notification to a third party indicating that the human driver is tired, generating a notification to the human driver suggesting a temperature change, or generating an automatic temperature change based on the emotion, recommending the human driver stop driving when tired, reducing recommendations when the human driver when the predicted emotion is a rushed emotion, changing lighting of the vehicle automatically, changing temperature of the vehicle automatically, turn up or down a volume of music, adjust a seat position, or a combination of two or more thereof, and wherein at least one of said vehicle responses are predefined to reduce distracted driving or increase alertness of said human driver.

10. The method of claim 1, wherein the vehicle response is further based on current use of the vehicle, and use of the vehicle includes one or more of geolocation of the vehicle, speed of the vehicle, direction of the vehicle, a route of the vehicle, a time of day, occupants in the vehicle, historical use of the vehicle, online data of a user accessible to the vehicle or server, or combinations of two or more thereof, and the vehicle response to the emotion includes one of waking up the human driver with a sound or air if the emotion is a sleepy emotion, generating warnings and/or signals to alert the human driver, generating recommendations to calm the human driver from an angry emotion or agitated emotion, sending a notification to a third party indicating that the human driver is tired, generating a notification to the human driver suggesting a temperature change, or generating an automatic temperature change based on the emotion, recommending the human driver stop driving when tired, reducing recommendations when the human driver when the predicted emotion is a rushed emotion, or a combination of two or more thereof.

11. The method of claim 1, wherein a user account with a profile for the human driver of the vehicle is provided, wherein the user account of the human driver is identified based on processing of credential data obtained from the human driver, the credential data used by electronics of the vehicle to associated the human driver to the user account and to the profile, the electronics of the vehicle configured to interface over a network with a server for use of the profile and associated updates to the profile as made via use of the vehicle or updates associated with learned behavior.

12. The method of claim 1, wherein the emotion includes one or more of a normal emotion, a frustrated emotion, an agitated emotion, an upset emotion, a hurried emotion, an urgency emotion, a rushed emotion, a stressed emotion, a calm emotion, a passive emotion, a sleepy emotion, a happy emotion, or an excited emotion, or combinations of two or more thereof, and said emotion is configured to change over time based on said captured face and voice input of the human driver.

13. The method of claim 1, wherein the processing of the voice input includes identifying a dialect of the human driver, the dialect includes one or more of a language dialect, a geographic region dialect, an age dialect, or combinations of two or more thereof.

14. The method of claim 1, wherein learned behavior of the human driver is generated based on express inputs by the human driver and/or learned behavior that is inferred from inputs and/or settings made over time to a vehicle when a user account of the human driver was active on the vehicle,

wherein a server that communicates wirelessly with the vehicle from time to time stores at least some of the inputs and/or settings made to the vehicle in a database for processing said learned behavior, said processing is configured to use machine learning.

15. The method of claim 1, wherein physical characteristic of the human driver are detected using one or more sensors or devices of the vehicle or devices worn by the user of the vehicle, the physical characteristic includes one or more of heart rate of the human driver, motions of a body of the human driver, eye gaze patterns of the human driver, body heat of the human driver, stress indicators of the human driver, voice micro tremors of the human driver, voice inflections or pitches of the human driver, emotional sounds of the human driver, skin temperature of the human driver, changes in face heat of the human driver, slurred speech of the human driver, camera captured facial expressions of the human driver, touch patterns of the human driver, retina scan sensing of the human driver, gesture detection of the human driver, or a combination of two or more thereof.

16. The method of claim 1, further comprising,

executing a machine learning algorithm to process a history of inputs received from the vehicle for the human driver when using vehicle via a profile of a user account, the machine learning algorithm is used to identify patterns in inputs made by the human driver based on a context of use of the vehicle, the patterns being used to predict use patterns of the vehicle by the human driver and moderate at least one aspect of the vehicle response that is responsive to the predicted emotion;

wherein the context of use of the vehicle includes identifying geolocation data of the vehicle and inputs or actions made that correspond to said geolocation data; and

capturing, via the microphone, a conversation between the human driver of the vehicle and a passenger of the vehicle, the conversation being analyzed to provide recommendations.