Methods and vehicles for using determined mood of a human driver and moderating vehicle response

• US 10,535,341 B2
• Filed: 03/27/2017
• Issued: 01/14/2020
• Est. Priority Date: 04/22/2011
• Status: Active Grant

• Current Assignee: Emerging Automotive, LLC
• Original Assignee: Emerging Automotive, LLC
• Inventors: Penilla, Angel A., Penilla, Albert S.
• Primary Examiner(s): Brushaber, Frederick M
• Application Number:
  US15/470,881
• Publication Number:
  US 20170200449A1
• Time in Patent Office: 1,023 Days
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: G01C 21/3641 Personalized guidance, e.g. limited guidance on previously travelled routes
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/45 characterised by the type of analysis window
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G10L 25/63 : for estimating an emotional state
• CPC Class Codes: G10L 25/84 : for discriminating voice from noise
• CPC Class Codes: G10L 25/90 : Pitch determination of speech signals
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/306 : User profiles

Abstract

Methods and systems for determining a mood of a human driver of a vehicle and using the mood for generating a vehicle response, is provided. One example method includes capturing, by a camera of the vehicle, a face of the human driver. The capturing is configured to capture a plurality of images over a period of time, and the plurality of images are analyzed to identify a facial expression and changes in the facial expression of the human driver over the period of time. The method further includes capturing, by a microphone of the vehicle, voice input of the human driver. The voice input is captured over the period of time. The voice input is analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time. The method processes, by a processor of the vehicle, a combination of the facial expression and the voice profile captured during the period of time to predict the mood of the human driver. The method generates the vehicle response that is responsive to the mood of the human driver. The vehicle response is configured to make at least one adjustment to a setting of the vehicle. The adjustment is selected based on the mood of the human driver. The vehicle response can be used to make the driver more calm and/or assist in reducing distracted driving.

Claims

1. A method for determining a mood of a human driver of a vehicle and using the mood for generating a vehicle response, comprising,

capturing, by a camera of the vehicle, a face of the human driver, the capturing is configured to capture a plurality of images over a period of time, the plurality of images being analyzed to identify a facial expression and changes in the facial expression of the human driver over the period of time, the plurality of images further being analyzed for eye gaze patterns;

capturing, by a microphone of the vehicle, voice input of the human driver, the voice input is captured over the period of time, the voice input being analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time;

processing, by a processor of the vehicle, a combination of the facial expression, the eye gaze patterns and the voice profile captured during the period of time to determine the mood of the human driver; and

generating the vehicle response that is responsive to the voice input and the mood determined of the human driver, the vehicle response is configured to make at least one adjustment to a setting of the vehicle, the adjustment is selected based on the mood of the human driver.

2. The method of claim 1, wherein said combination includes combining select features obtained from the facial expressions, the eye gaze patterns and the voice profile to build behavior models used for determining of the mood of the human driver, the determining includes a level of prediction of said mood.

3. The method of claim 1, wherein facial characteristics of the human driver are related to a plurality of facial expressions that have been previously associated with specific moods, the facial characteristics include one or more of placement of the human driver'"'"'s mouth, a way eyes are opened or closed by the human driver, cheekbone characteristics of the human driver, or biometric determining of features of the human driver, or a combination of two or more thereof.

4. The method of claim 1, wherein processing the voice profile includes processing an audio sample of the voice input and identifying frequency and/or magnitude characteristics in the audio sample, the voice profile is associated with a tone of voice that is correlated to a possible mood among a plurality of moods.

5. The method of claim 1, wherein facial characteristics of the human driver are related to a plurality of facial expressions that have been previously associated with specific moods, the facial characteristics include one or more of placement of the human driver'"'"'s mouth, a way eyes are opened or closed by the human driver, cheekbone characteristics of the human driver, or biometric determining of features of the human driver, or a combination of two or more thereof and the processing the voice profile includes processing an audio sample of the voice input and identifying frequency and/or magnitude characteristics in the audio sample, the voice profile is associated with a tone of voice that is correlated to a possible mood among a plurality of moods.

6. The method of claim 5, wherein the plurality of facial expressions are associated to a plurality of templates of facial characteristics, the plurality of templates being generated during a training or calibration session or generated over time based on learned facial characteristics, wherein learned facial characteristics are identified using one or more other detected physical or physiological conditions.

7. The method of claim 6, wherein said physiological conditions include one or more of heart rate, or body heat, or blood pressure, or skin temperature, or skin galvanic resistance, or temperature of skin, or blood rush to a face of the human driver, or eye iris or retina changes, or odor/scent changes, or combinations of two or more thereof, and the physical conditions include one or more of a changes in the face of the human driver, or changes in a typing rhythm, or changes in gait, or changes in voice, or changes in voice tone, or voice inflections, or voice speed or slowness, or sharp voice harmonics, or voice identifiers or patterns, or elevated heat patterns, or palm veins, or eye iris or retina changes, or heat rate changes, or eye gaze motions, or combinations of two or more thereof.

8. The method of claim 1, wherein the vehicle response to the mood includes one of waking up the human driver with a sound or air if the mood is a sleepy mood, or generating warnings and/or signals to alert the human driver, or generating recommendations to calm the human driver from an angry mood or agitated mood, or sending a notification to a third party indicating that the human driver is tired, or generating a notification to the human driver suggesting a temperature change, or generating an automatic temperature change based on the mood, or recommending the human driver stop driving when tired, or reducing recommendations when the determined mood is a rushed mood, or changing lighting of the vehicle automatically, or changing temperature of the vehicle automatically, or turn up or down a volume of music, or adjust a seat position, or a combination of two or more thereof, and

wherein at least one of said vehicle responses are predefined to reduce distracted driving or increase alertness of said human driver.

9. The method of claim 1, wherein

the vehicle response is further based on current use of the vehicle, and use of the vehicle includes one or more of geolocation of the vehicle, speed of the vehicle, direction of the vehicle, a route of the vehicle, a time of day, occupants in the vehicle, historical use of the vehicle, online data of a user accessible to the vehicle or server, or combinations of two or more thereof, and

the vehicle response to the mood includes one of waking up the human driver with a sound or air if the mood is a sleepy mood, generating warnings and/or signals to alert the human driver, generating recommendations to calm the human driver from an angry mood or agitated mood, sending a notification to a third party indicating that the human driver is tired, generating a notification to the human driver suggesting a temperature change, or generating an automatic temperature change based on the mood, recommending the human driver stop driving when tired, reducing recommendations when the human driver when the determined mood is a rushed mood, or a combination of two or more thereof.

10. The method of claim 1, wherein a user account with a profile for the human driver of the vehicle is provided, wherein the user account of the human driver is identified based on processing of credential data obtained from the human driver, the credential data used by electronics of the vehicle to associated the human driver to the user account and to the profile, the electronics of the vehicle configured to interface over a network with a server for use of the profile and associated updates to the profile as made via use of the vehicle or updates associated with learned behavior.

11. The method of claim 1, wherein the mood includes one or more of a normal mood, or a frustrated mood, or an agitated mood, or an upset mood, or a hurried mood, or an urgency mood, or a rushed mood, or a stressed mood, or a calm mood, or a passive mood, or a sleepy mood, or a happy mood, or an excited mood, or combinations of two or more thereof, and

said mood is configured to change over time based on said captured face and voice input of the human driver.

12. The method of claim 1, wherein the processing of the voice input includes identifying a dialect of the human driver, the dialect includes one or more of a language dialect, a geographic region dialect, an age dialect, or combinations of two or more thereof.

13. The method of claim 1, wherein a prediction of the mood is increased by capturing touch or gesture characteristic of the human driver when interfacing with a graphical user interface of the vehicle or systems of the vehicle.

14. The method of claim 1, wherein learned behavior of the human driver is generated based on express inputs by the human driver and/or learned behavior that is inferred from inputs and/or settings made over time to a vehicle when a user account of the human driver was active on the vehicle,

wherein a server that communicates wirelessly with the vehicle from time to time stores at least some of the inputs and/or settings made to the vehicle in a database for processing said learned behavior, said processing is configured to use machine learning.

15. The method of claim 1, wherein physical characteristic of the human driver are detected using one or more sensors or devices of the vehicle or devices worn by the user of the vehicle,

the physical characteristic includes one or more of heart rate of the human driver, or motions of a body of the human driver, or body heat of the human driver, or stress indicators of the human driver, or voice micro tremors of the human driver, or voice inflections or pitches of the human driver, or emotional sounds of the human driver, or skin temperature of the human driver, or changes in face heat of the human driver, or slurred speech of the human driver, or touch patterns of the human driver, or retina scan sensing of the human driver, or gesture detection of the human driver, or a combination of two or more thereof.

16. The method of claim 1, further comprising,

executing a machine learning algorithm to process a history of inputs received from the vehicle for the human driver when using vehicle via a profile of a user account, the machine learning algorithm is used to identify patterns in inputs made by the human driver based on a context of use of the vehicle, the patterns being used to predict use patterns of the vehicle by the human driver and change at least one aspect of the vehicle response that is responsive to the predicted mood.

17. The method of claim 16, wherein the context of use of the vehicle includes identifying geolocation data of the vehicle and inputs or actions made that correspond to said geolocation data.

18. A method for determining a mood of a human driver of a vehicle and using the mood for generating a vehicle response, comprising,

capturing, by a camera of the vehicle, a face of the human driver, the capturing is configured to capture a plurality of images over a period of time, the plurality of images being analyzed to identify a facial expression and changes in the facial expression that include eyes of the human driver over the period of time;

capturing, by a microphone of the vehicle, voice input that includes a request of the vehicle by the human driver, the voice input is captured over the period of time, the voice input being analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time;

processing, by a processor of the vehicle, a combination of the facial expression and the voice profile captured during the period of time to determine the mood of the human driver; and

generating the vehicle response that is responsive to the request in the voice input, the vehicle response is configured to make at least one adjustment to a setting of the vehicle or output information via a speaker of the vehicle, the adjustment modifies the vehicle response based on the determined mood of the human driver;

wherein a user account with a profile for the human driver of the vehicle is provided, the profile includes a plurality of learned facial expressions and learned voice profiles that are indicative of the mood of the human driver and used to determine the mood.

19. Non-transitory computer readable media having program instructions for determining a mood of a human driver of a vehicle and using the mood for generating a vehicle response, comprising,

program instructions for capturing, by a camera of the vehicle, a face of the human driver, the capturing is configured to capture a plurality of images over a period of time, the plurality of images being analyzed to identify a facial expression and changes in the facial expression of the human driver over the period of time;

program instructions for capturing, by a microphone of the vehicle, voice input of the human driver, the voice input is captured over the period of time, the voice input being analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time;

program instructions for processing, by a processor of the vehicle, a combination of the facial expression and the voice profile captured during the period of time to determine the mood of the human driver; and

program instructions for generating the vehicle response that is responsive to the voice input and determined mood of the human driver, the vehicle response is configured to make at least one adjustment to a setting of the vehicle or adjustment to information obtained from an Internet source, the adjustment modifies the vehicle response based on the mood of the human driver,

wherein the vehicle response is further adjusted based on current use of the vehicle that includes having one or more additional occupants in addition to the human driver and processing one or more voice comments of said one or more additional occupants.

20. The non-transitory computer readable media of claim 19, wherein use of the vehicle further includes one or more of geolocation of the vehicle, speed of the vehicle, direction of the vehicle, a route of the vehicle, a time of day, occupants in the vehicle, historical use of the vehicle, online data of the user accessible to the vehicle or server, or combinations of two or more thereof, and

the vehicle response to the mood includes one of waking up the human driver with a sound or air if the mood is a sleepy mood, generating warnings and/or signals to alert the human driver, generating recommendations to calm the human driver from an angry mood or agitated mood, sending a notification to a third part indicating that the human driver is tired, generating a notification to the human driver suggesting a temperature change, generating an automatic temperature change based on the mood, recommending the human driver stop driving when tired, reducing recommendations when the determined mood is a rushed mood, or a combination of two or more thereof.