Technology for assessing emotional state of vehicle operator

• US 10,322,727 B1
• Filed: 09/22/2017
• Issued: 06/18/2019
• Est. Priority Date: 01/18/2017
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Chan, Aaron Scott, Sanchez, Kenneth J.
• Primary Examiner(s): Garcia, Santiago
• Application Number:
  US15/713,576
• Time in Patent Office: 634 Days
• Field of Search: 340576
• CPC Class Codes: B60R 2001/1253 with cameras, video cameras or video screens
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness

Abstract

Systems and methods for analyzing sensor data to determine an inferred emotional state of a vehicle operator and generate a recommendation based on the inferred emotional state are disclosed. According to embodiments, an electronic device may analyze image data depicting a vehicle operator to identify a behavior. The electronic device may also compare vehicle dynamics data to baseline vehicle dynamics data to determine if the vehicle dynamics data differs from the baseline vehicle dynamics data by a threshold amount. Based on the identified behavior and the vehicle dynamics data, the electronic device may determine an inferred emotional state of the vehicle operator. The electronic device may generate a recommendation for the vehicle operator based on the inferred emotional state.

Claims

1. A computer implemented method in an electronic device of assessing an emotional state of an operator of a vehicle, the method comprising:

accessing image data captured by one or more image sensors located within the vehicle depicting the operator of the vehicle;

dynamically analyzing, by a processor, the image data to identify a current behavior of the operator that is depicted in the image data;

accessing vehicle dynamics data associated with operation of the vehicle;

comparing, by the processor, the vehicle dynamics data to baseline vehicle dynamics data to determine that a vehicle dynamics data value for at least one vehicle dynamics data parameter differs from a baseline vehicle dynamics data value for the vehicle dynamics data parameter by a threshold amount; and

in response to the comparing, determining, based on both the vehicle dynamics data and the current behavior of the operator identified by dynamically analyzing the image data captured by the one or more image sensors located within the vehicle, an inferred current emotional state of the operator.

2. The computer-implemented method of claim 1, further comprising:

generating a log indicating the inferred current emotional state of the operator; and

transmitting, via a network connection, the log to a server.

3. The computer-implemented method of claim 1, further comprising:

determining, based on the inferred current emotional state, that the operator is not currently in an appropriate emotional state for operating the vehicle.

4. The computer-implemented method of claim 1, further comprising:

generating a recommendation for the operator, the recommendation based on the inferred current emotional state; and

presenting the recommendation in a user interface of the electronic device.

5. The computer-implemented method of claim 1, wherein analyzing, by the processor, the image data to identify the current behavior of the operator that is depicted in the image data comprises:

accessing a set of baseline images depicting a plurality of exemplary movements, gestures, or positions;

comparing, by the processor, the image data depicting the operator of the vehicle to the baseline images to identify a set of movements, gestures, or positions of the operator as depicted in the image data; and

determining the current behavior of the operator based on the set of movements, gestures, or positions of the operator as depicted in the image data.

6. The computer-implemented method of claim 1, wherein accessing the vehicle dynamics data associated with operation of the vehicle comprises:

accessing vehicle dynamics data captured by one or more vehicle dynamics sensors associated with the vehicle.

7. The computer-implemented method of claim 1, wherein comparing, by the processor, the vehicle dynamics data to the baseline vehicle dynamics data comprises:

identifying, from the vehicle dynamics data, the at least one vehicle dynamics data parameter and the vehicle dynamics data value thereof;

identifying, from the baseline vehicle dynamics data, the baseline vehicle dynamics data value for the at least one vehicle dynamics data parameter;

calculating a difference between the vehicle dynamics data value and the baseline vehicle dynamics data value; and

determining that the difference exceeds the threshold amount.

8. The computer-implemented method of claim 7, wherein calculating the difference between the vehicle dynamics data value and the baseline vehicle dynamics data value comprises:

calculating a percentage difference between the vehicle dynamics data value and the baseline vehicle dynamics data value;

and wherein determining that the difference exceeds the threshold amount comprises determining that the percentage difference exceeds a percentage threshold amount.

9. The computer-implemented method of claim 1, wherein determining the inferred current emotional state of the operator comprises:

accessing a database that associates a plurality of inferred emotional states with a plurality of operator behaviors; and

determining, from the plurality of inferred emotional states, the inferred emotional state that matches the current behavior of the operator.

10. An electronic device configured to assess an emotional state of an operator of a vehicle, comprising:

a transceiver configured to communicate data via at least one network connection;

a memory configured to store non-transitory computer executable instructions and baseline vehicle dynamics data;

a processor configured to interface with the transceiver and the memory, and configured to execute the non-transitory computer executable instructions to cause the processor to;

access image data depicting the operator of the vehicle captured by the one or more image sensors located within the vehicle,

dynamically analyze the image data to identify a current behavior of the operator that is depicted in the image data,

access vehicle dynamics data associated with operation of the vehicle, compare the vehicle dynamics data to the baseline vehicle dynamics data to determine that the vehicle dynamics data differs from the baseline vehicle dynamics data by a threshold amount, and

in response to the comparing, determine, based on both the vehicle dynamics data and the current behavior of the operator identified by dynamically analyzing the image data captured by the one or more image sensors located within the vehicle, an inferred current emotional state of the operator.

11. The electronic device of claim 10, wherein the computer executable instructions further cause the processor to:

generate a log indicating the inferred current emotional state of the operator, and

transmit, via the transceiver, the log to a server.

12. The electronic device of claim 10, wherein the computer executable instructions further cause the processor to:

determine, based on the inferred current emotional state, that the operator is currently not in an appropriate emotional state for operating the vehicle.

13. The electronic device of claim 10, further comprising:

a user interface;

wherein the computer executable instructions further cause the processor to;

generate a recommendation for the operator, the recommendation based on the inferred current emotional state, and

cause the user interface to present the recommendation.

14. The electronic device of claim 10, wherein the memory further stores a set of baseline images depicting a plurality of exemplary movements, gestures, or positions, and wherein to analyze the image data to identify the current behavior of the operator that is depicted in the image data, the processor is configured to:

access the set of baseline images,

compare the image data depicting the operator of the vehicle to the set of baseline images to identify a set of movements, gestures, or positions of the operator as depicted in the image data, and

determine the current behavior of the operator based on the set of movements, gestures, or positions of the operator as depicted in the image data.

15. The electronic device of claim 10, wherein to access the vehicle dynamics data associated with operation of the vehicle, the processor is configured to:

access vehicle dynamics data captured by one or more vehicle dynamics sensors associated with the vehicle.

16. The electronic device of claim 10, wherein to compare the vehicle dynamics data to the baseline vehicle dynamics data, the processor is configured to:

identify, from the vehicle dynamics data, the at least one vehicle dynamics data parameter and the vehicle dynamics data value thereof,

identify, from the baseline vehicle dynamics data, the baseline vehicle dynamics data value for the at least one vehicle dynamics data parameter,

calculate a difference between the vehicle dynamics data value and the baseline vehicle dynamics data value, and

determine that the difference exceeds the threshold amount.

17. The electronic device of claim 16, wherein to calculate the difference between the vehicle dynamics data value and the baseline vehicle dynamics data value, the processor is configured to:

calculate a percentage difference between the vehicle dynamics data value and the baseline vehicle dynamics data value, and

determine that the percentage difference exceeds a percentage threshold amount.

18. The electronic device of claim 10, wherein the memory further associates a plurality of inferred emotional states with a plurality of operator behaviors, and wherein to determine the inferred current emotional state of the operator, the processor is configured to:

determine, from the plurality of inferred emotional states, the inferred emotional state that matches the current behavior of the operator.