Personal emotional profile generation for vehicle manipulation

• US 10,796,176 B2
• Filed: 10/29/2018
• Issued: 10/06/2020
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Mahmoud, Abdelrahman N., Mavadati, Seyedmohammad, Zijderveld, Gabriele
• Primary Examiner(s): Paige, Tyler D
• Application Number:
  US16/173,160
• Publication Number:
  US 20190073547A1
• Time in Patent Office: 708 Days
• Field of Search: 701 36
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/0098 Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 30/0255 based on user history
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/096716 where the received information does not generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096725 where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096741 where the source of the transmitted information selects which information to transmit to each vehicle
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G10L 25/48 : specially adapted for particular use
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

Personal emotional profile generation uses cognitive state analysis for vehicle manipulation. Cognitive state data is obtained from an individual. The cognitive state data is extracted, using one or more processors, from facial images of an individual captured as they respond to stimuli within a vehicle. The cognitive state data extracted from facial images is analyzed to produce cognitive state information. The cognitive state information is categorized, using one or more processors, against a personal emotional profile for the individual. The vehicle is manipulated, based on the cognitive state information, the categorizing, and the stimuli. The personal emotional profile is generated by comparing the cognitive state information of the individual with cognitive state norms from a plurality of individuals and is based on cognitive state data for the individual that is accumulated over time. The cognitive state information is augmented based on audio data collected from within the vehicle.

Claims

1. A computer-implemented method for cognitive state analysis comprising:

obtaining cognitive state data from an individual within a vehicle, wherein the cognitive state data is extracted, using one or more processors, from facial images of the individual that are captured as the individual responds to stimuli;

analyzing the cognitive state data extracted from facial images to produce cognitive state information;

developing norms for the individual;

categorizing, using one or more processors, the cognitive state information against a personal emotional profile for the individual; and

manipulating the vehicle, based on the cognitive state information, the categorizing, and the stimuli.

2. The method of claim 1 wherein the personal emotional profile is generated by comparing the cognitive state information of the individual with cognitive state norms from a plurality of individuals.

3. The method of claim 1 wherein the personal emotional profile is generated based on cognitive state data for the individual that is accumulated over time.

4. The method of claim 3 wherein the personal emotional profile is further generated based on two or more vehicle journeys by the individual.

5. The method of claim 4 wherein at least two of the two or more vehicle journeys are accomplished in two or more vehicles.

6. The method of claim 1 wherein the stimuli occurs within the vehicle.

7. The method of claim 1 wherein the stimuli occurs outside of the vehicle.

8. The method of claim 1 further comprising pushing media content to the individual based on the categorizing.

9. The method of claim 1 wherein the categorizing is based on a plurality of expressions by the individual.

10. The method of claim 9 wherein the categorizing is further based on a rate of change in the plurality of expressions by the individual.

11. The method of claim 1 wherein the personal emotional profile further comprises off-line behavior.

12. The method of claim 1 further comprising correlating a component of the personal emotional profile to a purchase intent.

13. The method of claim 1 further comprising using the personal emotional profile to predict future reactions to stimuli.

14. The method of claim 1 further comprising using the personal emotional profile to categorize digital media.

15. The method of claim 1 further comprising matching the individual with one or more people from a plurality of people, based on the categorizing.

16. The method of claim 1 wherein the cognitive state data is collected while the individual views a collection of digital media.

17. The method of claim 16 wherein the cognitive state information from a plurality of people is based on the plurality of people viewing the collection of digital media.

18. The method of claim 1 further comprising using response differences in the individual, over an interval of time, from the norms for the individual to make a recommendation.

19. The method of claim 1 further comprising developing norms for a demographic group.

20. The method of claim 19 further comprising using response differences for the individual from the norms for the demographic group to refine a profile.

21. The method of claim 1 further comprising augmenting the cognitive state information based on audio data collected from within the vehicle, wherein the audio data is collected contemporaneously with the facial images.

22. The method of claim 21 wherein the audio data includes speech.

23. The method of claim 21 wherein the audio data includes non-speech vocalizations.

24. The method of claim 1 further comprising tagging the cognitive state data with sensor data.

25. The method of claim 24 wherein the sensor data includes one or more of vehicle interior temperature, vehicle exterior temperature, time of day, day of week, season, level of daylight, weather conditions, road conditions, traffic conditions, headlight activation, windshield wiper activation, vehicle settings, entertainment center selection, or entertainment center volume.

26. The method of claim 1 wherein the manipulating is based on a deviation from the norms.

27. A computer program product embodied in a non-transitory computer readable medium for cognitive state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining cognitive state data from an individual within a vehicle, wherein the cognitive state data is extracted, using one or more processors, from facial images of the individual that are captured as the individual responds to stimuli;

analyzing the cognitive state data extracted from facial images to produce cognitive state information;

developing norms for the individual;

categorizing, using one or more processors, the cognitive state information against a personal emotional profile for the individual; and

manipulating the vehicle, based on the cognitive state information, the categorizing, and the stimuli.

28. A computer system for cognitive state analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain cognitive state data from an individual within a vehicle, wherein the cognitive state data is extracted, using one or more processors, from facial images of the individual that are captured as the individual responds to stimuli;

analyze the cognitive state data extracted from facial images to produce cognitive state information;

develop norms for the individual;

categorize, using one or more processors, the cognitive state information against a personal emotional profile for the individual; and

manipulate the vehicle, based on the cognitive state information, the categorizing, and the stimuli.