Image analysis for two-sided data hub

• US 10,401,860 B2
• Filed: 03/12/2018
• Issued: 09/03/2019
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Krupat, Jason, el Kaliouby, Rana, Radice, Jason, Zijderveld, Gabriele, Cabot, Chilton Lyons
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US15/918,122
• Publication Number:
  US 20180196432A1
• Time in Patent Office: 540 Days
• CPC Class Codes: A61B 2576/02 specially adapted for a particular organ or body part
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/6893 Cars
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/223 Posture, e.g. hand, foot, or seat position, turned or inclined
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: G05D 1/0291 Fleet control monitoring fleets in traffic control systems for road vehicles G08G1/127, G08G1/127
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/772 : Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/174 : Facial expression recognition
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G10L 15/1807 : using prosody or stress
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 25/63 : for estimating an emotional state
• CPC Class Codes: G10L 25/90 : Pitch determination of speech signals

Abstract

Image analysis is performed for a two-sided data hub. Data reception on a first computing device is enabled by an individual and a content provider. Cognitive state data including facial data on the individual is collected on a second computing device. The cognitive state data is analyzed on a third computing device and the analysis is provided to the individual. The cognitive state data is evaluated and the evaluation is provided to the content provider. A mood dashboard is displayed to the individual based on the analyzing. The individual opts in to enable data reception for the individual. The content provider provides content via a website.

Claims

1. A computer-implemented method for image analysis comprising:

collecting cognitive state data including facial data on an individual in a first vehicle;

analyzing the cognitive state data using a neural network trained to identify facial expressions and cognitive states;

generating a cognitive state profile for the individual in the first vehicle based on the analyzing;

transmitting the cognitive state profile to a two-sided data hub; and

providing vehicular manipulation of a second vehicle from the two-sided data hub.

2. The method of claim 1 wherein the collecting cognitive state data, including facial data on the individual, is accomplished using a webcam.

3. The method of claim 2 wherein the collecting cognitive state data further includes capturing physiological data.

4. The method of claim 1 wherein the collecting of cognitive state data is accomplished within the first vehicle.

5. The method of claim 1 wherein the analyzing further comprises comparing the cognitive state data with collected cognitive state data from other individuals.

6. The method of claim 5 wherein the comparing provides quantified self-information on cognitive states for the individual.

7. The method of claim 1 wherein the analyzing includes using an image classifier from a plurality of image classifiers to detect emotional content within the facial data for the individual.

8. The method of claim 1 wherein the analyzing the cognitive state data is used to enable the individual to track cognitive health.

9. The method of claim 1 further comprising augmenting the analyzing using audio data from the individual.

10. The method of claim 9 wherein the audio data includes voice data from the individual.

11. The method of claim 1 wherein the cognitive state data is indicative of drowsiness, fatigue, distraction, impairment, sadness, stress, happiness, anger, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, curiosity, humor, depression, envy, sympathy, embarrassment, poignancy, or mirth.

14. The method of claim 1, further comprising providing driving directions to the individual;

identifying a change in a cognitive state of the individual;

modifying the driving directions as a function of the change; and

providing the modified driving directions to the individual.

15. The method of claim 1, further comprising:

collecting further cognitive state data including facial data on an individual in the second vehicle;

analyzing the further cognitive state data using a neural network trained to identify facial expressions and cognitive states;

generating a cognitive state profile for the individual in the second vehicle based on the analyzing the further cognitive state data;

transmitting the cognitive state profile for the individual in the second vehicle to the two-sided data hub; and

providing vehicular manipulation of the first or second vehicle from the two-sided data hub.

16. The method of claim 1 wherein the vehicular manipulation includes adjusting an interior temperature.

17. The method of claim 1 wherein the vehicular manipulation includes brake activation.

12. A computer program product embodied in a non-transitory computer readable medium for image analysis, the computer program product comprising code which causes one or more processors to perform operations of:

collecting cognitive state data including facial data on an individual in a first vehicle;

analyzing the cognitive state data using a neural network trained to identify facial expressions and cognitive states;

generating a cognitive state profile for the individual in the first vehicle based on the analyzing;

transmitting the cognitive state profile to a two-sided data hub; and

providing vehicular manipulation of a second vehicle from the two-sided data hub.

13. A computer system for image analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

collect cognitive state data including facial data on an individual in a first vehicle;

analyze the cognitive state data using a neural network trained to identify facial expressions and cognitive states;

generate a cognitive state profile for the individual in the first vehicle based on the analyzing;

transmit the cognitive state profile to a two-sided data hub; and

provide vehicular manipulation of a second vehicle from the two-sided data hub.