Vehicular cognitive data collection using multiple devices

• US 10,592,757 B2
• Filed: 02/01/2018
• Issued: 03/17/2020
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Mahmoud, Abdelrahman N., Mavadati, Seyedmohammad, Turcot, Panu James
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US15/886,275
• Publication Number:
  US 20180157923A1
• Time in Patent Office: 775 Days
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Vehicle cognitive data is collected using multiple devices. A user interacts with various pieces of technology to perform numerous tasks and activities. Reactions can be observed and cognitive states inferred from reactions to the tasks and activities. A first computing device within a vehicle obtains cognitive state data which is collected on an occupant of the vehicle from multiple sources, wherein the multiple sources include at least two sources of facial image data. A second computing device generates analysis of the cognitive state data which is collected from the multiple sources. A third computing device renders an output which is based on the analysis of the cognitive state data. The cognitive state data from multiple sources is tagged. The cognitive state data from the multiple sources is aggregated. The cognitive state data is interpolated when collection is intermittent. The cognitive state analysis is interpolated when the cognitive state data is intermittent.

Claims

1. A computer-implemented method for cognitive state analysis comprising:

obtaining cognitive state data which is collected on an occupant of a vehicle from multiple sources;

inferring estimated cognitive state data where the cognitive state data collected is intermittent, wherein the inferring generates estimated data for time periods during which cognitive state data was not collected from one or more of the multiple sources;

generating analysis of the cognitive state data which is collected from the multiple sources; and

rendering an output based on the analysis of the cognitive state data.

2. The method of claim 1 wherein the cognitive state data from multiple sources is tagged.

3. The method of claim 2 wherein the cognitive state data is tagged with an identity value for the occupant of a vehicle.

4. The method of claim 2 further comprising partitioning the cognitive state data based on tagging.

5. The method of claim 1 wherein the multiple sources include at least one mobile device.

6. The method of claim 5 wherein the at least one mobile device includes a front-side camera.

7. The method of claim 1 wherein the analysis includes aggregating the cognitive state data from the multiple sources.

8. The method of claim 1 further comprising interpolating cognitive state data when the cognitive state data collected is intermittent.

9. The method of claim 1 further comprising interpolating cognitive state analysis when the cognitive state data collected is intermittent.

10. The method of claim 1 further comprising collecting other cognitive state data from the occupant of a vehicle on a continuous basis.

11. The method of claim 1 further comprising imputing additional cognitive state data for one or more time periods during which no cognitive state data was collected.

12. The method of claim 1 further comprising determining contextual information.

13. The method of claim 1 further comprising inferring cognitive states based on the cognitive state data which was collected.

14. The method of claim 13 wherein the cognitive states inferred include one or more of drowsiness, fatigue, distraction, impairment, sadness, stress, happiness, anger, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, curiosity, humor, depression, envy, sympathy, embarrassment, poignancy, or mirth.

15. A computer program product embodied in a non-transitory computer readable medium for cognitive state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining cognitive state data which is collected on an occupant of a vehicle from multiple sources;

inferring estimated cognitive state data where the cognitive state data collected is intermittent, wherein the inferring generates estimated data for time periods during which cognitive state data was not collected from one or more of the multiple sources;

generating analysis of the cognitive state data which is collected from the multiple sources; and

rendering an output based on the analysis of the cognitive state data.

16. A computer system for cognitive state analysis comprising:

a memory which stores instructions;

one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain cognitive state data which is collected on an occupant of a vehicle from multiple sources;

infer estimated cognitive state data where the cognitive state data collected is intermittent, wherein inferring estimated cognitive state data generates estimated data for time periods during which cognitive state data was not collected from one or more of the multiple sources;

generate analysis of the cognitive state data which is collected from the multiple sources; and

render an output based on the analysis of the cognitive state data.