VEHICULAR COGNITIVE DATA COLLECTION USING MULTIPLE DEVICES

• US 20180157923A1
• Filed: 02/01/2018
• Published: 06/07/2018
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, JR., Rana, Mahmoud, Abdelrahman N., Mavadati, Seyedmohammad, Turcot, Panu James
• Granted Patent: US 10,592,757 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Vehicle cognitive data is collected using multiple devices. A user interacts with various pieces of technology to perform numerous tasks and activities. Reactions can be observed and cognitive states inferred from reactions to the tasks and activities. A first computing device within a vehicle obtains cognitive state data which is collected on an occupant of the vehicle from multiple sources, wherein the multiple sources include at least two sources of facial image data. A second computing device generates analysis of the cognitive state data which is collected from the multiple sources. A third computing device renders an output which is based on the analysis of the cognitive state data. The cognitive state data from multiple sources is tagged. The cognitive state data from the multiple sources is aggregated. The cognitive state data is interpolated when collection is intermittent. The cognitive state analysis is interpolated when the cognitive state data is intermittent.

Claims

1. A computer-implemented method for cognitive state analysis comprising:

obtaining, on a first computing device within a vehicle, cognitive state data which is collected on an occupant of the vehicle from multiple sources, wherein the multiple sources include at least two sources of facial image data;

generating, using a second computing device, analysis of the cognitive state data which is collected from the multiple sources; and

rendering, on a third computing device, an output based on the analysis of the cognitive state data.

2. The method of claim 1 wherein the cognitive state data from multiple sources is tagged.

3. The method of claim 2 wherein the cognitive state data is tagged with an identity value for the occupant of a vehicle.

5. The method of claim 2 further comprising partitioning the cognitive state data based on tagging.

6. The method of claim 1 wherein the at least two sources of facial image data include one or more of an inside vehicle camera, an outside vehicle camera, a webcam, a phone camera, a tablet camera, a wearable camera, a room camera, a mobile device, a cell phone, a tablet computer, or a laptop computer.

7. The method of claim 1 further comprising collecting audio data and augmenting the cognitive state data with the audio data.

8. The method of claim 7 wherein the audio data includes voice data.

9. The method of claim 8 further comprising evaluating the voice data for timbre, prosody, vocal register, vocal resonance, pitch, loudness, speech rate, or language content.

10. The method of claim 8 wherein the voice data includes non-speech vocalizations.

11. The method of claim 10 wherein the non-speech vocalizations include grunts, yelps, squeals, snoring, sighs, laughter, filled pauses, unfilled pauses, or yawns.

12. The method of claim 1 wherein the multiple sources include at least one mobile device.

13. The method of claim 12 wherein the at least one mobile device includes a front-side camera.

15. The method of claim 1 wherein the analysis includes aggregating the cognitive state data from the multiple sources.

17. The method of claim 1 wherein the facial image data is collected intermittently while the occupant of a vehicle is looking toward a camera.

18. The method of claim 17 further comprising performing face detection to determine whether the occupant of a vehicle is looking toward the camera.

19. The method of claim 18 further comprising filtering out faces of one or more other people to determine whether the occupant of a vehicle is looking toward the camera.

20. The method of claim 1 further comprising interpolating cognitive state data when the cognitive state data collected is intermittent.

21. The method of claim 1 further comprising interpolating cognitive state analysis when the cognitive state data collected is intermittent.

22. The method of claim 1 further comprising collecting other cognitive state data from the occupant of a vehicle on a continuous basis.

24. The method of claim 1 further comprising imputing additional cognitive state data for one or more time periods during which no cognitive state data was collected.

25. The method of claim 1 further comprising determining contextual information.

27. The method of claim 1 further comprising inferring cognitive states based on the cognitive state data which was collected.

28. The method of claim 27 wherein the cognitive states inferred include one or more of drowsiness, fatigue, distraction, impairment, sadness, stress, happiness, anger, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, curiosity, humor, depression, envy, sympathy, embarrassment, poignancy, or mirth.

38. The method of claim 1 further comprising:

displaying the rendering in visual format to one or more occupants of the vehicle;

delivering an auditory message complementing the rendering, wherein the auditory message provides clarification of the rendering;

soliciting, using a digital voice, verbal feedback from a primary occupant of the vehicle; and

controlling operation of the vehicle, based on the verbal feedback that was solicited and the analysis of the cognitive state data.

39. The method of claim 38 further comprising obtaining additional cognitive state data based on the controlling operation of the vehicle.

40. The method of claim 1 further comprising interpolating the cognitive state analysis where the cognitive state data collected is intermittent, wherein the interpolating generates interpolated data for time periods during which cognitive state data was not collected from one or more of the multiple sources.

4. (canceled)

14. (canceled)

16. (canceled)

23. (canceled)

26. (canceled)

29-37. -37. (canceled)

41. (canceled)

42. A computer program product embodied in a non-transitory computer readable medium for cognitive state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining cognitive state data which is collected on an occupant of a vehicle from multiple sources, wherein the multiple sources include at least two sources of facial image data;

generating analysis of the cognitive state data which is collected from the multiple sources; and

rendering an output based on the analysis of the cognitive state data.

43. A computer system for cognitive state analysis comprising:

a memory which stores instructions;

one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain cognitive state data which is collected on an occupant of a vehicle from multiple sources, wherein the multiple sources include at least two sources of facial image data;

generate analysis of the cognitive state data which is collected from the multiple sources; and

render an output based on the analysis of the cognitive state data.