Vehicle manipulation using occupant image analysis

• US 10,627,817 B2
• Filed: 01/19/2018
• Issued: 04/21/2020
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Zijderveld, Gabriele, el Kaliouby, Rana, Mahmoud, Abdelrahman N, Mavadati, Seyedmohammad
• Primary Examiner(s): Paige, Tyler D
• Application Number:
  US15/875,644
• Publication Number:
  US 20180143635A1
• Time in Patent Office: 823 Days
• Field of Search: 701 36
• CPC Class Codes: A61B 2576/02 specially adapted for a particular organ or body part
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: A61B 5/6893 Cars
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2040/089 Driver voice
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2540/043 Identity of occupants
• CPC Class Codes: B60W 2540/045 Occupant permissions
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/223 Posture, e.g. hand, foot, or seat position, turned or inclined
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2554/00 : Input parameters relating to objects
• CPC Class Codes: B60W 2554/4047 : Attentiveness, e.g. distracted by mobile phone
• CPC Class Codes: B60W 2554/408 : Traffic behavior, e.g. swarm
• CPC Class Codes: B60W 2555/20 : Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2556/10 : Historical data
• CPC Class Codes: B60W 2556/50 : of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 40/08 : related to drivers or passengers
• CPC Class Codes: B60W 60/001 : Planning or execution of driving tasks
• CPC Class Codes: G05D 1/0291 : Fleet control monitoring fleets in traffic control systems for road vehicles G08G1/127, G08G1/127
• CPC Class Codes: G06F 18/2414 : Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/176 : Dynamic expression
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G08G 1/096716 : where the received information does not generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096741 : where the source of the transmitted information selects which information to transmit to each vehicle
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G10L 15/1807 : using prosody or stress
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 25/48 : specially adapted for particular use
• CPC Class Codes: G10L 25/63 : for estimating an emotional state
• CPC Class Codes: G10L 25/90 : Pitch determination of speech signals

Abstract

Vehicle manipulation is performed using occupant image analysis. A camera within a vehicle is used to collect cognitive state data including facial data, on an occupant of a vehicle. A cognitive state profile is learned, on a first computing device, for the occupant based on the cognitive state data. The cognitive state profile includes information on absolute time. The cognitive state profile includes information on trip duration time. Voice data is collected and the cognitive state data is augmented with the voice data. Further cognitive state data is captured, on a second computing device, on the occupant while the occupant is in a second vehicle. The further cognitive state data is compared, on a third computing device, with the cognitive state profile that was learned for the occupant. The second vehicle is manipulated based on the comparing of the further cognitive state data.

Claims

1. A computer-implemented method for vehicle manipulation comprising:

collecting, using a camera within a vehicle, cognitive state data including facial data, on an occupant of a vehicle;

learning, on a first computing device, a cognitive state profile for the occupant based on the cognitive state data;

capturing, on a second computing device, further cognitive state data on the occupant while the occupant is in a second vehicle;

comparing, on a third computing device, the further cognitive state data with the cognitive state profile that was learned for the occupant; and

manipulating the second vehicle based on the comparing of the further cognitive state data.

2. The method of claim 1 wherein the cognitive state profile includes information on absolute time.

3. The method of claim 2 wherein the absolute time includes time of day, day of week, day of month, or time of year information.

4. The method of claim 1 further comprising collecting voice data and augmenting the cognitive state data with the voice data.

5. The method of claim 4 wherein the voice data includes audio data.

6. The method of claim 4 further comprising evaluating the voice data for timbre, prosody, vocal register, vocal resonance, pitch, loudness, speech rate, or language content.

7. The method of claim 4 wherein the augmenting is based on lexical analysis of the voice data that looks at sentiment.

8. The method of claim 4 wherein the voice data includes non-speech vocalizations.

9. The method of claim 8 wherein the non-speech vocalizations include grunts, yelps, squeals, snoring, sighs, laughter, filled pauses, unfilled pauses, or yawns.

10. The method of claim 1 wherein the occupant is a passenger within the vehicle.

11. The method of claim 10 wherein the vehicle is an autonomous vehicle.

12. The method of claim 1 wherein the vehicle is a semi-autonomous vehicle.

13. The method of claim 1 further comprising performing facial recognition on the occupant.

14. The method of claim 13 further comprising using the cognitive state profile across a fleet of vehicles.

15. The method of claim 1 wherein the vehicle and the second vehicle are a same vehicle.

16. The method of claim 1 wherein the vehicle and the second vehicle are different vehicles.

17. The method of claim 1 wherein the vehicle and the second vehicle are part of a fleet of vehicles.

18. The method of claim 1 wherein the manipulating includes a locking out operation;

recommending a break for the occupant;

recommending a different route;

recommending how far to drive;

responding to traffic;

adjusting seats, mirrors, climate control, lighting, music, audio stimuli, interior temperature for the second vehicle;

brake activation;

or steering control.

19. The method of claim 1 wherein the manipulating is based on a make for the second vehicle, a vehicle class for the second vehicle, tires for the second vehicle, a weather pattern, and a traffic pattern.

20. The method of claim 1 further comprising capturing cognitive state data on a second occupant and manipulating the second vehicle based on the cognitive state data for the occupant and the cognitive state data for the second occupant.

21. The method of claim 1 further comprising comparing the further cognitive state data with additional cognitive state profiles learned from additional occupants of additional vehicles.

22. The method of claim 21 wherein the additional cognitive state profiles are selected based on demographics of the occupant.

23. The method of claim 21 wherein the additional cognitive state profiles are selected based on characteristics of the vehicle of the occupant.

24. The method of claim 21 wherein the additional cognitive state profiles are selected based on a route being travelled.

25. A computer program product embodied in a non-transitory computer readable medium for vehicle manipulation, the computer program product comprising code which causes one or more processors to perform operations of:

collecting, using a camera within a vehicle, cognitive state data including facial data, on an occupant of a vehicle;

learning a cognitive state profile for the occupant based on the cognitive state data;

capturing further cognitive state data on the occupant while the occupant is in a second vehicle;

comparing the further cognitive state data with the cognitive state profile that was learned for the occupant; and

manipulating the second vehicle based on the comparing of the further cognitive state data.

26. A computer system for vehicle manipulation comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

collect, using a camera within a vehicle, cognitive state data including facial data, on an occupant of a vehicle;

learn a cognitive state profile for the occupant based on the cognitive state data;

capture further cognitive state data on the occupant while the occupant is in a second vehicle;

compare the further cognitive state data with the cognitive state profile that was learned for the occupant; and

manipulate the second vehicle based on the comparing of the further cognitive state data.