DRIVER DISTRACTION DETECTION SYSTEM

• US 20160267335A1
• Filed: 03/13/2015
• Published: 09/15/2016
• Est. Priority Date: 03/13/2015
• Status: Abandoned Application

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Hampiholi, Vallabha Vasant
• Application Number:
  US14/657,070
• Publication Number:
  US 20160267335A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60K 28/06 responsive to incapacity of driver
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60R 16/037 for occupant comfort , e.g. for automatic adjustment of appliances according to personal settings, e.g. seats, mirrors, steering wheel
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/223 Posture, e.g. hand, foot, or seat position, turned or inclined
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: G08B 21/06 : indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G08B 5/36 : using visible light sources
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Embodiments are described for determining and responding to driver distractions. An example in-vehicle computing system of a vehicle includes an external device interface communicatively connecting the in-vehicle computing system to a mobile device, an inter-vehicle system communication module communicatively connecting the in-vehicle computing system to one or more vehicle systems of the vehicle, a processor, and a storage device storing instructions executable by the processor to receive image data from the mobile device, and determine a driver state based on the received image data. The instructions are further executable to receive vehicle data from one or more of the vehicle systems, determine a vehicle state based on the vehicle data, determine a distraction severity level based on the driver state and the vehicle state, and control one or more devices of the vehicle to perform a selected action based on the distraction severity level.

Claims

1. An in-vehicle computing system of a vehicle, the in-vehicle computing system comprising:

an external device interface communicatively connecting the in-vehicle computing system to a mobile device;

an inter-vehicle system communication module communicatively connecting the in-vehicle computing system to one or more vehicle systems of the vehicle;

a processor; and

a storage device storing instructions executable by the processor to;

receive image data from the mobile device via the external device interface, the image data imaging a driver and a driver environment;

determine a driver state based on the received image data;

responsive to determining that the driver state indicates that the driver is distracted;

receive vehicle data from one or more of the vehicle systems via the inter-vehicle system communication module;

determine a vehicle state based on the vehicle data;

determine a distraction severity level based on the driver state and the vehicle state; and

control one or more devices of the vehicle to perform a selected action based on the distraction severity level.

2. The in-vehicle computing system of claim 1, wherein

performing the selected action comprises presenting a visual warning responsive to the distraction severity level being within a first range;

performing the selected action comprises presenting an audio warning responsive to the distraction severity level being within a second range; and

performing the selected action comprises performing an automatic adjustment of a vehicle control warning responsive to the distraction severity level being within a third range.

3. The in-vehicle computing system of claim 2, further comprising a display device, wherein the visual warning comprises visual alert presented via the display device.

4. The in-vehicle computing system of claim 2, wherein the audio warning comprises an audio alert presented via one or more speakers in the vehicle.

5. The in-vehicle computing system of claim 2, wherein automatic adjustment of the vehicle control comprises automatic adjustment of engine operation.

6. The in-vehicle computing system of claim 1, wherein the mobile device comprises a wearable device including at least an outward-facing camera having a field of view that includes a vehicle environment, and a user-facing camera having a field of view that includes the driver of the vehicle.

7. The in-vehicle computing system of claim 6, wherein the instructions are further executable to receive position and motion data from the head-mounted device, determine the driver state based on image data and the position and motion data, and transmit image data comprising video data including one or more of the driver as imaged from the user-facing camera and the vehicle environment as imaged from the outward-facing camera.

8. The in-vehicle computing system of claim 7, wherein the image data includes an indication of driver gaze and objects of interest in a travel path of the vehicle, and wherein the driver state indicates that the driver is distracted responsive to determining that the driver gaze is directed to one or more objects of interest for a threshold period of time.

9. The in-vehicle computing system of claim 8, wherein the image data includes an indication of a trajectory of the objects of interest, and wherein the distraction severity level is based on a comparison of the trajectory of the objects of interest and the travel path of the vehicle.

10. A method for an in-vehicle computing system of a vehicle, the method comprising:

receiving driver data from a wearable device, the driver data including image data from a driver-facing camera;

receiving object data from one or more imaging devices of at least one of the wearable device and the vehicle, the object data including image data of a vehicle environment;

receiving vehicle data from one or more vehicle systems, the vehicle data including an indication of an operating condition of the vehicle;

determining whether a driver is distracted by correlating the driver data with the object data;

responsive to determining that the driver is distracted, selecting an action based on correlating the driver data with the object data and the vehicle data and performing the selected action; and

responsive to determining that the driver is not distracted, maintaining current operating parameters.

11. The method of claim 10, wherein the object data includes:

a total number of objects in a vehicle environment as determined from the image data from the one or more outward-facing cameras, and

object trajectory information for each of the objects in the vehicle environment as determined from a comparison of a plurality of frames of image data from the one or more outward-facing cameras, the object trajectory information indicating an estimated trajectory of each of the objects.

12. The method of claim 11, wherein vehicle data includes vehicle trajectory information determined from one or more of a navigational system of the vehicle, sensor output of the wearable device, and image data from the one or more outward-facing cameras, the vehicle trajectory information indicating an estimated trajectory of the vehicle.

13. The method of claim 12, further comprising comparing the estimated trajectory of each of the objects and the estimated trajectory of the vehicle to determine intersections between the estimated trajectories of the objects and the estimated trajectory of the vehicle, wherein the selected action is selected based on the number of intersections between the estimated trajectories of the objects and the estimated trajectory of the vehicle.

14. The method of claim 13, wherein the selected action is further selected based on a vehicle speed and a gaze direction of the driver.

15. The method of claim 13, wherein a first action is selected responsive to determining that the estimated trajectory of at least one of the objects intersects the estimated trajectory of the vehicle and the vehicle speed is below a speed threshold, and a second action is selected responsive to determining that the estimated trajectory of at least one of the objects intersects the estimated trajectory of the vehicle, the vehicle speed is above the speed threshold, and the gaze direction of the driver intersected the current location of each of the at least one objects within a threshold time period and for a threshold duration.

16. The method of claim 15, wherein a third action is selected responsive to determining that the estimated trajectory of at least one of the objects intersects the estimated trajectory of the vehicle, the vehicle speed is above the speed threshold, and the gaze direction of the driver did not intersect the current location of each of the at least one objects within the threshold time period or for the threshold duration.

17. The method of claim 16, wherein the first action is a visual alert presented via a display in the vehicle, the second action is an audible alert presented via one or more speakers in the vehicle, and the third action is a vehicle control command issued from the in-vehicle computing system to a vehicle system to control engine operation of the vehicle.

18. The method of claim 10, wherein maintaining the current operating parameters comprises not performing an action that is based on correlating the driver data with the object data and the vehicle data.

19. A system for identifying driver distraction, the system comprising:

a wearable device including a driver-facing camera, an outward-facing camera, and one or more sensors;

an in-vehicle computing system communicatively connected to the wearable device and one or more vehicle systems, the in-vehicle computing system comprising a first processor and a first storage device; and

a cloud computing device remote from the in-vehicle computing system and communicatively connected to the in-vehicle computing system via a network, the cloud computing device comprising a second processor and a second storage device,

one or more of the first storage device and the second storage device storing first instructions executable by a respective one or more of the first processor and the second processor to;

receive image data from the driver-facing camera and sensor data from the one or more sensors of the wearable device indicating a driver state,

receive image data from the outward-facing camera of the wearable device indicating object states of one or more objects,

receive vehicle data from one or more vehicle systems to indicate vehicle state, and

select an action to be performed based on the indicated driver state, object states, and vehicle state,

the first storage device storing second instructions executable by the first processor to transmit a command to one or more of a display device of the in-vehicle computing system, an audio device of the vehicle, and an engine control unit of the vehicle to perform the selected action.

20. The system of claim 19, wherein the vehicle state includes a trajectory of the vehicle, the object states include trajectories of the one or more objects, and the driver state includes a gaze direction of the driver.