Driver monitoring system

• US 10,640,123 B2
• Filed: 02/27/2017
• Issued: 05/05/2020
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Morita, Hidetoshi, Watanabe, Masashi
• Primary Examiner(s): King, Curtis J
• Application Number:
  US15/780,056
• Publication Number:
  US 20180345980A1
• Time in Patent Office: 1,163 Days
• Field of Search: None
• CPC Class Codes: B60W 2040/0809 Driver authorisation; Driver identity check
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0881 Seat occupation; Driver or passenger presence
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/082 Selecting or switching between different modes of propelling
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0055 only part of driving tasks shifted to occupants
• CPC Class Codes: B60W 60/0059 Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A driving monitoring system monitors a driver of a host vehicle by a vehicle control unit and a portable terminal which communicate with each other. The vehicle control unit controls the host vehicle. The portable terminal includes a display screen and a camera arranged to capture an image of an area opposing the display screen. The driver monitoring system determines whether the driver of the host vehicle looks at the display screen of the portable terminal during running of the host vehicle, based on an image captured by the camera; the driver monitoring system issues a warning in response to that the driver is determined to be looking at the display screen of the portable terminal.

Claims

1. A driver monitoring system monitoring a driver of a host vehicle by a vehicle control unit and a portable unit which communicate with each other, the vehicle control unit controlling the host vehicle, the portable unit having a display screen and an image capture apparatus arranged to capture an image of an area opposing the display screen,

the driver monitoring system comprising:

an occupant state determiner section configured to determine whether the driver of the host vehicle looks at the display screen of the portable unit during running of the host vehicle, based on an image captured by the image capture apparatus;

a safety controller section configured to issue a warning in response to that the occupant state determiner section determines that the driver looks at the display screen of the portable unit; and

an image recognizer section configured to determine whether the driver of the host vehicle accords with a manipulator of the portable unit based on (i) an image captured by an in-vehicle camera disposed in front of a driver seat to capture an image of the driver seat and (ii) an image captured by the image capture apparatus included in the portable unit,

wherein in response to that the driver of the host vehicle is determined to accord with the manipulator of the portable unit, the occupant state determiner section is configured to determine whether the driver of the host vehicle looks at the display screen of the portable unit during running of the host vehicle, based on the image captured by the image capture apparatus.

2. The driving monitoring system according to claim 1, wherein

the safety controller section is configured to issue the warning from at least either the portable unit or the host vehicle.

3. The driving monitoring system according to claim 1, wherein

the safety controller section is configured to issue the warning using at least one of a control of a running state of the host vehicle, a sound, a speech, a light, a vibration, a wind, and a temperature.

4. The driving monitoring system according to claim 1, wherein

a part or all of a process to determine a state of the driver is executed by at least one of (i) the portable unit, (ii) a control unit in the host vehicle, (iii) a control unit in a different vehicle, and (iv) an apparatus other than a vehicle.

5. The driving monitoring system according to claim 4, wherein

the apparatus other than the vehicle is a cloud server.

6. A driver monitoring system monitoring a driver of a host vehicle by a vehicle control unit and a portable unit which communicate with each other, the vehicle control unit controlling the host vehicle, the portable unit having a display screen and an image capture apparatus arranged to capture an image of an area opposing the display screen,

the driver monitoring system comprising:

an occupant state determiner section configured to determine whether the driver of the host vehicle looks at the display screen of the portable unit during an automated driving of the host vehicle, based on an image captured by the image capture apparatus; and

a safety controller section configured to perform a control of issuing a warning in response to that the occupant state determiner section determines that the driver looks at the display screen of the portable unit.

7. The driving monitoring system according to claim 6,

wherein

the safety controller section is configured to set a content of the warning depending on a level of the automated driving.

8. The driving monitoring system according to claim 6, wherein

the safety controller section is configured to provide information depending on a level of the automated driving.

9. The driving monitoring system according to claim 6, wherein:

the occupant state determiner section is configured to determine a degree of consciousness of the driver based on an image captured by the image capture apparatus; and

the safety controller section is configured to perform setting a content of the warning based on a degree of consciousness of the driver determined by the occupant state determiner section.

10. The driving monitoring system according to claim 6, wherein

the occupant state determiner section is provided in either the portable unit or the vehicle control unit.

11. The driving monitoring system according to claim 6, wherein

the safety controller section is configured to issue the warning either (i) when an operation by the driver is necessary, or (ii) when an operation by the driver is presumed to be necessary.

12. The driving monitoring system according to claim 6, wherein

the safety controller section is configured to issue the warning either (i) when a condition to end the automated driving is satisfied or (ii) when the automated driving is presumed to end.

13. The driving monitoring system according to claim 6, wherein

the safety controller section is configured to issue the warning when the host vehicle arrives at a destination with the automated driving.

14. The driving monitoring system according to claim 6, wherein

the safety controller section is configured to issue the warning from at least either the portable unit or the host vehicle.

15. The driving monitoring system according to claim 6, wherein

the safety controller section is configured to issue the warning using at least one of a control of a running state of the host vehicle, a sound, a speech, a light, a vibration, a wind, and a temperature.

16. The driving monitoring system according to claim 6, wherein

a part or all of a process to determine a state of the driver is executed by at least one of (i) the portable unit, (ii) a control unit in the host vehicle, (iii) a control unit in a different vehicle, and (iv) an apparatus other than a vehicle.

17. The driving monitoring system according to claim 16, wherein

the apparatus other than the vehicle is a cloud server.

18. The driving monitoring system according to claim 6, further comprising:

an image recognizer section configured to determine whether the driver of the host vehicle accords with a manipulator of the portable unit based on (i) an image captured by an in-vehicle camera disposed in front of a driver seat to capture an image of the driver seat and (ii) an image captured by the image capture apparatus included in the portable unit,

wherein in response to that the driver of the host vehicle is determined to accord with the manipulator of the portable unit, the occupant state determiner section is configured to determine whether the driver of the host vehicle looks at the display screen of the portable unit during running of the host vehicle, based on the image captured by the image capture apparatus.

19. A driver monitoring system monitoring a driver of a host vehicle by (i) a vehicle control unit in the host vehicle and (ii) a portable unit communicating with the vehicle control unit,

the vehicle control unit controlling the host vehicle,

the portable unit including a display screen arranged on one of side faces of the portable unit and an image capture apparatus arranged on the one of the side faces of the portable unit to capture an image in front of the display screen,

the driver monitoring system comprising:

at least one control circuit configured to

determine whether the driver of the host vehicle looks at the display screen of the portable unit during an automated driving of the host vehicle, based on the image in front of the display screen of the portable unit captured by the image capture apparatus included in the portable unit; and

perform a control of providing information used by the driver in driving, in response to determining that the driver looks at the display screen of the portable unit.

20. The driver monitoring system according to claim 19, wherein

among the at least one control circuit, an individual control circuit is configured to include

(i) at least one hardware circuit including at least either an analog circuit or a digital circuit, or

(ii) at least one processing unit along with memory storing instructions, or

(iii) a combination of the at least one hardware circuit and the at least one processing unit along with memory storing instructions.

21. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to provide the information depending on a level of the automated driving.

22. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to provide the information depending on a level of the automated driving.

23. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to

determine a degree of consciousness of the driver based on an image captured by the image capture apparatus, and

perform providing the information based on the determined degree of consciousness of the driver.

24. The driving monitoring system according to claim 19, wherein

the at least one control circuit is provided in either the portable unit or the vehicle control unit.

25. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to provide the information either (i) when an operation by the driver is necessary, or (ii) when an operation by the driver is presumed to be necessary.

26. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to provide the information either (i) when a condition to end the automated driving is satisfied or (ii) when the automated driving is presumed to end.

27. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to provide the information when the host vehicle arrives at a destination with the automated driving.

28. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to provide the information from at least either the portable unit or the host vehicle.

29. The driving monitoring system according to claim 19, wherein

the at least one control circuit is configured to provide the information using at least one of a control of a running state of the host vehicle, a sound, a speech, a light, a vibration, a wind, and a temperature.

30. The driving monitoring system according to claim 19, wherein

a part or all of a process to determine a state of the driver is executed by at least one of (i) the portable unit, (ii) a control unit in the host vehicle, (iii) a control unit I a different vehicle, and (iv) an apparatus other than a vehicle.

31. The driving monitoring system according to claim 30, wherein

the apparatus other than the vehicle is a cloud server.

32. The driving monitoring system according to claim 19, wherein:

the at least one control circuit is configured to determine whether the driver of the host vehicle accords with a manipulator of the portable unit based on (i) an image captured by an in-vehicle camera disposed in front of a driver seat to capture an image of the driver seat and (ii) an image captured by the image capture apparatus included in the portable unit; and

in response to that the driver of the host vehicle is determined to accord with the manipulator of the portable unit, the at least one control circuit is configured to determine whether the driver of the host vehicle looks at the display screen of the portable unit during running of the host vehicle, based on the image captured by the image capture apparatus included in the portable unit.

33. A driver monitoring system monitoring a driver of a host vehicle by a vehicle control unit and a portable unit which communicate with each other, the vehicle control unit controlling the host vehicle, the portable unit having a display screen and an image capture apparatus arranged to capture an image of an area opposing the display screen,

the driver monitoring system comprising:

an occupant state determiner section configured to determine whether the driver of the host vehicle looks at the display screen of the portable unit during an automated driving of the host vehicle, based on an image captured by the image capture apparatus;

a safety controller section configured to perform at least either a control of issuing a warning or a control of providing information used by the driver in driving, in response to that the occupant state determiner section determines that the driver looks at the display screen of the portable unit; and

an image recognizer section configured to determine whether the driver of the host vehicle accords with a manipulator of the portable unit based on (i) an image captured by an in-vehicle camera disposed in front of a driver seat to capture an image of the driver seat and (ii) an image captured by the image capture apparatus included in the portable unit,

wherein in response to that the driver of the host vehicle is determined to accord with the manipulator of the portable unit, the occupant state determiner section is configured to determine whether the driver of the host vehicle looks at the display screen of the portable unit during running of the host vehicle, based on the image captured by the image capture apparatus.