Apparatus and method to train autonomous driving model, and autonomous driving apparatus

• US 10,034,630 B2
• Filed: 11/15/2016
• Issued: 07/31/2018
• Est. Priority Date: 11/16/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Dong Hwa, Kim, Chang Hyun, Shim, Eun Soo, Choi, Ki Hwan, Choi, Hyun Jin
• Primary Examiner(s): Kan, Yuri
• Application Number:
  US15/351,629
• Publication Number:
  US 20170135621A1
• Time in Patent Office: 623 Days
• CPC Class Codes: A61B 3/0025 characterised by electronic signal processing, e.g. eye models
• CPC Class Codes: A61B 3/10 Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/225 of the fingers, e.g. by monitoring hand-grip force
• CPC Class Codes: A61B 5/318 Heart-related electrical modalities, e.g. electrocardiography [ECG]
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G06N 3/045 Combinations of networks

Abstract

Disclosed is an apparatus and method to train an autonomous driving model. The apparatus includes a driver information collection processor configured to collect driver information while a vehicle is being driven. The apparatus also includes a sensor information collection processor configured to collect sensor information from a sensor installed in the vehicle while the vehicle is being driven, and a model training processor configured to train the autonomous driving model based on the driver information and the sensor information.

Claims

1. An apparatus to train an autonomous driving model, the apparatus comprising:

a driver information collection processor configured to collect driver information while a vehicle is being driven by a driver;

a sensor information collection processor configured to collect sensor information from a sensor installed in the vehicle while the vehicle is being driven;

a model training processor configured to train the autonomous driving model based on the driver information and the sensor information; and

a model determination processor configured to determine whether the autonomous driving model is appropriate for the driver or a current driving situation based on either one or both of the driver information or navigation setting information comprising information on a starting location and an ending location.

2. The apparatus of claim 1, wherein the vehicle is driven in at least one of a real-road driving situation and a virtual driving situation.

3. The apparatus of claim 1, wherein the driver information comprises at least one of driver state information, driver behavior information, and driving pattern information.

4. The apparatus of claim 3, wherein,

the driver state information comprises at least one of information about a gaze direction, a heart rate, an electrocardiogram, and brain waves of a driver, and grip strength of the driver'"'"'s hand holding a steering wheel,

the driver behavior information comprises at least one of information about manipulation of a stereo device, air conditioning device, a mirror device, and a navigation device installed in the vehicle, and

the driving pattern information comprises at least one of information about a steering wheel manipulation information, horn manipulation information, turn-signal manipulation information, emergency button manipulation information, and brake pedal manipulation information.

5. The apparatus of claim 4, wherein the model training processor trains the autonomous driving model to learn about a driving area based on information about the gaze direction of the driver and trains the autonomous driving model to learn at least one of a dangerous situation and a level of danger in driving based on the driving pattern information.

6. The apparatus of claim 1, wherein the model training processor trains the autonomous driving model to learn about an integrated driving situation from the starting location to the ending location and based on the navigation setting information.

7. The apparatus of claim 1, wherein the sensor information comprises at least one of surrounding image information, surrounding voice information, current location information, light detection and ranging (lidar) sensor information, radar sensor information, vehicle speed information, and acceleration information.

8. The apparatus of claim 1, wherein the autonomous driving model is based on a deep neural network configured to learn spatial information and temporal information.

24. The apparatus of claim 1, further comprising:

a model training processor configured to retrain the autonomous driving model based on the collected driving information upon determining that the autonomous driving model is inappropriate; and

a driving control processor configured to automatically control operation of the vehicle using the collected driving information and the autonomous driving model upon determining that the autonomous driving model is appropriate.

25. The apparatus of claim 24, wherein the driving control processor is further configured to:

upon determining by the model determination processor that the autonomous driving model is inappropriate, request the driver to confirm whether to retrain the autonomous driving model.

9. A method to train an autonomous driving model, the method comprising:

collecting driver information while a vehicle is being driven by a driver;

collecting sensor information from a sensor installed in the vehicle while the vehicle is being driven;

training the autonomous driving model based on the driver information and the sensor information; and

determining whether the autonomous driving model is appropriate for the driver or a current driving situation based on either one or both of the driver information or navigation setting information comprising information on a starting location and an ending location.

10. The method of claim 9, wherein the driver information comprises at least one of driver state information, driver behavior information, and driving pattern information.

11. The method of claim 10, wherein,

the driver state information comprises at least one of information about a gaze direction, a heart rate, an electrocardiogram, and brain waves of a driver, and grip strength of the driver'"'"'s hand holding a steering wheel,

the driver behavior information comprises at least one of information about manipulation of a stereo device, air conditioning device, a mirror device, and a navigation device installed in the vehicle, and

the driving pattern information comprises at least one of a steering wheel manipulation information, horn manipulation information, turn-signal manipulation information, emergency button manipulation information, and brake pedal manipulation information.

12. The method of claim 11, wherein the training of the model comprises training the autonomous driving model to learn about a driving area based on information about the gaze direction of the driver and training the autonomous driving model to learn about at least one of a dangerous situation and a level of danger in driving based on the driving pattern information.

13. The method of claim 9, wherein the training of the model comprises training the autonomous driving model to learn about an integrated driving situation from the starting location to the ending location further based on the navigation setting information.

14. The method of claim 9, wherein the sensor information comprises at least one of surrounding image information, surrounding voice information, current location information, light detection and ranging (lidar) sensor information, radar sensor information, vehicle speed information, and acceleration information.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 9.

16. An autonomous driving apparatus, comprising:

a driving information collection processor configured to collect vehicle driving information comprising at least one of driver information, sensor information, and navigation setting information while a vehicle is being driven;

a command receiver configured to receive an autonomous driving command from a driver; and

a command processor configured to

train an autonomous driving model installed in the vehicle or autonomously drive the vehicle based on the collected driving information, and

determine whether to execute the autonomous driving command based on the autonomous driving model.

17. The autonomous driving apparatus of claim 16, wherein the command processor comprises:

a model determination processor configured to determine whether the autonomous driving model is appropriate for a driver and the current driving situation based on either one or both of the driver information and the navigation setting information;

a model training processor configured to retrain the autonomous driving model based on the collected driving information upon determining that the autonomous driving model is inappropriate; and

a driving control processor configured to automatically control operation of the vehicle using the collected driving information and the autonomous driving model upon determining that the autonomous driving model is appropriate.

18. The autonomous driving apparatus of claim 17, wherein the model determination processor checks whether the autonomous driving model is trained to learn about the driver based on the driver information and determines that the autonomous driving model is inappropriate upon determining that the autonomous driving model is untrained about the driver.

19. The autonomous driving apparatus of claim 18, wherein the model determination processor checks whether the autonomous driving model is trained to learn about a driving route from a starting location to an ending location based on the navigation setting information and determines that the autonomous driving model is inappropriate upon determining that the autonomous driving model is untrained about the driving route.

20. An apparatus, comprising:

a processor configured to

collect driver information corresponding to behavior information of a driver driving a vehicle, virtually or in real-time,

collect sensor information from sensors capturing either one or both of image and ambient information surrounding the vehicle and vehicle operational parameters corresponding to either one or both of a current driving route and current driving conditions,

train an autonomous driving model by combining the driver information and the sensor information, and

execute the autonomous driving model upon determining that the trained autonomous driving model corresponds to at least one of the driver or the current driving route.

21. The apparatus of claim 20, wherein the processor comprises:

a driver information collection processor configured to collect the driver information,

a sensor information collection processor configured to collect the sensor information corresponding to either one or both of the current driving route and the current driving conditions,

a model training processor configured to train the autonomous driving model, and

a driving control processor configured to execute the autonomous driving model upon determining that the trained autonomous driving model corresponds to at least one of the driver, the current driving route.

22. The apparatus of claim 20, further comprising:

a navigation information collection processor configured to collect navigation setting information comprising information regarding a starting location, a stopover location, and an ending location.

23. The apparatus of claim 20, wherein the processor is further configured to check whether the autonomous driving model has been trained within a preset period of time to prevent a retraining of the autonomous driving model.