System and method for transitioning between an autonomous and manual driving mode based on detection of a drivers capacity to control a vehicle

• US 10,471,963 B2
• Filed: 04/07/2017
• Issued: 11/12/2019
• Est. Priority Date: 04/07/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Huang, Zehua, Wang, Panqu, Chen, Pengfei
• Primary Examiner(s): Elchanti, Hussein
• Application Number:
  US15/482,624
• Publication Number:
  US 20180290660A1
• Time in Patent Office: 949 Days
• CPC Class Codes: B60K 2360/161 Explanation of functions, e.g. instructions
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/215 Selection or confirmation of options
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/082 Selecting or switching between different modes of propelling
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0053 from vehicle to occupant
• CPC Class Codes: B60W 60/0059 Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A system and method for transitioning between an autonomous and manual driving mode based on detection of a driver'"'"'s capacity to control a vehicle are disclosed. A particular embodiment includes: receiving sensor data related to a vehicle driver'"'"'s capacity to take manual control of an autonomous vehicle; determining, based on the sensor data, if the driver has the capacity to take manual control of the autonomous vehicle, the determining including prompting the driver to perform an action or provide an input; and outputting a vehicle control transition signal to a vehicle subsystem to cause the vehicle subsystem to take action based on the driver'"'"'s capacity to take manual control of the autonomous vehicle.

Claims

1. A system comprising:

a data processor; and

a driving control transition module, executable by the data processor, the driving control transition module being configured to perform a driving control transition operation for an autonomous vehicle, the driving control transition operation being configured to;

train a neural network of the driving control transition module using training data received from a remote source and use the trained neural network to modify the operation of the driving control transition module based on a vehicle context in which the autonomous vehicle is operating, the context including a current vehicle location;

receive sensor data related to a vehicle driver'"'"'s capacity to take manual control of an autonomous vehicle and receive sensor data related to the vehicle context, the sensor data including image data from a camera in the vehicle, the driving control transition operation being configured to use the image data to perform facial feature analysis of a face of the driver;

classify the vehicle driver into at least one of a plurality of driver state classifications based on the facial feature analysis of the face of the driver;

determine, based on the sensor data and the at least one of the plurality of driver state classifications, if the driver has the capacity to take manual control of the autonomous vehicle, wherein the driving control transition operation is configured to prompt the driver to perform an action or provide an input; and

output a vehicle control transition signal to a vehicle subsystem to cause the vehicle subsystem to take action based on the driver'"'"'s capacity to take manual control of the autonomous vehicle, the vehicle control transition signal corresponding to the at least one of the plurality of driver state classifications, the vehicle control transition signal causing the vehicle subsystem to direct the vehicle to safely pull over to the side of a roadway, if the driver is determined to not have the capacity to take manual control of the vehicle.

2. The system of claim 1 wherein the driving control transition operation being configured to determine, based on the sensor data, if the driver has performed the prompted action or provided the prompted input.

3. The system of claim 1 wherein the prompted action or input is an instruction for the driver to activate a button or speak an utterance.

4. The system of claim 1 wherein the prompted action or input is an instruction for the driver to respond to a prompt with a voice utterance, which is analyzed to determine the driver'"'"'s capacity to take manual control of the autonomous vehicle.

5. The system of claim 1 wherein the prompted action or input is an instruction for the driver to manipulate a vehicle control in a prompted manner.

6. A method comprising:

training a neural network of a driving control transition module using training data received from a remote source and using the trained neural network for modifying the operation of the driving control transition module based on a vehicle context in which an autonomous vehicle is operating, the context including a current vehicle location;

receiving sensor data related to a vehicle driver'"'"'s capacity to take manual control of an autonomous vehicle and receiving sensor data related to the vehicle context, the sensor data including image data from a camera in the vehicle;

using the image data to perform facial feature analysis of a face of the driver;

classifying the vehicle driver into at least one of a plurality of driver state classifications based on the facial feature analysis of the face of the driver;

determining, based on the sensor data and the at least one of the plurality of driver state classifications, if the driver has the capacity to take manual control of the autonomous vehicle, the determining including prompting the driver to perform an action or provide an input; and

outputting a vehicle control transition signal to a vehicle subsystem to cause the vehicle subsystem to take action based on the driver'"'"'s capacity to take manual control of the autonomous vehicle, the vehicle control transition signal corresponding to the at least one of the plurality of driver state classifications, the vehicle control transition signal causing the vehicle subsystem to direct the vehicle to safely pull over to the side of a roadway, if the driver is determined to not have the capacity to take manual control of the vehicle.

7. The method of claim 6 wherein the determining including determining, based on the sensor data, if the driver has performed the prompted action or provided the prompted input.

8. The method of claim 6 wherein the prompted action or input is an instruction for the driver to activate a button or speak an utterance.

9. The method of claim 6 wherein the prompted action or input is an instruction for the driver to respond to a prompt with a voice utterance, which is analyzed to determine the driver'"'"'s capacity to take manual control of the autonomous vehicle.

10. The method of claim 6 wherein the prompted action or input is an instruction for the driver to manipulate a vehicle control in a prompted manner.

11. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

train a neural network of a driving control transition module using training data received from a remote source and use the trained neural network to modify the operation of the driving control transition module based on a vehicle context in which an autonomous vehicle is operating, the context including a current vehicle location;

receive sensor data related to a vehicle driver'"'"'s capacity to take manual control of an autonomous vehicle and receive sensor data related to the vehicle context, the sensor data including image data from a camera in the vehicle;

use the image data to perform facial feature analysis of a face of the driver;

classify the vehicle driver into at least one of a plurality of driver state classifications based on the facial feature analysis of the face of the driver;

determine, based on the sensor data and the at least one of the plurality of driver state classifications, if the driver has the capacity to take manual control of the autonomous vehicle, the instructions being further configured to prompt the driver to perform an action or provide an input; and

output a vehicle control transition signal to a vehicle subsystem to cause the vehicle subsystem to take action based on the driver'"'"'s capacity to take manual control of the autonomous vehicle, the vehicle control transition signal corresponding to the at least one of the plurality of driver state classifications, the vehicle control transition signal causing the vehicle subsystem to direct the vehicle to safely pull over to the side of a roadway, if the driver is determined to not have the capacity to take manual control of the vehicle.

12. The non-transitory machine-useable storage medium of claim 11 wherein the instructions are further configured to determine, based on the sensor data, if the driver has performed the prompted action or provided the prompted input.

13. The non-transitory machine-useable storage medium of claim 11 wherein the prompted action or input is an instruction for the driver to activate a button or speak an utterance.

14. The non-transitory machine-useable storage medium of claim 11 wherein the prompted action or input is an instruction for the driver to manipulate a vehicle control in a prompted manner.