Sensor fusion for autonomous driving transition control

• US 10,481,602 B2
• Filed: 10/17/2016
• Issued: 11/19/2019
• Est. Priority Date: 10/17/2016
• Status: Active Grant

• Current Assignee: Steering Solutions IP Holding Corporation (Nexteer Automotive Group Ltd.)
• Original Assignee: Steering Solutions IP Holding Corporation (Nexteer Automotive Group Ltd.)
• Inventors: Chandy, Ashok
• Primary Examiner(s): Wiltey, Nicholas K
• Application Number:
  US15/295,039
• Publication Number:
  US 20180107214A1
• Time in Patent Office: 1,128 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2510/20 Steering systems
• CPC Class Codes: B60W 2510/202 Steering torque
• CPC Class Codes: B60W 2510/205 Steering speed
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B62D 1/286 Systems for interrupting non-mechanical steering due to driver intervention
• CPC Class Codes: B62D 15/025 Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation B62D1/28 takes precedence
• CPC Class Codes: B62D 5/0424 the axes of motor and final driven element of steering gear, e.g. rack, being parallel
• CPC Class Codes: B62D 6/10 characterised by means for sensing or determining torque
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa

Abstract

A system for sensor fusion for autonomous driving transition control includes a sensor fusion module and a decision making module. The sensor fusion module fuses a plurality of steering sensor data from one or more sensors of a steering system with a plurality of driver state data from a plurality of driver state sensors as a plurality of fused state data aligned in time. The decision making module determines whether to transition from an autonomous driving mode to a manual driving mode based on the fused state data.

Claims

1. A system for sensor fusion for autonomous driving transition control, the system comprising:

a processor operable to receive a plurality of steering sensor data from one or more sensors of a steering system and receive a plurality of driver state data from a plurality of driver state sensors, the processor configured to;

fuse the steering sensor data with the driver state data as a plurality of fused state data aligned in time;

determine whether to transition from an autonomous driving mode to a manual driving mode based on the fused state data;

command a steering motor actuator of the steering system autonomously in the autonomous driving mode;

command the steering motor actuator responsive to a driver input in the manual driving mode;

determine an attentive state of a driver based on time-averaging of the fused state data;

continue monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated to determine an attentive state of a driver based on time-averaging of the fused state data; and

revert from the manual driving mode to the autonomous driving mode based on determining that the attentive state of the driver no longer meets one or more transition validity criteria.

2. The system of claim 1, wherein the steering sensor data comprise sensed or derived data from one or more of a handwheel torque sensor and a handwheel position/velocity sensor.

3. The system of claim 1, wherein the driver state data comprise sensed or derived data from one or more of handwheel touch sensor and a camera.

4. The system of claim 3, wherein the driver state data sensed or derived from the handwheel touch sensor comprises one or more of:

an area of contact with a handwheel, a width of contact with the handwheel, a force of contact with the handwheel, and a position of contact with the handwheel, and wherein the driver state data sensed or derived from the camera comprises one or more of;

a driver body posture, a driver head pose, a driver eye gaze, and a driver hand position.

5. The system of claim 1, further comprising a recent data store of the fused state data and a history data store of the fused state data, wherein the history data store is operable to collect older values of the fused state data than collected in the recent data store.

6. The system of claim 5, wherein the processor is configured to average multiple samples of the fused state data from the history data store as time-averaged fused state data and determine whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data from the recent data store in combination with the time-averaged fused state data.

7. The system of claim 1, wherein the processor is further configured to revert from the manual driving mode to the autonomous driving mode based on determining that an input from one or more of the driver state sensors is obstructed while monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated.

8. A steering system comprising:

one or more sensors operable to produce a plurality of steering sensor data;

a plurality of driver state sensors operable to produce a plurality of driver state data;

a steering actuator motor; and

a control module comprising a processor operable to command the steering actuator motor in an autonomous driving mode and in a manual driving mode, the control module further operable to fuse the steering sensor data with the driver state data as a plurality of fused state data aligned in time, determine whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data, determine an attentive state of a driver based on time-averaging of the fused state data, continue monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated to determine an attentive state of a driver based on time-averaging of the fused state data, and revert from the manual driving mode to the autonomous driving mode based on determining that the attentive state of the driver no longer meets one or more transition validity criteria.

9. The steering system of claim 8, wherein the steering sensor data comprise sensed or derived data from one or more of a handwheel torque sensor and a handwheel position/velocity sensor.

10. The steering system of claim 8, wherein the driver state data comprise sensed or derived data from one or more of handwheel touch sensor and a camera.

11. The steering system of claim 10, wherein the driver state data sensed or derived from the handwheel touch sensor comprises one or more of:

an area of contact with a handwheel, a width of contact with the handwheel, a force of contact with the handwheel, and a position of contact with the handwheel, and wherein the driver state data sensed or derived from the camera comprises one or more of;

a driver body posture, a driver head pose, a driver eye gaze, and a driver hand position.

12. The steering system of claim 8, wherein the control module comprises a recent data store of the fused state data and a history data store of the fused state data, wherein the history data store is operable to collect older values of the fused state data than collected in the recent data store.

13. The steering system of claim 12, wherein the control module is operable to average multiple samples of the fused state data from the history data store as time-averaged fused state data and determine whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data from the recent data store in combination with the time-averaged fused state data.

14. The steering system of claim 8, wherein the processor is further configured to revert from the manual driving mode to the autonomous driving mode based on determining that an input from one or more of the driver state sensors is obstructed while monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated.

15. A method for sensor fusion for autonomous driving transition control, the method comprising:

acquiring, by a processor of a control module, a plurality of steering sensor data from one or more sensors of a steering system;

acquiring, by the processor of the control module, a plurality of driver state data from a plurality of driver state sensors;

fusing, by the processor of the control module, the steering sensor data with the driver state data as a plurality of fused state data aligned in time;

determining, by the processor of the control module, whether to transition from an autonomous driving mode to a manual driving mode based on the fused state data;

commanding, by the processor of the control module, a steering motor actuator of the steering system autonomously in the autonomous driving mode;

commanding, by the processor of the control module, the steering motor actuator responsive to a driver input in the manual driving mode;

determining, by the processor of the control module, an attentive state of a driver based on time-averaging of the fused state data;

continuing monitoring, by the processor of the control module, the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated to determine an attentive state of a driver based on time-averaging of the fused state data; and

reverting, by the processor of the control module, from the manual driving mode to the autonomous driving mode based on determining that the attentive state of the driver no longer meets one or more transition validity criteria.

16. The method of claim 15, wherein the steering sensor data comprise sensed or derived data from one or more of a handwheel torque sensor and a handwheel position/velocity sensor.

17. The method of claim 15, wherein the driver state data comprise sensed or derived data from one or more of handwheel touch sensor and a camera, the driver state data sensed or derived from the handwheel touch sensor comprises one or more of:

an area of contact with a handwheel, a width of contact with the handwheel, a force of contact with the handwheel, and a position of contact with the handwheel, and wherein the driver state data sensed or derived from the camera comprises one or more of;

a driver body posture, a driver head pose, a driver eye gaze, and a driver hand position.

18. The method of claim 15, further comprising:

storing the fused state data in a recent data store; and

storing the fused state data in the history data store, wherein the history data store collects older values of the fused state data than collected in the recent data store.

19. The method of claim 18, further comprising:

averaging multiple samples of the fused state data from the history data store as time-averaged fused state data; and

determining whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data from the recent data store in combination with the time-averaged fused state data.

20. The method of claim 15, further comprising:

reverting from the manual driving mode to the autonomous driving mode based on determining that an input from one or more of the driver state sensors is obstructed while monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated.