Scalable sensor fusion and autonomous x-by-wire control

• US 10,112,606 B2
• Filed: 01/22/2016
• Issued: 10/30/2018
• Est. Priority Date: 01/22/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jain, Rakesh, Madl, Gabor, Routray, Ramani R., Song, Yang
• Primary Examiner(s): Smith, Jelani A
• Assistant Examiner(s): Williams, Kelly D
• Application Number:
  US15/004,718
• Publication Number:
  US 20170210376A1
• Time in Patent Office: 1,012 Days
• Field of Search: None
• CPC Class Codes: B60T 2270/402 Back-up
• CPC Class Codes: B60T 2270/82 Brake-by-Wire, EHB
• CPC Class Codes: B60T 7/18 operated by wayside apparatus
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60T 8/171 Detecting parameters used in the regulation; Measuring values used in the regulation
• CPC Class Codes: B60T 8/885 using electrical circuitry
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B62D 15/025 Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation B62D1/28 takes precedence
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

An autonomous system for x-by-wire control includes processing nodes distributed and connected to one another. Sensors are connected to the processing nodes. Actuators are configured to directly control the autonomous system driven by and connected to the processing nodes for x-by-wire control. The processing nodes are configured to: partition and map processing tasks between the processing nodes, and merge and reduce results of individual processing tasks into an actionable table that specifies objectives for the autonomous system.

Claims

1. An autonomous system comprising:

a plurality of processing nodes distributed and coupled to one another;

a plurality of sensors coupled to the processing nodes; and

a plurality of actuators including at least one power control actuator, at least one brake actuator, and at least one steering actuator, the plurality of actuators configured to directly control the autonomous system driven by and coupled to the plurality of processing nodes for throttle-by-wire, brake-by-wire and steer-by-wire control;

wherein each processing node of the plurality of processing nodes is configured to;

partition and map processing tasks between the plurality of processing nodes;

merge and reduce results of individual processing tasks into an actionable table that specifies objectives comprising control values resulting from sensor fusion processing, the objectives are used to directly control the plurality of actuators for the autonomous system; and

transfer a workload from one or more processing nodes determined to have a fault to one or more active processing nodes, wherein a reconfiguration of processing nodes is achieved within a bounded time.

2. The autonomous system of claim 1, wherein each of the processing nodes comprise:

at least one memory device configured to store sensor data received from one or more of the plurality of sensors;

at least one processor configured to perform processing using the sensor data received from sensors, and generate the control values to directly drive any of the plurality of actuators; and

a networking interface configured to provide communication with the plurality of processing nodes, the plurality of sensors, and the plurality of actuators.

3. The autonomous system of claim 2, wherein each processing node of the plurality of processing nodes is further configured to:

read signals from a set of sensors of the plurality of sensors;

interpret the signals as sets of sensor values over time; and

perform the sensor fusion processing over the set of sensors based on the sets of sensor values,

wherein each processing node of the plurality of processing nodes uses map-reduce operations for the sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.

4. The autonomous system of claim 3, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, and estimation determination based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the plurality of sensors.

5. The autonomous system of claim 4, wherein each processing node of the plurality of processing nodes is configured to:

determine control values based on a result of the sensor fusion processing;

communicate the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors;

measure a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; and

provide processing to minimize the difference,

wherein throttle-by-wire, brake-by-wire and steer-by-wire control is provided for a vehicle.

6. The autonomous system of claim 4, wherein:

the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read;

the plurality of sensors comprises one or more;

camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and

the plurality of actuators comprises one or more;

an electronic power control, a steering control unit (SCU), and braking units (BUs).

7. The autonomous system of claim 1, wherein each processing node of the plurality of processing nodes is further configured to:

determine sensor failure and compensate for the sensor failure by decreasing failed sensor importance weighting.

8. A method for sensor fusion and autonomous control comprising:

receiving sensor signals from a plurality of sensors;

interpreting sensor signals as sets of sensor values over time;

performing distributed sensor fusion on the sets of sensor values by assigning a weight to each sensor of the plurality of sensors;

partitioning and mapping processing tasks for the sets of sensor values to a set of distributed processing nodes;

merging and reducing results of individual processing tasks performed by the distributed processing nodes into an actionable table, where such actionable table is used to specify objectives comprising control values resulting from sensor fusion processing, the objectives are used for an autonomous system;

directly controlling a plurality of actuators using the actionable table for throttle-by-wire, brake-by-wire and steer-by-wire control, wherein the plurality of actuators includes at least one power control actuator, at least one brake actuator and at least one steering actuator; and

transferring, by each of the distributed processing nodes, a workload from one or more processing nodes determined to have a fault to one or more active processing nodes wherein a reconfiguration of processing nodes is achieved within a bounded time.

9. The method of claim 8, further comprising:

storing, by each of the distributed processing nodes, sensor data received from at least one sensor of the plurality of sensors;

processing, by each of the distributed processing nodes, the sets of sensor values;

generating, by each of the distributed processing nodes, the control values to directly drive any of the plurality of actuators; and

communicating, by each of the distributed processing nodes, between the plurality of processing nodes, with the plurality of sensors, and with the plurality of actuators.

10. The method of claim 9, further comprising:

reading, by each of the distributed processing nodes, signals from a set of sensors of the plurality of sensors;

interpreting, by each of the distributed processing nodes, the signals as sets of sensor values over time; and

performing, by each of the distributed processing nodes, sensor fusion processing over the set of sensors based on the sets of sensor values,

wherein each processing node of the distributed processing nodes uses map-reduce operations for the distributed sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.

11. The method of claim 10, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, and determining estimations based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the plurality of sensors.

12. The method of claim 11, further comprising:

determining, by each of the distributed processing nodes, control values based on a result of the sensor fusion processing;

communicating, by each of the distributed processing nodes, the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors;

measuring, by each of the distributed processing nodes, a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values;

processing, by each of the distributed processing nodes, to minimize the difference; and

providing throttle-by-wire, brake-by-wire and steer-by-wire control for a vehicle.

13. The method of claim 11, wherein:

the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read;

the plurality of sensors comprises one or more;

camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and

the plurality of actuators comprises one or more;

an electronic power control, a steering control unit (SCU), and braking units (BUs).

14. The method of claim 8, further comprising:

determining, by each of the distributed processing nodes, sensor failure; and

compensating, by each of the distributed processing nodes, for the sensor failure by decreasing failed sensor importance weighting.

15. An autonomous computing environment for x-by-wire control of a vehicle comprising:

an actionable table that specifies objectives comprising control values resulting from sensor fusion processing, the objectives are used to directly control a plurality of actuators for the computing environment, the plurality of actuators including at least one power control actuator, at least one brake actuator and at least one steering actuator; and

a plurality of processing nodes distributed and coupled to one another, each of the processing nodes are configured to receive sensor data from a plurality of sensors and to control the plurality of actuators to directly control vehicle devices for throttle-by-wire, brake-by-wire and steer-by-wire control;

wherein each processing node of the plurality of processing nodes is configured to;

partition and map processing tasks for distribution between the plurality of processing nodes;

merge and reduce results of individual processing tasks into the actionable table; and

transfer a workload from one or more processing nodes determined to have a fault to one or more active processing nodes, wherein a reconfiguration of processing nodes is achieved within a bounded time.

16. The autonomous computing environment of claim 15, wherein each of the processing nodes comprise:

at least one memory device configured to store sensor data received from one or more of the plurality of sensors;

at least one processor configured to perform processing using the sensor data received from sensors, and generate control values to directly drive any of the plurality of actuators; and

a networking interface configured to provide communication with the plurality of processing nodes, the plurality of sensors, and the plurality of actuators.

17. The autonomous computing environment system of claim 16, wherein each processing node of the plurality of processing nodes is further configured to:

read signals from a set of sensors of the plurality of sensors;

interpret the signals as sets of sensor values over time; and

perform sensor fusion processing over the set of sensors based on the sets of sensor values, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, estimation determination based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the set of sensors, and each processing node of the plurality of processing nodes uses map-reduce operations for the sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.

18. The autonomous computing environment of claim 17, wherein each processing node of the plurality of processing nodes is configured to:

determine control values based on a result of the sensor fusion processing;

communicate the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors;

measure a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; and

provide processing to minimize the difference.

19. The autonomous computing environment of claim 17, wherein:

the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read;

the plurality of sensors comprises one or more;

camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and

the plurality of actuators comprises one or more;

an electronic power control, a steering control unit (SCU), and braking units (BUs).

20. The autonomous computing environment of claim 15, wherein the plurality of processing nodes is further configured to:

determine sensor failure and compensate for the sensor failure by decreasing failed sensor importance weighting.