Low-level sensor fusion

• US 10,317,901 B2
• Filed: 10/06/2016
• Issued: 06/11/2019
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: Siemens Electronic Design Automation GmbH (Siemens AG)
• Original Assignee: Mentor Graphics Development (Deutschland) GmbH (Siemens AG)
• Inventors: Oder, Rainer, Geiger, Andreas Erich, Mercep, Ljubo, Pollach, Matthias
• Primary Examiner(s): Antonucci, Anne M
• Application Number:
  US15/287,537
• Publication Number:
  US 20180067489A1
• Time in Patent Office: 978 Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

This application discloses a computing system to implement low-level sensor fusion in an assisted or automated driving system of a vehicle. The low-level sensor fusion can include receiving raw measurement data from sensors in the vehicle and temporally aligning the raw measurement data based on a time of capture. The low-level sensor fusion can include spatially aligning measurement coordinate fields of the sensors into an environmental coordinate field based, at least in part, on where the sensors are mounted in the vehicle, and then populating the environmental coordinate field with raw measurement data captured by the sensors based on the spatial alignment of the measurement coordinate fields to the environmental coordinate field. The low-level sensor fusion can detect at least one detection event or object based, at least in part, on the raw measurement data from multiple sensors as populated in the environmental coordinate field.

Claims

1. A method comprising:

spatially aligning, by a computer system, measurement coordinate fields of sensors to an environmental coordinate field associated with a vehicle based, at least in part, on where the sensors are mounted in the vehicle;

populating, by the computer system, an environmental model with raw measurement data captured by the sensors, wherein the environmental model stores the raw measurement data in the environmental coordinate field based on the spatial alignment of the measurement coordinate fields to the environmental coordinate field; and

processing, by the computer system, the raw measurement data from a plurality of the sensors, which was populated in the environmental model, to identify a detection event corresponding to a spatially aligned grouping of the raw measurement data from the plurality of the sensors, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the detection event.

2. The method of claim 1, further comprising temporally aligning, by the computer system, the raw measurement data based on when different portions of the raw measurement data was captured by the sensors, wherein populating the environmental model with the raw measurement data is based, at least in part, on the temporal alignment of the raw measurement data.

3. The method of claim 1, further comprising:

detecting, by the computer system, a presence of an object from the detection event; and

populating, by the computer system, the environmental model having the environmental coordinate field with data corresponding to the object.

4. The method of claim 3, wherein detecting the presence of the object from the sensor event further comprises:

tracking the detection event over time as the environmental model is updated with new raw measurement data; and

detecting the object from to the detection event based on the tracking of the detection event over time.

5. The method of claim 4, wherein tracking the detection event over time further comprises:

classifying the detection event as an object type based on the raw measurement data; and

monitoring the detection event over time according to a state change prediction model selected based on the classification of the detection event.

6. The method of claim 1, wherein processing the raw measurement data further comprises:

detecting a presence of the raw measurement data in a portion of the environmental model; and

identifying the detection event when the detected raw measurement data in the portion of the environmental model exceeds a predetermined noise threshold, corresponds to a change of intensity in the raw measurement data over time, corresponds to a change of reflection in the raw measurement data over time, or corresponds to a change of pixel value in the raw measurement data over time.

7. The method of claim 1, further comprising:

estimating, by the computer system, motion of the sensors capturing the raw measurement data; and

modifying, by the computer system, the spatial alignment between the measurement coordinate fields of the sensors and the environmental coordinate field for the environmental model associated with the vehicle.

8. An apparatus comprising at least one memory device storing instructions configured to cause one or more processing devices to perform operations comprising:

spatially aligning measurement coordinate fields of sensors to an environmental coordinate field associated with a vehicle based, at least in part, on where the sensors are mounted in the vehicle;

populating an environmental model with raw measurement data captured by the sensors, wherein the environmental model stores the raw measurement data in the environmental coordinate field based on the spatial alignment of the measurement coordinate fields to the environmental coordinate field; and

processing the raw measurement data from a plurality of the sensors, which was populated in the environmental model, to identify a detection event corresponding to a spatially aligned grouping of the raw measurement data from the plurality of the sensors, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the detection event.

9. The apparatus of claim 8, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising temporally aligning the raw measurement data based on when different portions of the raw measurement data was captured by the sensors, wherein populating the environmental model with the raw measurement data is based, at least in part, on the temporal alignment of the raw measurement data.

10. The apparatus of claim 8, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:

detecting a presence of an object from the detection event; and

populating the environmental model with data corresponding to the object.

11. The apparatus of claim 10, wherein detecting the presence of the object from the sensor event further comprises:

tracking the detection event over time as the environmental model is updated with new raw measurement data; and

detecting the object from the detection event based on the tracking of the detection event over time.

12. The apparatus of claim 11, wherein tracking the detection event over time further comprises:

classifying the detection event as an object type based on the raw measurement data; and

monitoring the detection event over time according to a state change prediction model selected based on the classification of the detection event.

13. The apparatus of claim 11, wherein processing the raw measurement data further comprises:

detecting a presence of the raw measurement data in a portion of the environmental model; and

identifying the detection event when the detected raw measurement data in the portion of the environmental model exceeds a predetermined noise threshold, corresponds to a change of intensity in the raw measurement data over time, corresponds to a change of reflection in the raw measurement data over time, or corresponds to a change of pixel value in the raw measurement data over time.

14. The apparatus of claim 8, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:

estimating motion of the sensors capturing the raw measurement data; and

modifying the spatial alignment between the measurement coordinate fields of the sensors and the environmental coordinate field for the environmental model associated with the vehicle.

15. A system comprising:

a memory device configured to store machine-readable instructions; and

a computer system including one or more processing devices, in response to executing the machine-readable instructions, configured to;

spatially align measurement coordinate fields of sensors to an environmental coordinate field associated with a vehicle based, at least in part, on where the sensors are mounted in the vehicle;

populate an environmental model with raw measurement data captured by the sensors, wherein the environmental model stores the raw measurement data in the environmental coordinate field based on the spatial alignment of the measurement coordinate fields to the environmental coordinate field; and

process the raw measurement data from a plurality of the sensors, which was populated in the environmental model, to identify a detection event corresponding to a spatially aligned grouping of the raw measurement data from the plurality of the sensors, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the detection event.

16. The system of claim 15, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

temporally align the raw measurement data based on when different portions of the raw measurement data was captured by the sensors; and

populate the environmental model with the raw measurement data based, at least in part, on the temporal alignment of the raw measurement data.

17. The system of claim 15, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

detect a presence of an object from the detection event; and

populate the environmental model with data corresponding to the object.

18. The system of claim 17, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

track the detection event over time as the environmental model is updated with new raw measurement data; and

detect the object from the detection event based on the tracking of the detection event over time.

19. The system of claim 18, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

classify the detection event as an object type based on the raw measurement data; and

monitor the detection event over time according to a state change prediction model selected based on the classification of the detection event.

20. The system of claim 18, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

detect a presence of the raw measurement data in a portion of the environmental model; and

identify the detection event when the detected raw measurement data in the portion of the environmental model exceeds a predetermined noise threshold, corresponds to a change of intensity in the raw measurement data over time, corresponds to a change of reflection in the raw measurement data over time, or corresponds to a change of pixel value in the raw measurement data over time.