Sensor event detection and fusion

• US 10,802,450 B2
• Filed: 01/30/2017
• Issued: 10/13/2020
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: Siemens Electronic Design Automation GmbH
• Original Assignee: Mentor Graphics Corporation (Siemens AG)
• Inventors: Mercep, Ljubo, Pollach, Matthias
• Primary Examiner(s): Shudy, Angelina
• Application Number:
  US15/419,914
• Publication Number:
  US 20180067463A1
• Time in Patent Office: 1,352 Days
• Field of Search: 701 23
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/4041 Position
• CPC Class Codes: B60W 2556/20 Data confidence level
• CPC Class Codes: B60W 2556/35 Data fusion
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01C 21/3407 specially adapted for specific applications
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01C 21/3667 Display of a road map G01C21/3614 takes precedence; guidance using 3D or perspective road maps G01C21/3635
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 15/931 of land vehicles
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G01S 2013/93271 : in the front of the vehicles
• CPC Class Codes: G01S 2013/93272 : in the back of the vehicles
• CPC Class Codes: G01S 2013/93274 : on the side of the vehicles
• CPC Class Codes: G01S 7/4808 : Evaluating distance, position or velocity data
• CPC Class Codes: G05B 17/02 : electric
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/024 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0255 : using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/0259 : using magnetic or electromagnetic means
• CPC Class Codes: G05D 1/0268 : using internal positioning means
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 16/29 : Geographical information databases
• CPC Class Codes: G06F 16/5854 : using shape and object relationship
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/161 : Decentralised systems, e.g. inter-vehicle communication
• CPC Class Codes: G08G 1/163 : involving continuous checking
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles

Abstract

This application discloses a computing system to implement sensor event detection and fusion system in an assisted or automated driving system of a vehicle. The computing system can monitor an environmental model to identify spatial locations in the environmental model populated with temporally-aligned measurement data. The computing system can analyze, on a per-sensor basis, the temporally-aligned measurement data at the spatial locations in the environmental model to detect one or more sensor measurement events. The computing system can utilize the sensor measurement events to identify at least one detection event indicative of an object proximate to the vehicle. The computing system can combine the detection event with at least one of another detection event, a sensor measurement event, or other measurement data to generate a fused detection event. A control system for the vehicle can control operation of the vehicle based, at least in part, on the detection event.

Claims

1. A method comprising:

monitoring, by a computing system, an environmental model stored in a memory system to identify when spatial locations in the environmental model become populated with temporally-aligned raw measurement data captured by multiple sensors mounted in a vehicle, wherein the environmental model stored in the memory system includes the raw measurement data captured by the multiple sensors, which has been both spatially-aligned and temporally-aligned in the environmental model;

analyzing, by the computing system on a per-sensor basis, the temporally-aligned raw measurement data at the spatial locations in the environmental model to detect one or more sensor measurement events, wherein each of the one or more sensor measurement events identify a location in the environmental model having been populated with the raw measurement data captured by one of the sensors at a level above a threshold for noise in an environment around the vehicle; and

utilizing, by the computing system, the raw measurement data at the locations identified by one or more sensor measurement events corresponding to the raw measurement data to identify at least one detection event indicative of an object proximate to the vehicle, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the at least one detection event.

2. The method of claim 1, further comprising populating, by the computing system, at least one of the spatial locations in the environmental model with at least one indication of the detection event or the one or more sensor measurement events.

3. The method of claim 1, wherein the identification of the detection event further comprises generating the detection event when the temporally-aligned raw measurement data corresponding to one of the sensor measurement events exceeds a detection threshold.

4. The method of claim 1, further comprises combining, by the computing system, the detection event with at least one of another detection event corresponding to a different sensor, one or more of the sensor measurement events corresponding to a different sensor, or the temporally-aligned raw measurement data to generate a fused detection event indicative of the object proximate to the vehicle.

5. The method of claim 1, further comprising generating, by the computing system, a confidence level associated with the detection event, wherein the confidence level corresponds to a probability that the detection event corresponds to an object.

6. The method of claim 5, further comprising annotating, by the computing system, the detection event populated in the environmental model with the confidence level.

7. An apparatus comprising at least one memory device storing instructions configured to cause one or more processing devices to perform operations comprising:

monitoring an environmental model stored in a memory system to identify when spatial locations in the environmental model become populated with temporally-aligned raw measurement data captured by multiple sensors mounted in a vehicle, wherein the environmental model stored in the memory system includes the raw measurement data captured by the multiple sensors, which has been both spatially-aligned and temporally-aligned in the environmental model;

analyzing, on a per-sensor basis, the temporally-aligned raw measurement data at the spatial locations in the environmental model to detect one or more sensor measurement events, wherein each of the one or more sensor measurement events identify a location in the environmental model having been populated with the raw measurement data captured by one of the sensors at a level above a threshold for noise in an environment around the vehicle; and

utilizing the raw measurement data at the locations identified by one or more sensor measurement events corresponding to the raw measurement data to identify at least one detection event indicative of an object proximate to the vehicle, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the at least one detection event.

8. The apparatus of claim 7, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising populating at least one of the spatial locations in the environmental model with at least one indication of the detection event or the one or more sensor measurement events.

9. The apparatus of claim 7, wherein the identification of the detection event further comprises generating the detection event when the temporally-aligned raw measurement data corresponding to one of the sensor measurement events exceeds a detection threshold.

10. The apparatus of claim 7, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising combining the detection event with at least one of another detection event corresponding to a different sensor, one or more of the sensor measurement events corresponding to a different sensor, or the temporally-aligned raw measurement data to generate a fused detection event indicative of the object proximate to the vehicle.

11. The apparatus of claim 7, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising generating a confidence level associated with the detection event, wherein the confidence level corresponds to a probability that the detection event corresponds to an object.

12. The apparatus of claim 11, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising annotating the detection event populated in the environmental model with the confidence level.

13. A system comprising:

a memory device configured to store machine-readable instructions; and

a computing system including one or more processing devices, in response to executing the machine-readable instructions, configured to;

monitor an environmental model stored in a memory system to identify when spatial locations in the environmental model become populated with temporally-aligned raw measurement data captured by multiple sensors mounted in a vehicle, wherein the environmental model stored in the memory system includes the raw measurement data captured by the multiple sensors, which has been both spatially-aligned and temporally-aligned in the environmental model;

analyze, on a per-sensor basis, the temporally-aligned raw measurement data at the spatial locations in the environmental model to detect one or more sensor measurement events, wherein each of the one or more sensor measurement events identify a location in the environmental model having been populated with the raw measurement data captured by one of the sensors at a level above a threshold for noise in an environment around the vehicle; and

utilize the raw measurement data at the locations identified by one or more sensor measurement events corresponding to the raw measurement data to identify at least one detection event indicative of an object proximate to the vehicle, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the at least one detection event.

14. The system of claim 13, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to populate at least one of the spatial locations in the environmental model with at least one indication of the detection event or the one or more sensor measurement events.

15. The system of claim 13, wherein the identification of the detection event further comprises generating the detection event when the temporally-aligned raw measurement data corresponding to one of the sensor measurement events exceeds a detection threshold.

16. The system of claim 13, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to combine the detection event with at least one of another detection event corresponding to a different sensor, one or more of the sensor measurement events corresponding to a different sensor, or the temporally-aligned raw measurement data to generate a fused detection event indicative of the object proximate to the vehicle.

17. The system of claim 13, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

generate a confidence level associated with the detection event, wherein the confidence level corresponds to a probability that the detection event corresponds to an object; and

annotate the detection event populated in the environmental model with the confidence level.