Event classification and object tracking

• US 10,520,904 B2
• Filed: 01/30/2017
• Issued: 12/31/2019
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: Siemens Electronic Design Automation GmbH (Siemens AG)
• Original Assignee: Mentor Graphics Corporation (Siemens AG)
• Inventors: Mercep, Ljubo, Pollach, Matthias
• Primary Examiner(s): Ramesh, Krishnan
• Application Number:
  US15/419,931
• Publication Number:
  US 20180067491A1
• Time in Patent Office: 1,065 Days
• Field of Search: 701 23
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/4041 Position
• CPC Class Codes: B60W 2556/20 Data confidence level
• CPC Class Codes: B60W 2556/35 Data fusion
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01C 21/3407 specially adapted for specific applications
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01C 21/3667 Display of a road map G01C21/3614 takes precedence; guidance using 3D or perspective road maps G01C21/3635
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 15/931 of land vehicles
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G01S 2013/93271 : in the front of the vehicles
• CPC Class Codes: G01S 2013/93272 : in the back of the vehicles
• CPC Class Codes: G01S 2013/93274 : on the side of the vehicles
• CPC Class Codes: G01S 7/4808 : Evaluating distance, position or velocity data
• CPC Class Codes: G05B 17/02 : electric
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/024 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0255 : using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/0259 : using magnetic or electromagnetic means
• CPC Class Codes: G05D 1/0268 : using internal positioning means
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 16/29 : Geographical information databases
• CPC Class Codes: G06F 16/5854 : using shape and object relationship
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/161 : Decentralised systems, e.g. inter-vehicle communication
• CPC Class Codes: G08G 1/163 : involving continuous checking
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles

Abstract

This application discloses a computing system to implement object tracking in an assisted or automated driving system of a vehicle. The computing system can assign a pre-classification to a detection event in an environmental model, update the environmental model with new sensor measurements and corresponding detection events over time, and track the detection event in the updated environmental model. The computing system can track the detection event by predicting a future state of the detection event with a state change model selected based on the assigned pre-classification, comparing the predicted future state to an actual future state of the detection event in an update to the environmental model, and determining the detection event corresponds to an object proximate to the vehicle based on the comparison. A control system for the vehicle can control operation of the vehicle based, at least in part, on the tracked detection event.

Claims

1. A method comprising:

assigning, by a computing system, a pre-classification to a detection event in an environmental model having an environmental coordinate field that spans multiple non-overlapping measurement coordinate fields of sensors mounted in a vehicle, wherein the detection event corresponds to spatially-aligned and temporally-aligned raw sensor measurements indicative of an object proximate to the vehicle, and wherein the environmental model is stored by the computing system and includes both the spatially-aligned and temporally-aligned raw sensor measurements and an indication of the detection event;

updating, by the computing system, the environmental model to include new raw sensor measurements from the sensors mounted in the vehicle and corresponding detection events over time; and

tracking, by the computing system, the detection event in the updated environmental model based on the assigned pre-classification of the detection event, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the tracked detection event.

2. The method of claim 1, further comprising selecting, by the computing system, a state change model based, at least in part, on the assigned pre-classification of the detection event, wherein the tracking of the detection event is based, at least in part, on the state change model.

3. The method of claim 2, wherein the tracking of the detection event further comprises:

predicting a future state of the detection event based, at least in part, on the state change model;

comparing the predicted future state to an actual future state of the detection event in an update to the environmental model; and

determining the detection event corresponds to the object proximate to the vehicle based on the comparison.

4. The method of claim 3, wherein predicting the future state of the detection event is based on at least one of a location of the detection event in the environmental model or a prior movement of the detection event in the environmental model.

5. The method of claim 1, wherein tracking the detection event in the updated environmental model further comprises tracking the detection event through the environmental coordinate field of the environmental model, which spans the multiple non-overlapping measurement coordinate fields of the sensors mounted in the vehicle.

6. The method of claim 1, wherein the pre-classification corresponds to a type of the object capable of being located proximate to the vehicle.

7. The method of claim 1, further comprising annotating, by the computing system, the environmental model to include the assigned pre-classification or the tracked detection event.

8. An apparatus comprising at least one memory device storing instructions configured to cause one or more processing devices to perform operations comprising:

assigning a pre-classification to a detection event in an environmental model having an environmental coordinate field that spans multiple non-overlapping measurement coordinate fields of sensors mounted in a vehicle, wherein the detection event corresponds to spatially-aligned and temporally-aligned raw sensor measurements indicative of an object proximate to the vehicle, and wherein the environmental model is stored by the memory device and includes both the spatially-aligned and temporally-aligned raw sensor measurements and an indication of the detection event;

updating the environmental model to include new raw sensor measurements from the sensors mounted in the vehicle and corresponding detection events over time; and

tracking the detection event in the updated environmental model based on the assigned pre-classification of the detection event, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the tracked detection event.

9. The apparatus of claim 8, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising selecting a state change model based, at least in part, on the assigned pre-classification of the detection event, wherein the tracking of the detection event is based, at least in part, on the state change model.

10. The apparatus of claim 9, wherein the tracking of the detection event further comprises:

predicting a future state of the detection event based, at least in part, on the state change model;

comparing the predicted future state to an actual future state of the detection event in an update to the environmental model; and

determining the detection event corresponds to the object proximate to the vehicle based on the comparison.

11. The apparatus of claim 10, wherein predicting the future state of the detection event is based on at least one of a location of the detection event in the environmental model or a prior movement of the detection event in the environmental model.

12. The apparatus of claim 8, wherein tracking the detection event in the updated environmental model further comprises tracking the detection event through the environmental coordinate field of the environmental model, which spans the multiple non-overlapping measurement coordinate fields of the sensors mounted in the vehicle.

13. The apparatus of claim 8, wherein the pre-classification corresponds to a type of the object capable of being located proximate to the vehicle.

14. The apparatus of claim 8, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising annotating the environmental model to include the assigned pre-classification or the tracked detection event.

15. A system comprising:

a memory device configured to store machine-readable instructions; and

a computing system including one or more processing devices, in response to executing the machine-readable instructions, configured to;

assign a pre-classification to a detection event in an environmental model having an environmental coordinate field that spans multiple non-overlapping measurement coordinate fields of sensors mounted in a vehicle, wherein the detection event corresponds to spatially-aligned and temporally-aligned raw sensor measurements indicative of an object proximate to the vehicle, and wherein the environmental model is stored by the memory device and includes both the spatially-aligned and temporally-aligned raw sensor measurements and an indication of the detection event;

update the environmental model to include new raw sensor measurements from the sensors mounted in the vehicle and corresponding detection events over time; and

track the detection event in the updated environmental model based on the assigned pre-classification of the detection event, wherein a control system for the vehicle is configured to control operation of the vehicle based, at least in part, on the tracked detection event.

16. The system of claim 15, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

select a state change model based, at least in part, on the assigned pre-classification of the detection event; and

track the detection event based, at least in part, on the state change model.

17. The system of claim 16, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to track of the detection event by:

predicting a future state of the detection event based, at least in part, on the state change model;

comparing the predicted future state to an actual future state of the detection event in an update to the environmental model; and

determining the detection event corresponds to the object proximate to the vehicle based on the comparison.

18. The system of claim 17, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to predict the future state of the detection event based on at least one of a location of the detection event in the environmental model or a prior movement of the detection event in the environmental model.

19. The system of claim 15, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to track the detection event in the updated environmental model by tracking the detection event through the environmental coordinate field of the environmental model, which spans the multiple non-overlapping measurement coordinate fields of the sensors mounted in the vehicle.

20. The system of claim 15, wherein the pre-classification corresponds to a type of the object capable of being located proximate to the vehicle.