Sensor modification based on an annotated environmental model

• US 10,678,240 B2
• Filed: 09/08/2016
• Issued: 06/09/2020
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: Siemens Electronic Design Automation GmbH (Siemens AG)
• Original Assignee: Mentor Graphics Corporation (Siemens AG)
• Inventors: Pollach, Matthias Arnold, Mercep, Ljubo
• Primary Examiner(s): Goldman, Richard A
• Application Number:
  US15/260,149
• Publication Number:
  US 20180067488A1
• Time in Patent Office: 1,370 Days
• Field of Search: 701 27, 701 41, 701 45, 701 49, 701 70, 701 96, 701300, 701301, 701302, 701408, 340435, 340465, 340931
• CPC Class Codes: B60W 2510/18 Braking system
• CPC Class Codes: B60W 2510/20 Steering systems
• CPC Class Codes: B60W 40/02 related to ambient conditions
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

Systems, methods, and computer-readable storage mediums which use annotated environmental models for sensor modification are disclosed. A computing system receives an environmental model for a vehicle. The environmental model can include data from a plurality of modalities. Object annotations are received from sensors which are used to modify the environmental model, creating an annotated environmental model. A classification of a current situation is generated using the annotated environmental model, and a sensor is modified using the classification.

Claims

1. A method comprising:

receiving, at a processor of a sensor fusion system of a vehicle, an environmental model comprising raw measurement data from a plurality of sensors of the vehicle, the plurality of sensors having distinct modalities, wherein the raw measurement data is temporally aligned in the environmental model and spatially aligned in the environmental model based on where the sensors are mounted in the vehicle;

receiving, at the processor, object annotations indicating a detection of at least one object based on the raw measurement data in the environmental model, the object annotations comprising a confidence level associated with the detection of the at least one object;

modifying, via the processor, the environmental model to include the object annotations, resulting in an annotated environmental model, wherein the object annotations are temporally aligned and spatially aligned in the annotated environmental model;

generating, via the processor, a classification result of a current situation of the vehicle based, at least in part, on the raw measurement data and the object annotations in the annotated environmental model; and

transmitting, via the processor, the classification result to a processor associated with driving functionality of the vehicle, wherein the processor associated with driving functionality is configured to control operation the vehicle based, at least in part, on the classification result.

2. The method of claim 1, further comprising:

generating, via the processor and based on the classification result, a feedback signal; and

transmitting, from the processor to at least one sensor in the plurality of sensors, the feedback signal which causes at least one of repositioning the at least one sensor, or expanding a field of view of the at least one sensor, or changing a rate of exposure of the at least one sensor.

3. The method of claim 1, wherein the classification result is further determined based on one or more of a region of interest in the environmental model, a predicted environmental model, and localization information.

4. The method of claim 1, wherein the distinct modalities comprise one or more of raw measurement data from one or more sensors on the vehicle, fused sensor data, high level data, and data from one or more external sources.

5. The method of claim 1, further comprising providing the classification result to a processor configured to adjust a region of interest in the environmental model based, at least in part, on the classification result.

6. The method of claim 1, further comprising:

determining that the classification result identifies an uncertain vehicle situation; and

providing classification data associated with the classification result to an external computing system located away from the vehicle, wherein the external computing system is configured to use the classification data to generate an update to the sensor fusion system based on the uncertain vehicle situation.

7. The method of claim 6, further comprising:

receiving, from the external computing system, the update; and

modifying the processor based on the update from the external computing system.

8. The method of claim 1, further comprising:

storing the annotated environmental model and classification result to a memory associated with the vehicle; and

upon detecting an incident associated with the vehicle, evaluating the incident using the annotated environmental model and classification result stored in the memory.

9. An apparatus comprising at least one memory device storing instructions configured to cause one or more processing devices of a sensor fusion system to perform operations comprising:

receiving an environmental model comprising raw measurement data from a plurality of sensors of the vehicle, the plurality of sensors having distinct modalities, wherein the raw measurement data is temporally aligned in the environmental model and spatially aligned in the environmental model based on where the sensors are mounted in the vehicle;

receiving object annotations indicating a detection of at least one object based on the raw measurement data in the environmental model, the object annotations comprising a confidence level associated with the detection of the at least one object;

modifying the environmental model to include the object annotations, resulting in an annotated environmental model, wherein the object annotations are temporally aligned and spatially aligned in the annotated environmental model;

generating a classification result of a current situation of the vehicle based, at least in part, on the raw measurement data and the object annotations in the annotated environmental model; and

transmitting the classification result to a processor associated with driving functionality of the vehicle, wherein the processor associated with driving functionality is configured to control operation the vehicle based, at least in part, on the classification result.

10. The apparatus of claim 9, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:

generating, based on the classification result, a feedback signal; and

transmitting, to the at least one sensor in the plurality of sensors, the feedback signal, causing at least one of repositioning the at least one sensor, or expanding a field of view of the at least one sensor, or changing a rate of exposure of the at least one sensor.

11. The apparatus of claim 9, wherein the classification result is further determined based on one or more of a region of interest in the environmental model, a predicted environmental model, and localization information.

12. The apparatus of claim 9, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising providing the classification result to a processor configured to adjust a region of interest in the environmental model based, at least in part, on the classification result.

13. The apparatus of claim 9, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:

determining that the classification result identifies an uncertain vehicle situation; and

providing classification data associated with the classification result to an external computing system located away from the vehicle, wherein the external computing system is configured to use the classification data to generate an update to the sensor fusion system based on the uncertain vehicle situation.

14. The apparatus of claim 13, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:

receiving, from the external computing system, the update; and

modifying the one or more processing devices based on the update from the external computing system.

15. A sensor fusion system comprising:

a memory device configured to store machine-readable instructions; and

one or more processing devices winch, in response to executing the machine-readable, instructions, perform operations comprising;

receiving an environmental model comprising raw measurement data from a plurality of sensors of the vehicle, the plurality of sensors having distinct modalities, wherein the raw measurement data is temporally aligned in the environmental model and spatially aligned in the environmental model based on where the sensors are mounted in the vehicle;

receiving object annotations indicating a detection of at least one object based on the raw measurement data in the environmental model, the object annotations comprising a confidence level associated with the detection of the at least one object;

modifying the environment model to include the object annotations, resulting in an annotated environmental model, wherein the object annotations are temporally aligned and spatially aligned in the annotated environmental model;

generating a classification result of a current situation of the vehicle based, at least in part, on the raw measurement data and the object annotations in the annotated environmental model; and

transmitting the classification result to a processor associated with driving functionality of the vehicle, wherein the processor associated with driving functionality is configured to control operation the vehicle based, at least in part, on the classification result.

16. The sensor fusion system of claim 15, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

generate, based on the classification result, a feedback signal; and

transmit, to the at least one sensor in the plurality of sensors, the feedback signal, causing at least one of repositioning the at least one sensor, or expanding a field of view of at least one sensor, or changing a rate of exposure of the at least one sensor.

17. The sensor fusion system of claim 15, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to provide the classification result to a processor configured to adjust a region of interest in the environmental model based, at least in part, on the classification result.

18. The sensor fusion system of claim 15, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

determine that the classification result identifies an uncertain vehicle situation; and

provide classification data associated with the classification result to an external computing system located away from the vehicle, wherein the external computing system is configured to use the classification data to generate an update to the sensor fusion system based on the uncertain vehicle situation.

19. The sensor fusion system of claim 18, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

receive, from the external computing system, the update; and

modify the one or more processing devices based on the update from the external computing system.