OBJECT RECOGNITION AND CLASSIFICATION USING MULTIPLE SENSOR MODALITIES

• US 20180068206A1
• Filed: 09/08/2016
• Published: 03/08/2018
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: Siemens Electronic Design Automation GmbH
• Original Assignee: Mentor Graphics Corporation (Siemens AG)
• Inventors: Pollach, Matthias Arnold, Mercep, Ljubo
• Granted Patent: US 10,740,658 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

Object recognition and classification based on data from multiple sensor modalities is disclosed. A computing system can detect an event in a monitored portion of an environment coordinate field associated with a vehicle. The computing system can retrieve data associated with the detected event from a plurality of modalities. At least one of the modalities can include raw data from a sensor associated with the vehicle. The computing system can classify the detected event based at least in part on the retrieved data and one or more parameters of a classifier.

Claims

1. A method comprising:

detecting, by a computing system, an event in a monitored portion of an environment coordinate field associated with a vehicle;

retrieving, by the computing system, data associated with the detected event from a plurality of modalities, wherein at least one of the modalities includes raw data from a sensor associated with the vehicle; and

classifying, by the computing system, the detected event based, at least in part, on the retrieved data and one or more parameters of a classifier.

2. The method of claim 1, wherein detecting the event includes:

monitoring a region of interest in the environment coordinate field; and

detecting an event in the region of interest in the environmental coordinate field.

4. The method of claim 1, wherein the retrieved data associated with the detected event includes raw data from two or more separate sensors.

5. The method of claim 1, wherein the retrieved data associated with the detected event includes the raw data from the sensor and high level data from a sensor fusion unit associated with the computing system.

6. The method of claim 1, wherein classifying the detected event includes providing the retrieved data to the classifier, wherein the classifier classifies the detected event using the retrieved data and the parameters of the classifier.

7. The method of claim 6, wherein pre-classification to track the detected event is performed by the classifier.

8. The method claim 7, further comprising:

tracking the detected event over time by collecting additional data associated with the detected event from the plurality of modalities; and

classifying the detected event as the object based, at least in part, on the tracking of the detected event over time.

3. The method of claim 3, wherein retrieving the data comprises retrieving, from the plurality of modalities, data associated with the event from the region of interest in the environmental coordinate field.

9. An apparatus comprising at least one memory device storing instructions configured to cause one or more processing devices to perform operations comprising:

detecting an event in a monitored portion of an environment coordinate field associated with a vehicle;

retrieving data associated with the detected event from a plurality of modalities, wherein at least one of the modalities includes raw data from a sensor associated with the vehicle; and

classifying the detected event based, at least in part, on the retrieved data and one or more parameters of a classifier.

10. The apparatus of claim 9, wherein detecting the event includes:

monitoring a region of interest in the environment coordinate field; and

detecting an event in the region of interest in the environmental coordinate field.

11. The apparatus of claim 10, wherein retrieving the data includes retrieving, from the plurality of modalities, data associated with the event from the region of interest in the environmental coordinate field.

12. The apparatus of claim 9, wherein the retrieved data associated with the detected event includes the raw data from the sensor and high level data from a sensor fusion unit.

13. The apparatus of claim 9, wherein pre-classification to track the detected event is performed by the classifier.

14. The apparatus of claim 13, wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:

tracking the detected event over time by collecting additional data associated with the detected event from the plurality of modalities; and

classifying the detected event as the object based, at least in part, on the tracking of the detected event over time.

15. A system comprising:

a memory device configured to store machine-readable instructions; and

a computing system including one or more processing devices, in response to executing the machine-readable instructions, configured to;

detect an event in a monitored portion of an environment coordinate field associated with a vehicle;

retrieve data associated with the detected event from a plurality of modalities, wherein at least one of the modalities includes raw data from a sensor associated with the vehicle; and

classify the detected event based, at least in part, on the retrieved data and one or more parameters of a classifier.

16. The system of claim 15, wherein to detect the event the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

monitor a region of interest in the environment coordinate field; and

detect an event in the region of interest in the environmental coordinate field.

17. The system of claim 16, wherein to retrieve the data the one or more processing devices, in response to executing the machine-readable instructions, are configured to retrieve, from the plurality of modalities, data associated with the event from the region of interest in the environmental coordinate field.

18. The system of claim 15, wherein the retrieved data associated with the detected event includes the raw data from the sensor and high level data from a sensor fusion unit.

19. The system of claim 15, wherein pre-classification to track the detected event is performed by the classifier.

20. The system of claim 19, wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:

track the detected event over time by collecting additional data associated with the detected event from the plurality of modalities; and

classify the detected event as the object based, at least in part, on the tracking of the detected event over time.