Method and device for analyzing surrounding objects and/or surrounding scenes, such as for object and scene class segmenting

• US 8,934,666 B2
• Filed: 09/30/2009
• Issued: 01/13/2015
• Est. Priority Date: 10/10/2008
• Status: Active Grant

• Current Assignee: ADC Automotive Distance Control Systems GmbH (Continental AG)
• Original Assignee: ADC Automotive Distance Control Systems GmbH (Continental AG)
• Inventors: Lke, Stefan, Schiele, Bernt, Wojek, Christian
• Primary Examiner(s): YENTRAPATI, AVINASH
• Application Number:
  US13/123,017
• Publication Number:
  US 20110200230A1
• Time in Patent Office: 1,931 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

The invention relates to a method and an object detection device for analyzing objects in the environment and/or scenes in the environment. The object detection device includes a data processing and/or evaluation device. In the data processing and/or evaluation device, image data (x^t) is evaluated on the basis of a Conditional Random Field (CRF) model and the CRF model provides additional object nodes (o^t_n) which take into account information from an object detector.

Claims

1. A method for analysing objects in an environment and/or scenes in an environment with an evaluation unit, the method comprising:

evaluating image data (x^t) by the evaluation unit based on a conditional random field model having a first model layer which segments the image data into one or more scenes; and

providing additional object nodes (o^t_n) to the conditional random field model by forming a second model layer occupied by the additional object nodes which take into account information from an object detector that classifies the image data into one or more objects, the additional object nodes of the second model layer being connected to nodes of the first model layer which support object hypotheses of the corresponding additional object nodes, wherein a shared segmentation of the image data of object and scene classes is conducted.

3. The method according to claim 1, wherein within a scope of the analysis of objects in the environment and/or scenes in the environment, adjacent relationships (N₂, N₃) of nodes of the conditional random field model are modelled.

5. The method according to claim 3, wherein the adjacent relationships (N₂, N₃) are modelled using edge potentials.

7. The method according to claim 1, wherein within a scope of the analysis of objects in the environment and/or scenes in the environment, a support vector machine is also used for object detection and/or object classification.

9. The method according to claim 1, wherein on the basis of the conditional random field model within a scope of object detection and/or scene classification, moved cameras and/or moved objects (O) are modelled.

11. The method according to claim 1, wherein the one or more objects (O) and the one or more scenes are modelled in a shared segmentation stage, wherein the modelling is conducted at the first and second model layers with different information granularity and different segmentations in a dynamic conditional random field model.

13. The method according to claim 11, wherein the second model layer has reduced information granularity, the second model layer being used in order to represent the object nodes (o^t_n) and to conduct the method on the basis of dynamic image sequences which contain objects (O) of any size.

15. The method according to claim 1, wherein a pixel-by-pixel object and/or scene class segmentation is conducted.

17. The method according to claim 1, wherein objects (O) of any size are integrated into a conditional random field model framework and an inference is conducted jointly for objects (O) and context and/or scene classes.

19. A method according to claim 1, wherein edge potentials are modelled by taking into account or evaluating a scale supplied by the object detector.

2. A method for analysing objects in an environment and/or scenes in an environment with an evaluation unit, the method comprising:

evaluating image data (x^t) by the evaluation unit based on a conditional random field model having a first model layer which segments the image data into one or more scenes; and

providing additional object nodes (o^t_n) to the conditional random field model by forming a second model layer occupied by the additional object nodes which take into account information from an object detector that classifies the image data into one or more objects, the additional object nodes of the second model layer being connected to nodes of the first model layer which support object hypotheses of the corresponding additional object nodes,

wherein a shared segmentation of the image data of object and scene classes is conducted and dynamic information is integrated within the scope of object detection and scene classification.

4. The method according to claim 2 wherein within a scope of the analysis of objects in the environment and/or scenes in the environment, adjacent relationships (N₂, N₃) of nodes of the conditional random field model are modelled.

6. The method according to claim 4, wherein the adjacent relationships (N₂, N₃) are modelled using edge potentials.

8. The method according to claim 2, wherein within a scope of the analysis of objects in the environment and/or scenes in the environment, a support vector machine is also used for object detection and/or object classification.

10. The method according to claim 2, wherein on the basis of the conditional random field model within a scope of object detection and/or scene classification, moved cameras and/or moved objects (O) are modelled.

12. The method according to claim 2, wherein the one or more objects (O) and the one or more scenes are modelled in a shared segmentation stage, wherein the modelling is conducted at the first and second model layers with different information granularity and different segmentations in a dynamic conditional random field model.

14. The method according to claim 12, wherein the second model layer has reduced information granularity, the second model layer being used in order to represent the object nodes (o^t_n) and to conduct the method on the basis of dynamic image sequences which contain objects (O) of any size.

16. The method according to claim 2, wherein a pixel-by-pixel object and/or scene class segmentation is conducted.

18. The method according to claim 2, wherein objects (O) of any size are integrated into a conditional random field model framework and an inference is conducted jointly for objects (O) and context and/or scene classes.

20. A method according to claim 2, wherein edge potentials are modelled by taking into account or evaluating a scale supplied by the object detector.

21. An object detection device for analysing objects in an environment and/or scenes in an environment with a data processing and/or evaluation device,

wherein in the data processing and/or the evaluation device, image data (x^t) is evaluated on the basis of a conditional random field model having a first model layer which segments the image data into one or more scenes, and

the data processing and/or the evaluation device provides additional object nodes (o^t_n) to the conditional random field model by forming a second model layer occupied by the additional object nodes which take into account information from an object detector that classifies the image data into one or more objects, the additional object nodes of the second model layer being connected to nodes of the first model layer which support object hypotheses of the corresponding additional object nodes, wherein a shared segmentation of the image data of object and scene classes is conducted.

23. The device according to claim 21, wherein the object detection device is integrated into a camera device or into a sensor device.

22. A vehicle with an object detection device for analysing objects in an environment and/or scenes in an environment, with a data processing and/or evaluation device,

wherein in the data processing and/or the evaluation device, image data (x^t) is evaluated on the basis of a conditional random field model having a first model layer which segments the image data into one or more scenes and the data processing and/or the evaluation device provides additional object nodes (o^t_n) to the conditional random field model by forming a second model layer occupied by the additional object nodes which take into account information from an object detector (Ω

) that classifies the image data into one or more objects, the additional object nodes of the second model layer being connected to nodes of the first model layer which support object hypotheses of the corresponding additional object nodes, wherein a shared segmentation of the image data of object and scene classes is conducted, and

the vehicle includes a data storage device for storing image data, which is connected in terms of data to the object detection device and/or to the data processing and/or the evaluation device.

24. The vehicle according to claim 22, wherein the object detection device is integrated into a camera device or into a sensor device.