Object motion detection system based on combining 3D warping techniques and a proper object motion detection

• US 8,564,657 B2
• Filed: 05/17/2010
• Issued: 10/22/2013
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Original Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Inventors: Michalke, Thomas, Kastner, Robert
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): ANDERSON II, JAMES M
• Application Number:
  US12/781,272
• Publication Number:
  US 20100315505A1
• Time in Patent Office: 1,254 Days
• Field of Search: None
• US Class Current: 348/118
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/251 involving models

Abstract

The invention proposes a method for detecting dynamic objects in the scene in a driver assistance system of a vehicle, comprising the steps of:

• • feeding signals from sensors (internal sensors, 3D sensors, cameras) of the vehicle to the driver assistance system,
  • generation of a surrounding model using 3D world positions based on the sensor signals and,
  • combination of 3D Warping-based approaches and optical flow based approaches, whose gained novel key points are:
  • a) Restriction of the search space of an optical flow method based on the computation results of the 3D warping method,
  • b) Additionally, to a) also an optimal parameterization of the optical flow approach in terms of search direction, amplitude, etc.,
  • c) Refinement and verification of the detection results of one of the methods based on the computation results of the other method (if both approaches are running in parallel),
  • storing the detected dynamic objects and their measured motion parameters for their use in, e.g. a collision avoidance or path planning system.

Claims

1. A method for detecting dynamic objects in the visually sensed scene of a driver assistance system of a vehicle or robot with ego-motion, comprising the steps of:

providing visual signals of an environment of a vision sensor attached to a vehicle or robot with ego-motion, wherein the vision sensor is a dense depth sensor providing 3D data and wherein an additional sensor provides information on ego-motion,

detecting a proper motion of objects on the input field of the vision sensor based on a detected optical flow,

detecting the motion of objects based on a 3D representation model of the environment and using a 3D warping on the basis of predicted and measured 3D data, the predicted 3D data being generated based on measured 3D data and data representing the ego-motion,

combining the 3D warping-based and the optical flow based object motion detection, by using both detecting steps in parallel and wherein the results of one detecting step is used to verify and refine the results of the other detecting step, wherein the 3D warping defines regions where the optical flow based object motion detection is applied and wherein the optical flow based object motion detection defines regions where the 3D warping is applied, and

storing information on the detected dynamic objects and their measured motion parameters.

2. The method according to claim 1, wherein the information on the detected dynamic objects is used for collision avoidance or path planning.

3. The method according to claim 1, wherein the 3D Warping-based and the optical-flow-based object motion detection are combined such that the search space of the respectively other detection method is restricted and the control parameters are determined and supplied to the respectively other detection method.

4. The method according to claim 1, wherein the 3D Warping based object motion detection parameterizes, as to the direction and amplitude, the optical-flow-based object motion recognition as to the regions, which indicate the lateral motion of objects.

5. The method according to claim 1, wherein only those regions in the 3D Warping based object motion detection which indicate a lateral motion relative to the ego-motion are processed by the optical flow based object motion recognition.

6. The method according to claim 1, wherein the 3D-Warping-based and the optical-flow-based object motion detection are combined such that the detection results of the respectively other detection method is refined and verified.

7. The method according to claim 1, wherein information from a 3D depth sensor is used to generate 3D data.

8. The method according to claim 1, wherein information from a 2D depth sensor is used to generate 3D data.

9. The method according to claim 1, wherein information from a 1D depth sensor is used to generate dense 3D data.

10. The method according to claim 1, the additional sensor provides information on the ego-motion computed based on a single track model.

11. The method according to claim 1, wherein a dense depth sensor provides information of the ego-motion computed based on a single track model.

12. The method according to claim 10, wherein input data for the single track model comes from additional sensors for vehicle velocity and/or yaw-rate.

13. A driver assistance computing unit, designed to carry out a method according to claim 1.

14. A vehicle being equipped with a driver assistance computing unit according to claim 13.

15. An autonomous robot being equipped with a computing unit carrying out a method according to claim 1.

16. A non-transitory computer readable medium, implementing a method according to claim 1 when run on a computing unit.

19. The method according to claim 11, wherein input data for the single track model comes from additional sensors for vehicle velocity and/or yaw-rate.

17. A driver assistance system with a driving path and/or surrounding model generation apparatus, the model generation apparatus comprising:

providing means for providing visual signals of the environment of a vision sensor with ego-motion, wherein the vision sensor is a dense depth sensor providing 3D data and wherein an additional sensor provides information on ego-motion,

detecting means for detecting the proper motion of objects on the input field of the vision sensor based on the detected optical flow,

detecting means for detecting the motion of objects based on a 3D representation model of the environment and using a 3D warping on the basis of predicted and measured 3D data, the predicted 3D data being generated based on measured 3D data and data representing the ego-motion,

combining means for combining the 3D warping-based and the optical flow based object motion detection by using both detecting means in parallel and wherein the results of one detecting means is used to verify and refine the results of the other detecting means, wherein the 3D warping defines regions where the optical flow based object motion detection is applied and wherein the optical flow based object motion detection defines regions where the 3D warping is applied, and

storing means for storing information on the detected dynamic objects and their measured motion parameters.

18. The system according to claim 17, wherein the vision sensor is accessible on a controller area network bus and/or from an additional sensor.