Real-time obstacle detection with a calibrated camera and known ego-motion

• US 7,446,798 B2
• Filed: 02/02/2004
• Issued: 11/04/2008
• Est. Priority Date: 02/05/2003
• Status: Active Grant

• Current Assignee: Koninklijke Philips N.V.
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Comaniciu, Dorin, Xie, Binglong
• Primary Examiner(s): WONG, ALLEN C
• Application Number:
  US10/770,044
• Publication Number:
  US 20040183905A1
• Time in Patent Office: 1,737 Days
• Field of Search: 348/143, 348/148, 382/104, 701/1, 701/301
• US Class Current: 348/148
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 2013/9324 Alternative operation using ultrasonic waves
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

A method and system of real-time obstacle detection from a moving vehicle is provided. The method and system use a calibrated image capturing device. The method and system use a motion estimation technique to pick points with reliable image motion flows, and performs very fast sparse matching between the image motion flows and true motion flows calculated from the ego-motion of the image capturing device. Any mismatch between the image motion flows and the true motion flows are verified over time to achieve robust obstacle detection.

Claims

1. A method for real-time obstacle detection from a vehicle moving relative to a road, comprising:

calibrating an image capturing device, such that any image point can be mapped to a real-world point;

receiving one or more images from the image capturing device;

receiving information about the ego-motion of the image capturing device;

selecting one or more real-world points according to a tractability criterion inversely proportional with an uncertainty associated with localization of the real-world points;

applying a motion estimation method on the one or more real-world points, the one or more images, and the known plane to determine an image motion flow of the one or more real-world points;

determining a true motion flow of the one or more real-world points based on the one or more real world points, the one or more images, and the ego-motion; and

determining that the one or more real-world points is one or more obstacle points by comparing the image motion flow and the true motion flow.

2. The method of claim 1, wherein applying a motion estimation technique comprises applying an optical flow method with statistical fusion.

3. The method of claim 2, wherein receiving information from the one or more sensors comprises receiving information from at least one of a gyro and a GPS.

4. The method of claim 1, wherein receiving information about the ego-motion of the image capturing device comprises receiving information from one or more sensors onboard the vehicle.

5. The method of claim 1, wherein receiving information about the ego-motion of the image capturing device comprises determining the information based on the one or more images.

6. The method of claim 1, wherein determining that the one or more real-world point is one or more obstacle points by comparing the image motion flow and the true motion flow comprises:

if the image motion flow does not match the true motion flow, determining that the one or more real-world points are one or more obstacle points; and

if the image motion flow matches the true motion flow, determining that the one or more real-world points are not obstacle points.

7. The method of claim 1, further comprising:

tracking data of a plurality of instances of the one or more obstacle points over a time interval; and

determining that the one or more obstacle points is one or more actual obstacles based on the data.

8. The method of claim 7, wherein determining that the one or more obstacle points is one or more actual obstacles based on the data comprises determining that the one or more obstacle points is one or more actual obstacles based on the data and information based on other detection methods.

9. The method of claim 8, wherein determining that the one or more obstacle points is one or more actual obstacles based on the data and information based on other detection methods comprises determining that the one or more obstacle points is one or more actual obstacles based on the data and information based on at least one of radar obstacle detection, lidar obstacle detection, and ultrasonic obstacle detection.

10. The method of claim 7, further comprising classifying a type of the one or more actual obstacles using automated detection.

11. The method of claim 7, further comprising classifying a type of the one or more actual obstacles using a graphical user interface.

12. The method of claim 1, wherein calibrating the image capturing device comprises calibrating a video camera.

13. A system for real-time obstacle detection from a vehicle moving relative to a road, comprising:

an image capturing device;

means for calibrating an image capturing device, such that any image point can be mapped to a real-world point;

means for receiving one or more images from the image capturing device;

means for receiving information about the ego-motion of the image capturing device;

means for selecting one or more real-world points according to a tractability criterion inversely proportional with an uncertainty associated with localization of the real-world points;

means for applying a motion estimation method on the one or more real-world points, the one or more images, and the road to determine an image motion flow of the one or more real-world points;

means for determining a true motion flow of the one or more real-world points based on the one or more real world points, the one or more images, and the ego-motion; and

means for determining that the one or more real-world points is one or more obstacle points by comparing the image motion flow and the true motion flow.

14. The system of claim 13, wherein the image capturing device comprises a video camera.

15. The system of claim 13, further comprising:

means for tracking data of a plurality of instances of the one or more obstacle points over a time interval; and

means for determining that the one or more obstacle points is one or more actual obstacles based on the data.

16. The system of claim 13, further comprising one or more onboard vehicle sensors for determining the ego-motion of the image capturing device.

17. The system of claim 16, wherein the one or more onboard vehicle sensors comprises a gyro sensor.

18. The system of claim 16, wherein the one or more onboard vehicle sensors comprises a GPS sensor.

19. A computer readable medium encoded with computer executable instructions for performing a method for real-time obstacle detection from a vehicle moving relative to a road, comprising:

calibrating an image capturing device, such that any image point can be mapped to a real-world point;

receiving one or more images from the image capturing device;

receiving information about the ego-motion of the image capturing device;

selecting one or more real-world points according to a tractability criterion inversely proportional with an uncertainty associated with localization of the real-world points;

applying a motion estimation method on the one or more real-world points, the one or more images, and the road to determine an image motion flow of the one or more real-world points;

determining a true motion flow of the one or more real-world points based on the one or more real world points, the one or more images, and the ego-motion; and

determining that the one or more real-world points is one or more obstacle points by comparing the image motion flow and the true motion flow.