Single-camera distance estimation

• US 10,008,002 B2
• Filed: 02/28/2012
• Issued: 06/26/2018
• Est. Priority Date: 02/28/2012
• Status: Active Grant

• Current Assignee: NXP USA, Inc. (NXP Semiconductors NV)
• Original Assignee: NXP Canada, Inc. (NXP Semiconductors NV)
• Inventors: Lalonde, Jeffrey, Martel, Luc, Laganiere, Robert
• Primary Examiner(s): Mung, On S
• Application Number:
  US13/208,919
• Publication Number:
  US 20130222589A1
• Time in Patent Office: 2,310 Days
• Field of Search: 348148
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/168 Driving aids for parking, e.g. acoustic or visual feedback on parking space

Abstract

Systems and methods for estimating distances to objects or features are described. The distance estimation tracks features across images captured from a single camera mounted to an automobile and determines features that are off-ground. Features are tracked based on a ground motion estimate and respective motion of the features tracked across two or more successive images. For each off-ground feature a distance to the camera is calculated based on motion of the feature in the images captured by the camera projected to a ground-plane and the ground motion estimate.

Claims

1. A method for determining a distance to an obstacle from a camera mounted to an automobile, the method comprising:

tracking a plurality of image features across two or more successive images captured by the camera;

when an average of a ground motion estimate is above a threshold,

generating top-view images by transforming the camera images to appear to have been taken from above, and

determining a ground motion estimate based on vertical motion of a plurality of the tracked image features that are in a lower half of the top-view images;

identifying a plurality of the tracked plurality of image features as offground features based on the ground motion estimate and respective motion of the plurality of image features tracked across two or more of the top-view images, each of the plurality of identified off-ground features corresponding to at least a portion of an object above or below a ground surface;

calculating for each of the plurality of off-ground features of the plurality of image features a distance from the portion of the object corresponding to the respective off-ground feature to the camera based on motion of the respective off-ground feature in the top-view image and the ground motion estimate.

2. The method of claim 1, further comprising:

identifying a most recent one of the successive images as a snapshot,

determining if an average displacement of tracked image features between a next one of the successive images and the snapshot is above another threshold;

identifying the next one of the successive images as a new snapshot when the average displacement is above the other threshold; and

determining the ground motion estimate using the new snapshot when the new snapshot is identified.

3. The method of claim 2, wherein the average displacement is determined using tracked image features with a displacement above a minimum threshold.

4. The method of claim 2, further comprising:

determining the ground motion estimate based on motion of a plurality of tracked image features across two or more successive snapshots.

5. The method of claim 4, wherein determining the ground motion estimate comprises:

determining an estimate of ground motion parameters describing motion between a current snapshot and one or more previous snapshots, the ground motion parameters defining a 2-dimensional rigid transformation based on {θ

, tx, ty},

wherein;

θ

is a rotation of the camera;

tx is a translation of the camera in a first direction; and

ty is a translation of the camera in a second direction perpendicular to the first direction.

6. The method of claim 5, wherein determining an estimate of the ground motion parameters comprises:

approximating ground motion as an approximate affine transformation using a linear least squares fit of the plurality of tracked image features used to determine the ground motion estimate; and

refining the approximate affine transformation using an iterative Newton;

Gauss method.

17. The method of claim 1, further comprising:

identifying one of the successive images as a snapshot for use in determining the ground motion estimate.

18. The method of claim 3, further comprising:

determining the ground motion estimate based on motion of a plurality of tracked image features across two or more successive snapshots.

7. A processor for executing instructions stored in memory, the processor for determining a distance to an obstacle from a camera mounted to an automobile, the instructions configuring the processor to:

generate top-view images from camera images by transforming the camera images to appear to have been taken from above;

track a plurality of image features across two or more successive top-view images when an average of a ground motion estimate for the top-view images is above a threshold;

determine the ground motion estimate based on motion of a plurality of the tracked image features across the two or more successive top-view images;

identify a plurality of the tracked plurality of image features as off-ground features based on the ground motion estimate and respective motion of the plurality of image features tracked across the two or more successive top-view images, each of the plurality of identified off-ground features corresponding to at least a portion of an object above or below a ground surface; and

for each of the plurality of off-ground features of the plurality of image features, calculate a distance from the portion of the object corresponding to the respective off-ground feature to the camera based on motion of the respective off-ground feature in the top view images and the ground motion estimate.

8. The processor of claim 7, wherein the instructions when executed by the processor, further configuring the processor to for each image captured by the camera, generate the corresponding successive image by projecting the image to the ground-plane.

9. The processor of claim 7, wherein the instructions when executed by the processor, further configuring the processor to:

generate one or more snapshots from one or more of the successive top-view images.

10. The processor of claim 9, wherein generation of a new snapshot comprises:

determining if an average displacement of tracked image features between a next one of the successive images and a latest snapshot is above a threshold; and

generating the new snapshot of the next one of the successive top-view image when the average displacement is above a threshold.

11. The processor of claim 10, wherein the average displacement is determined using tracked image features with a displacement above a minimum threshold.

12. The processor of claim 7 wherein determining the ground motion estimate comprises:

determining an estimate of ground motion parameters describing motion between a current snapshot and one or more previous snapshots, the ground motion parameters defining a 2-dimensional rigid transformation based on {θ

, tx, ty},

wherein;

θ

is a rotation of the camera;

tx is a translation of the camera in a first direction; and

ty is a translation of the camera in a second direction perpendicular to the first direction.

13. The processor of claim 12, wherein determining an estimate of the ground motion parameters comprises:

approximating ground motion as an approximate affine transformation using a linear least squares fit of the plurality of tracked image features used to determine the ground motion estimate; and

refining the approximate affine transformation using an iterative Newton-Gauss method.

15. The processor of claim 8, wherein the instructions when executed by the processor, further configuring the processor to:

generate one or more snapshots from one or more of the successive images.

16. The processor of claim 11, wherein the instructions when executed by the processor, further configuring the processor to determine the ground motion estimate based on motion of a plurality of tracked image features across two or more successive snapshots.

14. A system for determining a distance to an obstacle, the system comprising:

a camera to be mounted to an automobile;

a memory for storing instructions; and

a processor for executing instructions, the processor for determining a distance to an obstacle from the camera when mounted to the automobile, the instructions configuring the system to;

track a plurality of image features across two or more successive images;

once a predetermined number of the plurality of image features are tracked and there is at least a predetermined amount of displacement between the plurality of tracked image features in the two or more successive images;

determine a ground motion estimate based on motion of the plurality of tracked image features being tracked across the two of the successive images;

identify a plurality of the tracked plurality of image features as offground features based on the ground motion estimate and respective motion of the plurality of image features tracked across two or more successive images, each of the plurality of identified off-ground features corresponding to at least a portion of an object above or below a ground surface; and

for each of the plurality of off-ground features of the plurality of image features, calculate a distance from the portion of the object corresponding to the respective off-ground feature to the camera based on motion of the respective off-ground feature in the images captured by the camera projected to a ground-plane and the ground motion estimate.