Maneuverless passive range estimation using monocular image sequences

• US 10,235,577 B1
• Filed: 04/25/2016
• Issued: 03/19/2019
• Est. Priority Date: 03/17/2011
• Status: Active Grant

• Current Assignee: UtopiaCompression Corporation
• Original Assignee: UtopiaCompression Corporation
• Inventors: Avadhanam, Sharath, Sheorey, Sameer, Bai, He, Yadegar, Joseph
• Primary Examiner(s): Li, Tracy Y.
• Application Number:
  US15/138,146
• Time in Patent Office: 1,058 Days
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A method to estimate range to a moving rigid body from a moving platform using a monocular camera. The method does not require the camera platform to maneuver in order to estimate range. The method relies on identification and tracking of certain principal features of the object. The method extracts a silhouette of an object from an obtained image and identifies two principal linear components of the silhouette. A normalized distance between the point of intersection of the two linear components and a centroid of the silhouette is computed, compared to a data set and used to determine a direction of movement of the object.

Claims

1. In a computerized system including a camera mounted on a moving vehicle, a method of predicting a collision between a foreign moving object and said moving vehicle, the method comprising the steps of:

acquiring with said camera, consecutively in real time, a plurality of images of said foreign moving object;

detecting said foreign moving object within each of said plurality of images;

identifying two related feature points p₁ and p₂ on said detected foreign moving object within each of said plurality of images;

calculating ranges r¹(t) and r²(t) to said identified feature points p₁ and p₂ on said detected foreign moving object based on changes in the positions of the feature points p₁ and p₂ in the sequential images; and

determining that a collision condition is imminent if at a time t*≥

t_current in the future if |rⁱ(t*)|≤

R_safe for i=1 or 2, t*−

t_current≥

0 and t*−

t_current≤

T_safe

where t_current is the time at which the ranges r¹(t) and r²(t) to the feature points p₁ and p₂ are computed and

where T_safe and R_safe define a safe time and a safe distance which allow for the moving vehicle or the foreign moving object to maneuver to avoid a collision.

2. The method of claim 1 wherein

3. The method of claim 1 further comprising the step of checking for said collision condition for t* that is greater than t_current and closest to it among all possible t*.

4. The method of claim 2 further comprising the step of checking for said collision condition for t* that is greater than t_current and closest to it among all possible t*.

5. The method of claim 1 where

6. The method of claim 2 where

7. The method of claim 6 further comprising the step of checking for said collision condition for t* that is greater than t_current and closest to it among all possible t*.

8. The method of claim 1 wherein said collision condition is selectively used by said computerized system to automatically alter a trajectory of said moving vehicle.

9. The method of claim 6 wherein said collision condition is selectively used by said computerized system to automatically alter a trajectory of said moving vehicle.

10. In a computerized system including a camera mounted on a moving vehicle, a method of predicting a collision between a foreign moving object and said moving vehicle, the method comprising the steps of

acquiring with said camera, consecutively in real time, a plurality of images of said foreign moving object;

detecting said foreign moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected foreign moving object within each of said plurality of images;

calculating ranges r¹(t) and r²(t) to said identified feature points p₁ and p₂ on said detected foreign moving object based on changes in the positions of the feature points p₁ and p₂ in the sequential images; and

determining that a collision condition is imminent if at a time t*≥

t_current in the future if |rⁱ(t*)|≤

R_safe for i=1 or 2, t*−

t_current≥

0 and t*−

t_current≤

T_safe

where t_current is the time at which the ranges r¹(t) and r²(t) to the feature points p₁ and p₂ are computed and

where T_safe and R_safe define a safe time and a safe distance which allow for the moving vehicle or the foreign moving object to maneuver to avoid a collision;

wherein;

the two feature points p₁ and p₂ on said detected foreign moving object satisfy a predetermined geometric relationship with a velocity vector of said detected foreign moving object, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁, j₁) and (i₂, j₂) of said feature points p₁ and p₂ within said image; and

the calculating ranges step further includes recursively calculating the range of said detected foreign moving object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, y₁, y₂,

wherein the step of recursively calculating said range comprises recursively calculating said range where an angle α

is known and is equal to either 0 or π

/2 radians, and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁, j₁) and (i₂, j₂);

[u, v, w] represents a 3D velocity vector of the object;

angle α

is the angle between the vectors [u v w] and [x₁−

x₂ y₁−

y₂ z₁−

z₂]; and

a determination of whether α

is equal to either 0 or π

/2 is made using an Extended Kalman Filter (EKF) where

11. The method of claim 10 wherein α

is estimated using a bank of EKFs, and α

=0 or α

=π

/2 is computed for each filter in each bank and then a voting mechanism is used to determine if α

=0 or α

π

/2 where

if the number of filters with α

=0 is larger than the number of filters for which α

=π

/2, then α

=0, and

if the number of filters with α

=π

/2 is larger than the number of filters for which α

=0, then α

=π

/2.

12. The method of claim 10 wherein α

is estimated using a particle filter, and α

=0 or α

=π

/2 is computed for each particle in the filter and then a voting mechanism is used to determine if α

=0 or α

=π

/2 where

if the number of particles with α

=0 is larger than the number of particles for which α

=π

/2, then α

=0, and

if the number of particles with α

=π

/2 is larger than the number of particles for which α

=0, then α

=π

/2.

13. In a computerized system including a camera mounted on a moving vehicle, a method of predicting a collision between a foreign moving object and said moving vehicle, the method comprising the steps of:

acquiring with said camera, consecutively in real time, a plurality of images of said foreign moving object;

detecting said foreign moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected foreign moving object within each of said plurality of images;

calculating ranges r¹(t) and r²(t) to said identified feature points p₁ and p₂ on said detected foreign moving object based on changes in the positions of the feature points p₁ and p₂ in the sequential images; and

determining that a collision condition is imminent if at a time t*≥

t_current in the future if |rⁱ(t*)|≤

R_safe for i=1 or 2, t*−

t_current≥

0 and t*−

t_current≤

T_safe

where t_current is the time at which the ranges r¹(t) and r²(t) to the feature points p₁ and p₂ are computed and

where T_safe and R_safe define a safe time and a safe distance which allow for the moving vehicle or the foreign moving object to maneuver to avoid a collision;

wherein the two feature points p₁ and p₂ on said detected foreign moving object satisfy a predetermined geometric relationship with a velocity vector of said detected foreign moving object, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁, j₁) and (i₂, j₂) of said feature points p₁ and p₂ within said image; and

wherein the calculating the ranges step further includes recursively calculating the range of said detected foreign moving object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship,

wherein the step of recursively calculating said range comprises recursively calculating said range where an angle α

is known and is equal to either 0 or π

/2 radians, and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁, j₁) and (i₂, j₂);

[u, v, w] represents a 3D velocity vector of the object;

angle α

is the angle between the vectors [u v w] and [x₁−

x₂ y₁−

y₂ z₁−

z₂];

said step of determining pixel locations (i₁, j₁) and (i₂, j₂) comprises extracting a silhouette of said moving object by image segmentation.

14. The method of claim 13 wherein said step of image segmentation comprises the steps of:

receiving a sequence of images of fixed size around said detected moving object;

computing image features from said images by extracting principal components from said images and performing principal component analysis (PCA) on a sub-sample of images;

using the PCA to compute at least one color component with a maximum variance from an input color image;

normalizing said image features; and

weighting the image features to improve segmentation;

segmenting the image features into a plurality of clusters;

identifying the largest clusters;

marking the largest clusters and those clusters which touch an image boundary as background;

marking the remaining clusters as foreground and identifying said remaining clusters as said silhouette of said moving object.

15. The method of claim 14 wherein a segmentation result from a previous image of said sequence of images is used to maintain continuity of said silhouette between images.

16. The method of claim 15, wherein said silhouette from said previous image (shape prior) is registered with a current image by shifting the shape prior until a centroid of an edge map of the previous image matches the current image.

17. The method of claim 16 wherein said step of shifting is estimated by matching a brightest or darkest pixel in the current image and the previous image.

18. In a computerized system including a camera mounted on a moving vehicle, a method of predicting a collision between a foreign moving object and said moving vehicle, the method comprising the steps of:

acquiring with said camera, consecutively in real time, a plurality of images of said foreign moving object;

detecting said foreign moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected foreign moving object within each of said plurality of images;

calculating ranges r¹(t) and r²(t) to said identified feature points p₁ and p₂ on said detected foreign moving object based on changes in the positions of the feature points p₁ and p₂ in the sequential images; and

determining that a collision condition is imminent if at a time t*≥

t_current in the future if |rⁱ(t*)|≤

R_safe for i=1 or 2, t*−

t_current≥

0 and t*−

t_current≤

T_safe

where t_current is the time at which the ranges r¹(t) and r²(t) to the feature points p₁ and p₂ are computed and

where T_safe and R_safe define a safe time and a safe distance which allow for the moving vehicle or the foreign moving object to maneuver to avoid a collision;

wherein the two feature points p₁ and p₂ are two robust feature points p₁ and p₂ on said detected foreign moving object, where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected foreign moving object, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁, j₁) and (i₂, j₂) of said robust feature points p₁ and p₂ within said image; and

wherein the calculating the ranges step further includes recursively calculating the range of said object based on changes in the positions of the robust feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, y₁, y₂, and

wherein said step of determining pixel locations (i₁, j₁) and (i₂, j₂) of said robust feature points p₁ and p₂ within said image comprises the steps of

extracting a silhouette of said moving object comprising a plurality of pixels associated with said detected foreign moving object,

fitting a straight line to the silhouette using a least squares method wherein said straight line defines a principal axis of said silhouette,

projecting all silhouette points to the principal axis,

calculating a truncated histogram of said silhouette points, and

calculating a centroid and interquartile range (IQR) from said histogram,

wherein said robust feature points p₁ and p₂ comprise two points having a distance of IQR away from the centroid along the principal axis.

19. The method of claim 18 wherein the step of recursively calculating said range further comprises recursively calculating said range where an angle α

is known and is equal to either 0 or π

/2 radians, and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁, j₁) and (i₂, j₂);

[u, v, w] represents a 3D velocity vector of the object; and

angle α

is the angle between the vectors [u v w] and [x₁−

x₂ y₁−

y₂ z₁−

z₂].