Maneuverless passive range estimation using monocular image sequences

• US 9,342,746 B1
• Filed: 03/15/2012
• Issued: 05/17/2016
• Est. Priority Date: 03/17/2011
• Status: Active Grant

• Current Assignee: UtopiaCompression Corporation
• Original Assignee: UtopiaCompression Corporation
• Inventors: Tiwari, Abhishek, Bai, He, Ganguli, Anurag, Avadhanam, Sharath, Yadegar, Joseph, Newton, Jason
• Primary Examiner(s): LI, TRACY Y
• Application Number:
  US13/420,964
• Time in Patent Office: 1,524 Days
• Field of Search: 348/148
• US Class Current: 1/1
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/07 Target detection

Abstract

A method to estimate range to a moving rigid body from a moving platform using a monocular camera. The method does not require the camera platform to maneuver in order to estimate range. The method relies on identification and tracking of certain principal features of the object. The method extracts a silhouette of an object from an obtained image and identifies two principal linear components of the silhouette. A normalized distance between the point of intersection of the two linear components and a centroid of the silhouette is computed, compared to a data set and used to determine a direction of movement of the object.

Claims

1. In a computerized system including a camera mounted on a moving vehicle, a maneuverless, passive method for determining a range of a foreign moving object from said moving vehicle, wherein said foreign moving object is not part of said computerized system, the method comprising the steps of:

acquiring with said camera, consecutively in real time, a plurality of images of said foreign moving object wherein no positional data is received from said foreign moving object;

detecting said foreign moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected object within each of said plurality images, where p₁ and p₂ are utilized to determine a direction of travel of said detected object and where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected object; and

recursively calculating the range of said detected object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship without any positional data being received from said detected object and without any assumptions of position or movement of the detected object based on background references.

2. The method of claim 1 wherein said predetermined geometric relationship comprises an assumption that the detected object is traveling in a direction along a line connecting feature points p₁ and p₂.

3. The method of claim 2 wherein said predetermined geometric relationship further comprises an assumption that the two feature points represent a leading point of the detected object and a trailing point of the detected object.

4. The method of claim 2 wherein said step of identifying said feature points p₁ and p₂ comprises the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image, and said step of recursively calculating said range includes converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂.

5. The method of claim 1 wherein said predetermined geometric relationship comprises an assumption that the two feature points represent a leading point of the detected object and a trailing point of the detected object.

6. The method of claim 5 wherein said step of identifying said feature points p₁ and p₂ comprises the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image, and said step of recursively calculating said range includes converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂.

7. The method of claim 1 wherein said step of identifying said feature points p₁ and p₂ comprises the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image, and said step of recursively calculating said range includes converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂.

8. In a computerized system including a camera mounted on a moving vehicle, wherein the camera acquires consecutively in real time a plurality of images of a moving object within a field of view of the camera, a method for determining a range of said moving object from said moving vehicle, the method comprising the steps of:

detecting said moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected object where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected object, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image; and

recursively calculating the range of said object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂, and

wherein the step of recursively calculating said range comprises recursively calculating said range where a rotation matrix R is known and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁,j₁) and (i₂,j₂);

[u, v, w] represents a 3D velocity vector of the object; and

rotation matrix Rε

S0(3) is such that (1/∥

[u v w]∥

)[u v w]^T=(1/∥

[x₁−

x₂ y₁−

y₂ z₁−

z₂]∥

)R[x₁−

x₂ y₁−

y₂ z₁−

z₂]^T.

9. In a computerized system including a camera mounted on a moving vehicle, wherein the camera acquires consecutively in real time a plurality of images of a moving object within a field of view of the camera, a method for determining a range of said moving object from said moving vehicle, the method comprising the steps of:

detecting said moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected object where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected object, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image; and

recursively calculating the range of said object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂, and

wherein the step of recursively calculating said range comprises recursively calculating said range where an angle α

is known and is equal to either 0 or π

/2 radians, and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁,j₁) and (i₂,j₂);

[u, v, w] represents a 3D velocity vector of the object; and

angle α

is the angle between the vectors [u v w] and [x₁−

x₂ y₁−

y₂ z₁−

z₂].

10. In the method of claim 9, said step of determining pixel locations (i₁,j₁) and (i₂,j₂) comprising the steps of:

extracting a silhouette of said object comprising a plurality of pixels associated with said detected object;

extracting a first principal component of said silhouette comprising a first, longer straight line containing a maximum number of said pixels associated with said detected object where (i₁,j₁) and (i₂,j₂) comprise pixel location end points of said first principal component;

extracting a second principal component of said silhouette comprising a second, shorter straight line containing a maximum number of said pixels associated with said detected object where said second straight line makes an angle of greater than 45 degrees with said first straight line, and where (i₃,j₃) and (i₄,j₄) comprise pixel location end points of said shorter principal component;

computing a centroid of said silhouette C(t_n);

computing a point of intersection I(t_n) of said first principal components and said second principal component;

computing a distance D(t_n) between C(t_n) and I(t_n) normalized by a length of said first principal component;

using a statistical T-test on a sequence {D(t_n-Q), . . . , D(t_n)} to test whether a mean of the normalized distances is greater than 0.5 or less than 0.3,

wherein if the mean is greater than 0.5 then the angle α

=0 and the pixel locations of the feature point are (i₁,j₁) and (i₂,j₂), and

further wherein if the mean is less than 0.3 then the angle α

=π

/2 and the pixel location of the feature point are (i₃,j₃) and (i₄,j₄).

11. In the method of claim 9, said step of recursively calculating said range comprises the steps of:

recursively calculating said range wherein said angle α

=0 to obtain an estimated range r<sub>α

=0</sub>^est and an estimated velocity V<sub>α

=0</sub>^est and computing likelihoods p_range(r<sub>α

=0</sub>^est) and p_speed(V<sub>α

=0</sub>^est);

recursively calculating said range where said angle α

=π

/2 to obtain an estimated range r<sub>α

=π

/2</sub>^est and an estimated velocity V<sub>α

=π

/2</sub>^est and computing likelihoods p_range(r<sub>α

=π

/2</sub>^est) and p_speed(V<sub>α

=π

/2</sub>^est); and

computing

12. The method of claim 11 wherein M and m comprise positive real numbers such that m<

M and M and wherein M and m are tunable design parameters.

13. In a computerized system including a camera mounted on a moving vehicle, wherein the camera acquires consecutively in real time a plurality of images of a moving object within a field of view of the camera, a method for determining a range of said moving object from said moving vehicle, the method comprising the steps of:

detecting said moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected object where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected object, said predetermined geometric relationship comprising an assumption that the detected object is traveling in a direction along a line connecting feature points p₁ and p₂, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image; and

recursively calculating the range of said object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂, and

wherein the step of recursively calculating said range comprises recursively calculating said range where a rotation matrix R is known and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁,j₁) and (i₂,j₂);

[u, v, w] represents a 3D velocity vector of the object; and

rotation matrix Rε

S0(3) is such that (1/∥

[u v w]∥

)[u v w]^T=(1/∥

[x₁−

x₂ y₁−

y₂ z₁−

z₂]∥

)R[x₁−

x₂ y₁−

y₂ z₁−

z₂]^T.

15. In the method of claim 13, said step of determining pixel locations (i₁,j₁) and (i₂,j₂) comprising the steps of:

extracting a silhouette of said object comprising a plurality of pixels associated with said detected object;

extracting a first principal component of said silhouette comprising a first, longer straight line containing a maximum number of said pixels associated with said detected object where (i₁,j₁) and (i₂,j₂) comprise pixel location end points of said first principal component;

extracting a second principal component of said silhouette comprising a second, shorter straight line containing a maximum number of said pixels associated with said detected object where said second straight line makes an angle of greater than 45 degrees with said first straight line, and where (i₃,j₃) and (i₄,j₄) comprise pixel location end points of said shorter principal component;

computing a centroid of said silhouette C(t_n);

computing a point of intersection I(t_n) of said first principal components and said second principal component;

computing a distance D(t_n) between C(t_n) and I(t_n) normalized by a length of said first principal component;

using a statistical T-test on a sequence {D(t_n-Q), . . . , D(t_n)} to test whether a mean of the normalized distances is greater than 0.5 or less than 0.3,

wherein if the mean is greater than 0.5 then the angle α

=0 and the pixel locations of the feature point are (i₁,j₁) and (i₂,j₂), and

further wherein if the mean is less than 0.3 then the angle α

=π

/2 and the pixel location of the feature point are (i₃,j₃) and (i₄,j₄).

14. In a computerized system including a camera mounted on a moving vehicle, wherein the camera acquires consecutively in real time a plurality of images of a moving object within a field of view of the camera, a method for determining a range of said moving object from said moving vehicle, the method comprising the steps of:

detecting said moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected object where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected object, said predetermined geometric relationship comprising an assumption that the detected object is traveling in a direction along a line connecting feature points p₁ and p₂, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image; and

recursively calculating the range of said object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂, and

wherein the step of recursively calculating said range comprises recursively calculating said range where an angle α

is known and is equal to either 0 or π

/2 radians, and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁,j₁) and (i₂,j₂);

[u, v, w] represents a 3D velocity vector of the object; and

angle α

is the angle between the vectors [u v w] and [x₁−

x₂ y₁−

y₂ z₁−

z₂].

16. In the method of claim 14, said step of recursively calculating said range comprises the steps of:

recursively calculating said range wherein said angle α

=0 to obtain an estimated range r<sub>α

=0</sub>^est and an estimated velocity V<sub>α

=0</sub>^est and computing likelihoods p_range(r<sub>α

=0</sub>^est) and p_speed(V<sub>α

=0</sub>^est);

recursively calculating said range where said angle α

=π

/2 to obtain an estimated range r<sub>α

=π

/2</sub>^est and an estimated velocity V<sub>α

=π

/2</sub>^est and computing likelihoods p_range(r<sub>α

=π

/2</sub>^est) and p_speed(V<sub>α

=π

/2</sub>^est); and

computing

17. The method of claim 16 wherein M and m comprise positive real numbers such that m<

M and M and wherein M and m are tunable design parameters.

18. In a computerized system including a camera mounted on a moving vehicle, wherein the camera acquires consecutively in real time a plurality of images of a moving object within a field of view of the camera, a method for determining a range of said moving object from said moving vehicle, the method comprising the steps of:

detecting said moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected object where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected object, said predetermined geometric relationship comprising an assumption that the two feature points represent a leading point of the detected object and a trailing point of the detected object, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image; and

recursively calculating the range of said object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂, and

wherein the step of recursively calculating said range comprises recursively calculating said range where a rotation matrix R is known and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁,j₁) and (i₂,j₂);

[u, v, w] represents a 3D velocity vector of the object; and

rotation matrix Rε

S0(3) is such that (1/∥

[u v w]∥

) [u v w]^T=(1/∥

[x₁ x₂ y₁−

y₂ z₁ z₂]∥

)R[x₁ x₂ y₁−

y₂ z₁−

z₂]^T.

19. In a computerized system including a camera mounted on a moving vehicle, wherein the camera acquires consecutively in real time a plurality of images of a moving object within a field of view of the camera, a method for determining a range of said moving object from said moving vehicle, the method comprising the steps of:

detecting said moving object within each of said plurality of images;

identifying two feature points p₁ and p₂ on said detected object where p₁ and p₂ satisfy a predetermined geometric relationship with a velocity vector of said detected object, said predetermined geometric relationship comprising an assumption that the two feature points represent a leading point of the detected object and a trailing point of the detected object, said step of identifying said feature points p₁ and p₂ further comprising the step of determining pixel locations (i₁,j₁) and (i₂,j₂) of said feature points p₁ and p₂ within said image; and

recursively calculating the range of said object based on changes in the positions of the feature points p₁ and p₂ in the sequential images and further based on said predetermined geometric relationship, said step of recursively calculating said range further comprising the step of converting said pixel locations to relative azimuth and elevation angles denoted by β

₁, β

₂, γ

₁, γ

₂, and

wherein the step of recursively calculating said range comprises recursively calculating said range where an angle α

is known and is equal to either 0 or π

/2 radians, and the following conditions are defined;

(x₁, y₁, z₁) and (x₂, y₂, z₂) represent 3D locations of said feature points (i₁,j₁) and (i₂,j₂);

[u, v, w] represents a 3D velocity vector of the object; and

angle α

is the angle between the vectors [u v w] and [x₁−

x₂ y₁−

y₂ z₁−

z₂].

20. In the method of claim 19, said step of determining pixel locations (i₁,j₁) and (i₂,j₂) comprising the steps of:

extracting a silhouette of said object comprising a plurality of pixels associated with said detected object;

extracting a first principal component of said silhouette comprising a first, longer straight line containing a maximum number of said pixels associated with said detected object where (i₁,j₁) and (i₂,j₂) comprise pixel location end points of said first principal component;

extracting a second principal component of said silhouette comprising a second, shorter straight line containing a maximum number of said pixels associated with said detected object where said second straight line makes an angle of greater than 45 degrees with said first straight line, and where (i₃,j₃) and (i₄,j₄) comprise pixel location end points of said shorter principal component;

computing a centroid of said silhouette C(t_n);

computing a point of intersection I(t_n) of said first principal components and said second principal component;

computing a distance D(t_n) between C(t_n) and I(t_n) normalized by a length of said first principal component;

using a statistical T-test on a sequence {D(t_n-Q), . . . , D(t_n)} to test whether a mean of the normalized distances is greater than 0.5 or less than 0.3,

wherein if the mean is greater than 0.5 then the angle α

=0 and the pixel locations of the feature point are (i₁,j₁) and (i₂,j₂), and

further wherein if the mean is less than 0.3 then the angle α

=π

/2 and the pixel location of the feature point are (i₃,j₃) and (i₄,j₄).

21. In the method of claim 19, said step of recursively calculating said range comprises the steps of:

recursively calculating said range wherein said angle α

=0 to obtain an estimated range r<sub>α

=0</sub>^est and an estimated velocity V<sub>α

=0</sub>^est and computing likelihoods p_range(r<sub>α

=0</sub>^est and p_speed(V<sub>α

=0</sub>^est);

recursively calculating said range where said angle α

=π

/2 to obtain an estimated range r<sub>α

=π

/2</sub>^est and an estimated velocity V<sub>α

=π

/2</sub>^est and computing likelihoods p_range(r<sub>α

=π

/2</sub>^est) and p_speed(V<sub>α

=π

/2</sub>^est); and

computing

22. The method of claim 21 wherein M and m comprise positive real numbers such that m<

M and M and wherein M and m are tunable design parameters.