Camera image georeferencing systems

• US 8,253,797 B1
• Filed: 03/05/2008
• Issued: 08/28/2012
• Est. Priority Date: 03/05/2007
• Status: Active Grant

• Current Assignee: PureTech Systems, Inc. (Teledyne Technologies Incorporated)
• Original Assignee: PureTech Systems, Inc. (Teledyne Technologies Incorporated)
• Inventors: Thomas, Monroe M., Maali, Fereydoun
• Primary Examiner(s): Blair, Douglas
• Assistant Examiner(s): Survillo, Oleg
• Application Number:
  US12/043,036
• Time in Patent Office: 1,637 Days
• Field of Search: 348/143, 348/113, 348/116, 348/144, 345/419, 382/113, 382/154, 382284-286, 382/288, 382/293, 382/294
• US Class Current: 348/143
• CPC Class Codes: G01C 11/36 Videogrammetry, i.e. electronic processing of video signals from a single source or from different sources to give parallax or range information
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A method for constructing a georeferencing-enabled camera model and its deployment in georeferencing targets located in video sequences due to a single video camera. A video surveillance camera is modeled by a collection of rays converging at a virtual camera point and the retina resolution cell coordinates associated with those rays wherein the ray equations are first established, in the course of a calibration process, for a given camera view, with the aid of other views of the same video surveillance camera, as necessary, and using such a model for mapping image coordinates to terrain coordinates and vice versa in the intended view or its adaptation for use in other views of the same video surveillance camera.

Claims

1. A method of deriving at least one camera model, for security surveillance in at least one underlying terrain and georeferencing pixel coordinates in a camera view of a single security surveillance camera, the camera view being deficient of located landmarks in sufficient number or desired dispersion to produce the at least one camera model, said method comprising the steps of:

a) obtaining, from the single security surveillance camera, a principal camera image comprising a principal view of the at least one underlying terrain, the principal view comprising at least two located landmarks;

b) for each one of the at least two located landmarks of the principal view, producing, by at least one computer processor, a principal node associating physical coordinates of the located landmark in the at least one underlying terrain to pixel coordinates in the principal camera image;

c) generating, by at least one computer processor, a principal nodelist comprising all available principal nodes of the principal view;

d) obtaining from the single security surveillance camera, at least one auxiliary camera image comprising at least one auxiliary view of the at least one underlying terrain, the at least one auxiliary view comprising a plurality of located landmarks;

e) for each one of the plurality of located landmarks of the at least one auxiliary view, producing, by at least one computer processor, an auxiliary node associating physical coordinates of the located landmark in the at least one underlying terrain to pixel coordinates in the at least one auxiliary camera image;

f) generating, by at least one computer processor, at least one auxiliary nodelist comprising all available auxiliary nodes of the at least one auxiliary view;

g) deriving, by at least one computer processor, from both the principal nodelist and the at least one auxiliary nodelist, at least two alignment nodes, each one of the at least two alignment nodes exhibiting respectively the same pixel coordinates in the at least one auxiliary camera image as the pixel coordinates in the principal camera image;

h) geometrically aligning, by at least one computer processor, the principal view to the at least one auxiliary view through pairing of the at least two alignment nodes of the principal nodelist and the at least one auxiliary nodelist;

i) generating, by at least one computer processor, an augmented principal nodelist by adding the nodes of the at least one auxiliary nodelist to the principal nodelist using a first set of geometrical computations, wherein the augmented principal nodelist comprises at least one enlarged set of located landmarks associated with the principal view; and

j) using the augmented principal nodelist, obtaining, by at least one computer processor, the at least one camera model by at least one second set of geometrical computations;

k) wherein the at least one camera model is utilized in security surveillance in the at least one underlying terrain.

2. The method according to claim 1, further comprising the step of, using the at least one camera model and geographical data representing the at least one underlying terrain, mapping image coordinates to terrain coordinates by at least one third set of geometrical computations.

3. The method according to claim 1, further comprising the step of recording for the principal view, positional data relating to a physical position of the single security surveillance camera in object space, said positional data enabling subsequent reacquisition of the principal view by the single security surveillance camera.

4. The method according to claim 3, wherein:

the positional data comprises at least one camera-pan setting and at least one camera-tilt setting; and

a substantially constant camera zoom setting of the single security surveillance camera is maintained in the principal view and the at least one auxiliary view.

5. The method according to claim 1, wherein said step (g) further comprises the steps of:

1) locating within the principal nodelist a first principal alignment node associated with a first one of the at least two located landmarks of the principal view;

2) locating within the principal nodelist a second principal alignment node associated with a second one of the at least two located landmarks of the principal view; and

3) deriving from the at least one auxiliary nodelist, at least one auxiliary alignment node set comprising;

i) a first auxiliary alignment node exhibiting the same pixel coordinates in the at least one auxiliary camera view as the first principal alignment node in the principal camera view, and

ii) a second auxiliary alignment node exhibiting the same pixel coordinates in the at least one auxiliary camera view as the second principal alignment node in the principal camera view.

6. The method according to claim 5, wherein the at least one auxiliary alignment node set of step (3) is derived by the steps of:

a) locating within the at least one auxiliary camera view at least three nodes that together yield a triangular element encompassing the pixel coordinates of at least one of the at least two alignment nodes of the principal camera image; and

b) providing, by interpolation, a virtual alignment node exhibiting the same pixel coordinates as the at least one of the at least two alignment nodes of the principal camera image.

7. The method according to claim 1, wherein said step (g) further comprises the steps of:

1) for an auxiliary camera view lacking the at least two alignment nodes, locating within either of the principal camera view and the at least one auxiliary camera view at least three nodes of a respective camera view that together yield a triangular element encompassing the pixel coordinates of a located alignment node of the other view;

2) providing, by interpolation, a virtual alignment node corresponding to the alignment node, the corresponding virtual alignment node exhibiting the same pixel coordinates as the alignment node of the other view.

8. The method according to claim 1, wherein said first set of geometrical computations further comprises the steps of:

1) forming at least two rays extending from a virtual camera point to each of the at least two alignment nodes of the principal nodelist and the at least one auxiliary nodelist;

2) intersecting the at least two rays by a sphere of fixed arbitrary radius, centered on the virtual camera point, to generate a plurality of ray-end points, the plurality of ray-end points comprising;

i) a first ray-end point and a second ray-end point in the auxiliary view, and

ii) a third ray-end point and a fourth ray-end point in the principal view;

3) selecting within the at least one auxiliary nodelist a tertiary node to be added as a corresponding tertiary node to the primary nodelist;

4) defining the tertiary node as the intersection of an intersecting sphere-set comprising three spheres respectively centered on the first ray-end point, the second ray-end point, and the virtual camera point, wherein each sphere of the first intersecting sphere-set comprise radii equal to a distance extending from their centers to the tertiary point in the auxiliary view;

5) applying to the principal view, a second intersecting sphere-set comprising three corresponding spheres respectively centered on the third ray-end point, the fourth ray-end point, and the virtual camera point, wherein each respective corresponding sphere of the second sphere set comprise radii equal to the spheres of the first intersecting sphere-set;

6) identifying at least one possible tertiary point in the principal view by locating the resulting intersections of the three corresponding spheres of the second intersecting sphere-set;

7) confirming the corresponding tertiary point in the principal view using the signed distance of the at least one possible tertiary point from a plane formed by the virtual camera point and the plurality of ray-end points in the auxiliary view and the principal view to disambiguate the result;

8) adding the resulting corresponding tertiary point to the principal nodelist; and

9) generating an augmented principal nodelist by repeating the above steps (3) through (8) for all other nodes in the at least one auxiliary nodelist.

9. The method according to claim 1, wherein the at least one second set of geometrical computations comprises the steps of:

1) inputting the augmented principal nodelist to at least one triangular mesh generation operation to output a set of triangular elements, each triangular element of the set comprising non-intersecting sides;

2) generating at least one triangular element list comprising the set of triangular elements;

3) computing ray equations, in three-dimensional space, for nodes associated with each vertex of each triangular element of the set of triangular elements, each resulting ray comprising coordinates of a virtual camera point and coordinates of a respective vertex; and

4) generating the at least one camera model by;

i) for the principal camera view, analyzing each resolution cell of the principal camera image to determine, for each associated pixel coordinates, existence of an encompassing triangular element in the at least one triangular element list, the encompassing triangular element comprising the associated pixel coordinates, and

ii) deriving for each such resolution cell comprising a respective encompassing triangular element, an equation for a ray, through at least one interpolation of the rays derived for vertices of the respective encompassing triangular element, wherein said interpolation is influenced by proximity of the resolution cell to the vertices of the respective encompassing triangular element.

10. The method according to claim 9, further comprising the steps of:

k) using the at least one camera model in combination with geographical data representing the underlying terrain and at least one third set of geometrical computations to map image coordinates to terrain coordinates; and

l) recording in at least one lookup table, for each located terrain point, the terrain point coordinates and corresponding range, wherein said at least one third set of geometrical computations comprises the steps of;

i) for each ray of the at least one camera model, locating a terrain point in the object space, wherein each such located terrain point comprises an intersection of a ray with the underlying terrain, and

ii) for each such located terrain point, calculating in the object space terrain point coordinates and a range of the terrain point coordinates from the virtual camera point.

11. The method according to claim 9, further comprising the step of generating at least one (X,Y,Z,R) lookup table for the at least one camera model.

12. The method of claim 9, further comprising the step of reducing off-line processing requirements by deferring the steps subsequent to said generating of the augmented principal nodelist to on-line operation.

13. The method according to claim 9, wherein step (2) further comprises the steps of:

a) generating, for the augmented principal nodelist, at least two triangular element lists, wherein each of the at least two triangular element lists comprises a different initial exterior point of a convex hull, in camera space, of the augmented principal nodelist;

b) locating, within each of the at least two triangular element lists, encompassing triangular elements comprising an excessively large angle; and

c) for each such encompassing triangular element comprising the excessively large angle, replacing the encompassing triangular element with an encompassing element from an alternative element list of the at least two triangular element lists;

d) wherein the quality of encompassing triangular elements resulting from the at least one triangular mesh generation operation is improved.

14. The method according to claim 9, further comprising the steps of:

k) identifying, in camera space, a target blob comprising a discrete region of camera space associated with at least one target of the camera view;

l) determining for the target blob, at least one blob attribute in camera space, the at least one blob attribute at least comprising pixel coordinates of a lowest blob point and pixel coordinates of a blob centroid point;

m) mapping the pixel coordinates of the lowest blob point to a terrain point using the at least one camera model;

n) computing a range value, R_low, comprising a distance from the terrain point to the virtual camera point;

o) associating the same range value R_low to the blob centroid point;

p) intersecting the ray extending through the pixel coordinates of the blob centroid point with a sphere centered on the virtual camera point and comprising a radius equal to R_low,

q) disambiguating the result by selecting from at least one resulting intersection of the ray, a ray-end point that exhibits the lowest Z coordinate value, wherein coordinates of the ray-end point comprise the location of the at least one target in object space.

15. The method according to claim 14, further comprising enabling georeferencing of the at least one target in object space, in a plurality of time-stamped frames of at least one georeferencing-enabled video sequence, to provide estimated true kinematics of the at least one target in object space.

16. The method according to claim 9, further comprising the steps of:

k) identifying, in camera space, at least one target blob comprising a discrete region of camera space associated with at least one target of said camera view;

l) determining for the at least one target blob, at least one blob attribute in camera space, the at least one blob attribute at least comprising pixel coordinates of a lowest blob point and pixel coordinates of a target bounding box comprising at least four corners;

m) mapping the pixel coordinates of the lowest blob point to a terrain point using the at least one camera model;

n) computing a range value, R_low, comprising a distance from the terrain point to the virtual camera point;

o) assigning the range value R_low to the at least four corners of the target bounding box;

p) generating a spherical quadrilateral by intersecting the rays associated with the at least four corners of the target bounding box by a sphere of radius R_low and centered on the virtual camera point, the spherical quadrilateral comprising a spherical quadrilateral area;

q) disambiguating the intersections of sphere and the rays in above step (p), by selecting from resulting intersections of the rays said intersections exhibiting lower Z coordinate value;

r) determining in camera space,

i) a number of pixels in the target bounding box, and

ii) a number of pixels claimed by the at least one target blob; and

s) computing for the object space, a mean frustum area associated with each pixel, wherein said computing comprises dividing of the spherical quadrilateral area by the number of pixels in the target bounding box and multiplying the number of pixels claimed by the at least one target blob in camera space by an equivalent mean frustum area associated with each pixel;

t) wherein an approximation of real size of the at least one target in object space is achieved.

17. A method of georeferencing at least one target, of a camera view of a security surveillance camera, the camera view comprising an associated georeferencing-enabled camera model, the camera model relating image pixel-coordinates of the camera view to rays in three-dimensional space, relative to a virtual camera point, said method comprising the steps of:

a) identifying, by at least one computer processor, in camera space, a target blob comprising a discrete region of camera space associated with the at least one target of the camera view;

b) determining, by at least one computer processor, for the target blob, at least one blob attribute in camera space, the at least one blob attribute at least-comprising pixel coordinates of a lowest blob point and pixel coordinates of a blob centroid point;

c) mapping, by at least one computer processor, the pixel coordinates of the lowest blob point to a terrain point using the georeferencing-enabled camera model;

d) computing, by at least one computer processor, a range value, R_low, comprising a distance from the terrain point to the virtual camera point;

e) associating, by at least one computer processor, the same range value R_low to the blob centroid point;

f) intersecting, by at least one computer processor, a ray extending through the pixel coordinates of the blob centroid point with a sphere centered on the virtual camera point and comprising a radius equal to R_low; and

g) disambiguating, by at least one computer processor, the result by selecting from at least one resulting intersection of the ray, a ray-end point that exhibits the lowest Z coordinate value, wherein coordinates of the ray-end point comprise a location of the at least one target in object space.

18. The method according to claim 17, further comprising enabling georeferencing of the at least one target in object space, in a plurality of time-stamped frames of at least one georeferencing-enabled video sequence, to provide estimated true kinematics of the at least one target in object space.

19. A method of arriving at an approximation of a real size of a target, of a camera view of a security surveillance camera, the camera view comprising an associated camera model, the camera model relating image pixel-coordinates of the camera view to rays in three-dimensional space, relative to a virtual camera point, said method comprising the steps of:

a) identifying, by at least one computer processor, in camera space, at least one target blob comprising a discrete region of camera space associated with the at least one target of the camera view;

b) determining, by at least one computer processor, for the at least one target blob, at least one blob attribute in camera space, the at least one blob attribute at least comprising pixel coordinates of a lowest blob point and pixel coordinates of a target bounding box comprising at least four corners;

c) mapping, by at least one computer processor, the pixel coordinates of the lowest blob point to a terrain point using the associated camera model;

d) computing, by at least one computer processor, a range value, R_low, comprising a distance from the terrain point to the virtual camera point;

e) assigning, by at least one computer processor, the range value R_low to the at least four corners of the target bounding box;

f) generating, by at least one computer processor, a spherical quadrilateral by intersecting rays associated with the at least four corners of the target bounding box by a sphere of radius R_low and centered on the virtual camera point, said spherical quadrilateral comprising a spherical quadrilateral area;

g) disambiguating, by at least one computer processor, intersections of the sphere and the rays in above said step (f), by selecting from resulting intersections of the rays, the intersections exhibiting lower Z coordinate value;

h) determining, by at least one computer processor, in camera space,

i) a number of pixels in the target bounding box, and

ii) a number of pixels spanned by the at least one target blob; and

i) computing for object space, a mean frustum area associated with each pixel, by at least one computer processor, wherein said computing comprises dividing of the spherical quadrilateral area by the number of pixels in the target bounding box and multiplying the number of pixels claimed by the at least one target blob in camera space by an equivalent mean frustum area associated with each pixel.

20. A method, of deriving at least one georeferencing-enabled camera model, usable for security surveillance in at least one underlying terrain and georeferencing image pixel coordinates in a camera view from a virtual camera point of a single security surveillance camera, wherein the at least one georeferencing-enabled camera model comprises at least one lookup table linking the image pixel coordinates of a camera image to rays in three-dimensional space that in turn lead to corresponding points on an underlying terrain, said method comprising the steps of:

a) obtaining, by at least one computer processor, from the single security surveillance camera, a selected camera view of the underlying terrain, the selected camera view comprising an associated camera image;

b) recording, by at least one computer processor, zoom settings of the single security surveillance camera, while retaining a substantially constant zoom setting throughout an operation with the operation extending to model construction and deployment for a selected camera view of the single security surveillance camera;

c) creating, by at least one computer processor, a nodelist comprising a collection of nodes, wherein each node comprises an abstraction of coordinates of a landmark in the at least one underlying terrain along with corresponding image pixel coordinates of the landmark;

d) generating, by at least one computer processor, an element list by inputting the nodelist, to at least one triangular mesh generation operation, in a camera space, the at least one triangular mesh generation operation yielding triangular elements with non-intersecting sides;

e) computing, by at least one computer processor, ray equations in three-dimensional space for each of the nodes associated with vertices of the triangular elements; and

f) constructing, by at least one computer processor, the camera model for the selected camera view by scanning across each resolution cell of the associated camera image and determining for each pixel coordinates an encompassing triangular element in the element list, and for each such encompassing triangular element located, deriving an equation for a ray associated with the virtual camera point and the pixel coordinates of that resolution cell encompassed by the encompassing triangular element, through at least one interpolation of rays associated with vertices of the encompassing element, with the at least one interpolation being influenced by proximity of the resolution cell to the vertices of the encompassing triangular element.