Determination of the three-dimensional location of a target viewed by a camera

• US 8,049,658 B1
• Filed: 05/25/2007
• Issued: 11/01/2011
• Est. Priority Date: 05/25/2007
• Status: Expired due to Fees

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Smith, Donna F., Lagonik, Louis
• Primary Examiner(s): Sotomayor; John
• Assistant Examiner(s): BYTHROW, PETER M
• Application Number:
  US11/807,309
• Time in Patent Office: 1,621 Days
• Field of Search: 342/26.R, 342/26.A, 342/26.B, 342/26.C, 342/26.D, 342/52, 342 70- 72, 342/118, 342/139, 342/140, 342/146, 342/175, 382/103, 382/154, 3409951-99528, 701/117, 701/207, 701/208, 701/218, 701/223
• US Class Current: 342/52
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/046 using an intermediate storage device, e.g. a recording/reproducing device

Abstract

A method for determining, in three dimensions, the location of a moving ground object observed in a region by a video camera. A terrain map supplies altitude for the latitude and longitude of each terrain point. The terrain information is combined with camera location, field of view, and orientation to produce a computed terrain map relative to the camera. A video analytics step processes the video and locates moving target(s) in two Cartesian coordinates. The coordinates are processed with the camera information to produce target location in terms of azimuth and elevation angle. The computed map information is combined with the angular target location to produce latitude, longitude, and altitude of the target. The target location information is used for further investigation or to attack the target. Also, a method for determining the third dimension of a 2-D radar track to cue a camera or fuse with camera data.

Claims

1. A method for determining the location of a moving object in a region of interest, said method comprising the steps of:

observing said region of interest with a camera to produce a frame-sequential video signal representing at least a part of said region of interest;

providing geospatial coordinates of the location of said camera and information relating to an azimuth angle, elevation angle, and field of view of said camera;

applying said video signal to video analytics processing for detecting objects which move from frame to frame of said video signal and for identifying a location in a two-dimensional Cartesian coordinate system of an object which moves from frame to frame of said video signal;

converting said location of said object to a location in azimuth and elevation angle;

providing a map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the camera location to each map point;

processing said map together with said location of said object in azimuth and elevation angle together with said location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera, for producing information relating to a location in three dimensions of said moving object; and

displaying the three-dimensional location of said moving object.

2. A method according to claim 1, wherein said step of providing a map including the altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the camera location to each map point, comprises the steps of:

providing a terrain map including altitudes of geospatial coordinate points, at least in the regions of interest;

processing said terrain map together with said location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera, to produce a computed map including latitude, longitude and altitude as a function of azimuth and elevation.

3. A method according to claim 1, wherein said step of processing said map together with said location of said object in azimuth and elevation angle together with said location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera, for producing information relating to the location in three dimensions of said moving object, includes the steps of:

identifying a plurality of contiguous or juxtaposed azimuth-range cells covering terrain points available from said map, so that plural terrain points lie in each azimuth-range cell;

for each terrain point, calculating range, azimuth and elevation relative to the camera location;

locating the one of said azimuth-range cells in which the target is located;

identifying the terrain point within that one of said azimuth-range cells which is closest to said target; and

deeming said terrain point within that one of said azimuth-range cells which is closest to said target to be the location of said target.

6. A method according to claim 1, for performing close surveillance of a moving object in a region of interest, said method comprising the steps of:

additionally observing said region of interest with radar to produce a two-dimensional radar track including at least range and azimuth of an object in the region of interest;

providing geospatial coordinates of a location of said radar;

providing a further map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the radar location to each map point;

processing said further map together with said location of said radar track in range and azimuth angle together with said location of said radar for producing three dimensional radar object location information; and

cueing a camera with said three dimensional radar object location information to observe said three dimension object location to further identify the object.

7. A method according to claim 1, for performing close surveillance of a moving object in a region of interest, said method comprising the steps of:

additionally observing said region of interest with radar to produce a two-dimensional radar track, including at least range and azimuth, of an object in the region of interest;

providing geospatial coordinates of a location of said radar;

providing a further map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the radar location to each map point;

processing said further map together with said location of said radar track in range and azimuth angle together with said location of said radar for producing three dimensional radar object location information;

correlating the processed three dimensional radar object location information to the processed three dimensional object location obtained from said information from said camera;

fusing the information from the two object locations to improve an accuracy of the three-dimensional object location; and

guiding a surveillance craft to said three-dimensional object location.

4. A method for performing close surveillance of a moving object in a region of interest, said method comprising the steps of:

observing said region of interest with a camera to produce a frame-sequential video signal representing at least a part of said region of interest;

providing geospatial coordinates of a location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera;

applying said video signal to video analytics processing for detecting objects which move from frame to frame of said video signal and for identifying a location in a two-dimensional Cartesian coordinate system of an object which moves from frame to frame of said video signal;

converting said location of said object to a location in azimuth and elevation angle;

providing a map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the camera location to each map point;

processing said map together with said location of said object in azimuth and elevation angle together with said location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera, for producing information relating to, a location in three dimensions of said moving object; and

guiding one of a surveillance craft and a surveillance device to said three-dimensional location of said moving object.

5. A method for attacking a moving object in a region of interest, said method comprising the steps of:

observing said region of interest with a camera to produce a frame-sequential video signal representing at least a part of said region of interest;

providing geospatial coordinates of a location of said camera and information relating to azimuth angle, elevation angle, and field of view of said camera;

applying said video signal to video analytics processing for detecting objects which move from frame to frame of said video signal and for identifying a location in a two-dimensional Cartesian coordinate system of an object which moves from frame to frame of said video signal;

converting said location of said object to a location in azimuth and elevation angle;

providing a map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the camera location to each map point;

processing said map together with said location of said object in azimuth and elevation angle together with said location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera, for producing information relating to a location in three dimensions of said moving object; and

directing a weapon toward said three-dimensional location for disabling said moving object.

8. A system for determining the location of a moving object in a region of interest, said system comprising:

a camera, and

a processor associated with the camera, the processor executing instructions for performing the steps of;

observing said region of interest with the camera to produce a frame-sequential video signal representing at least a part of said region of interest;

providing geospatial coordinates of the location of said camera and information relating to an azimuth angle, elevation angle, and field of view of said camera;

applying said video signal to video analytics processing for detecting objects which move from frame to frame of said video signal and for identifying a location in a two-dimensional Cartesian coordinate system of an object which moves from frame to frame of said video signal;

converting said location of said object to a location in azimuth and elevation angle;

providing a map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the camera location to each map point;

processing said map together with said location of said object in azimuth and elevation angle together with said location of said camera and information relating to the azimuth'"'"'angle, elevation angle, and field of view of said camera, for producing information relating to a location in three dimensions of said moving object; and

displaying the three-dimensional location of said moving object.

9. A system according to claim 8, wherein said step of providing a map including the altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the camera location to each map point, comprises the steps of:

providing a terrain map including altitudes of geospatial coordinate points, at least in the regions of interest;

processing said terrain map together with said location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera, to produce a computed map including latitude, longitude and altitude as a function of azimuth and elevation.

10. A system according to claim 8, wherein said step of processing said map together with said location of said object in azimuth and elevation angle together with said location of said camera and information relating to the azimuth angle, elevation angle, and field of view of said camera, for producing information relating, to the location in three dimensions of said moving object, includes the steps of:

identifying a plurality of contiguous or juxtaposed azimuth-range cells covering terrain points available from said map, so that plural terrain points lie in each azimuth-range cell;

for each terrain point, calculating range, azimuth and elevation relative to the camera location;

locating the one of said azimuth-range cells in which the target is located;

identifying the terrain point within that one of said azimuth-range cells which is closest to said target; and

deeming said terrain point within that one of said azimuth-range cells which is closest to said target to be the location of said target.

11. A system according to claim 8, wherein the processor executes instructions for performing the further steps of:

additionally observing said region of interest with radar to produce a two-dimensional radar track including at least range and azimuth of an object in the region of interest;

providing geospatial coordinates of a location of said radar;

providing a further map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the radar location to each map point;

processing said further map together with said location of said radar track in range and azimuth angle together with said location of said radar for producing three dimensional radar object location information; and

cueing a camera with said three dimensional radar object location information to observe said three dimension object location to further identify the object.

12. A system according to claim 8, wherein the processor executes instructions for performing the further steps of:

additionally observing said region of interest with radar to produce a two-dimensional radar track, including at least range and azimuth, of an object in the region of interest;

providing geospatial coordinates of a location of said radar;

providing a further map including altitudes of geospatial coordinate points, at least in the region of interest, together with computed range, azimuth, and elevation from the radar location to each map point;

processing said further map together with said location of said radar track in range and azimuth angle together with said location of said radar for producing three dimensional radar object location information;

correlating the processed three dimensional radar object location information to the processed three dimensional object location obtained from said information from said camera;

fusing the information from the two object locations to improve an accuracy of the three-dimensional object location; and

guiding a surveillance craft to said three-dimensional object location.

13. A system according to claim 8, wherein the camera is a controllable field of view camera.

14. A system according to claim 8, further comprising a global positioning system associated with the camera.

15. A system according to claim 8, further comprising a camera controller associated with the camera, the camera controller configured for controlling at least one of pan, tilt, zoom, speed, and field-of-view of the camera.

16. A system according to claim 8, further comprising a source of radar data associated with the processor.

17. A system according to claim 8, further comprising a source of ground sensor data associated with the processor.

18. A system according to claim 8, further comprising a display device associated with the camera for displaying the three-dimensional location of said moving object.

19. A system according to claim 8, further comprising an anti-incursion weapon or close surveillance device associated with the processor.

20. A system according to claim 8, further comprising a video analytics system for performing a pixel-to-pixel comparison of video from the camera to identify motion from frame to frame in any pixel-to-pixel changes.