Real time 3D imaging for remote surveillance

• US 8,717,418 B1
• Filed: 02/08/2011
• Issued: 05/06/2014
• Est. Priority Date: 02/08/2011
• Status: Active Grant

• Current Assignee: John Prince
• Original Assignee: John Prince
• Inventors: Prince, John
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): PONTIUS, JAMES M
• Application Number:
  US12/931,698
• Time in Patent Office: 1,183 Days
• Field of Search: 348/43, 348/35, 348/135, 348/145, 348/46, 348/47, 348/50, 345/419, 351/212, 382/104, 382/133, 382/154, 382/168, 396/287, 701/45, 701/301
• US Class Current: 348/46
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/133 Equalising the characteristics of different image components, e.g. their average brightness or colour balance
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/218 using spatial multiplexing

Abstract

By defining an angular separation in a train of sequential images, and using an interlaced sequence of pairs of images matched by that defining angle, it is possible to create live 3D video from a single camera mounted on a remote vehicle as though in the in the immediate vicinity of the object being viewed. Such a camera can be mounted on a moving vehicle such as a plane or a satellite In addition, computational power is provided to adaptively (and predictively) smooth out motion irregularities between these image pairs, so that smooth 3D video may be obtained. Continual feature-based correlation between successive frames allows corrections for various transformations so that there is a one-on-one correspondence in size, projection, orientation, etc. between matched frames, which enables capture and display of smooth 3D video.

Claims

1. A method for forming images in stereo pairs comprising the steps of:

moving a detecting device, said detecting device having imaging means, along a scan path;

capturing a sequence of image data formed as said imaging means moves along said scan path, said image data comprising an image frame sampled at an image spatial sampling density, and an anchor frame which is a spatial subset of said image data, said anchor frame sampled at an anchor feature spatial sampling density which is greater than said image spatial sampling density, the position of an anchor feature within said image frame being estimated from previous anchor feature positions to form a trend line;

forming navigation information representative of travel of said detecting device along said scan path;

said travel including perturbations and transformations;

forming output images from said image data, including removing image distortion artifacts arising from said perturbations and transformations of said detecting device using said navigation information;

said output images formed by spatially translating the positions of said image frames using said anchor feature identified in said anchor frames by said trend line;

said sequence of image data forming interlaced frames appearing in such a manner and divided by such intervals on said detecting device that when subsequently transmitted and viewed will appear as three-dimensional images on a display.

2. The method in claim 1 wherein said step of moving said detecting device is a step of moving said detecting device remotely across a field of interest.

3. The method in claim 1 wherein said step of forming navigation information includes determining an anchor feature within each specific area of interest as a fixation point and:

determining said trend line by taking moving averages of the apparent motion of said anchor feature on said detecting device; and

accounting for and compensating for said perturbations in the movement of said anchor feature.

4. The method in claim 3 wherein accounting for and compensating for said perturbations of said anchor feature include averaging in a manner which will smooth image outputs to said display.

5. The method of claim 1 where said anchor feature spatial sampling density is at least ten times greater than said image spatial sampling density, and a image data size of said anchor frame is less than one percent of a image data size of said image frame.

6. The method in claim 5 wherein said anchor frame is examined for an anchor feature, said anchor feature used to form said trend line which includes past and image frames such that image frames which have not yet been acquired have a pre-determined spatial reference with respect to previously acquired images.

7. The method as in claim 6 wherein pixels are oversampled to sense trends predictively in relationships of a region of interest around, or apparent motion of, said anchor feature.

8. The method in claim 7 wherein said oversampling is randomized to eliminate at least one of bias or aliasing.

9. The method in claim 1 wherein said sequence of successive viewed image data corresponds to an angular separation in a range of 0.9 to 1.1 degrees with respect to said anchor feature.

10. The method in claim 1 wherein movements, losses or occlusions in a region of interest around said anchor feature are amended to smooth said sequence of image data when sent to said display.

11. The method in claim 1 wherein said transformations are simplified and short-cut in a manner to speed up image outputs to said display.

12. The method in claim 1 wherein said intervals correspond to certain angular separations between said sequential frames selected to enable stereo viewing as if in a vicinity of objects being viewed.

13. The method in claim 1 wherein a zoom request causes said intervals to be formed as subsets of said image frames, said subsets of said image frames selected to provide an angular separation which provides a requested zoom effect.

14. The method in claim 13 wherein said zoom effect is created instantaneously through software selection of said subsets of successive said image frames.

15. The method in claim 1 in which said detecting device having imaging means includes at least one of:

a digital camera operative in the visible spectrum, an infra-red camera operative in the infra-red spectrum, microwave imaging device or a sonar acoustic imaging device.

16. An apparatus forming images in stereo pairs having:

a detecting device which moves along a scan path;

the detecting device capturing a sequence of image data formed as said imaging means moves along said scan path, said image data comprising an image frame sampled at an image spatial sampling density, and an anchor frame which is a spatial subset of said image data, said anchor frame sampled at an anchor feature spatial sampling density which is greater than said image spatial sampling density, the position of an anchor feature within said image frame being estimated from previous anchor feature positions to form a trend line;

the detecting device forming navigation information representative of travel of said detecting device along said scan path;

said travel including perturbations and transformations;

the detecting device forming output images from said image data, including removing image distortion artifacts arising from said perturbations and transformations of said detecting device using said navigation information;

said output images formed by spatially translating the positions of said image frames using said anchor feature identified in said anchor frames by said trend line;

said sequence of image data forming interlaced frames appearing in such a manner and divided by such intervals on said detecting device that when subsequently transmitted and viewed will appear as three-dimensional images on a display.