Correcting frame-to-frame image changes due to motion for three dimensional (3-D) persistent observations

• US 9,294,755 B2
• Filed: 10/20/2010
• Issued: 03/22/2016
• Est. Priority Date: 10/20/2010
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Robinson, Ian S., Bloomer, John D., Vahey, Michael D.
• Primary Examiner(s): Parker, Allen
• Assistant Examiner(s): LE, LONG HUYNH
• Application Number:
  US12/908,540
• Publication Number:
  US 20120098933A1
• Time in Patent Office: 1,980 Days
• Field of Search: 248/46, 248/218.1, 382/103, 382/104, 382/284
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

An imaging platform minimizes inter-frame image changes when there is relative motion of the imaging platform with respect to the scene being imaged, where the imaging platform may be particularly susceptible to image change, especially when it is configured with a wide field of view or high angular rate of movement. In one embodiment, a system is configured to capture images and comprises: a movable imaging platform having a sensor that is configured to capture images of a scene, each image comprising a plurality of pixels; and an image processor configured to: digitally transform captured images with respect to a common field of view (FOV) such that the transformed images appear to be taken by a non-moving imaging platform, wherein the pixel size and orientation of pixels of each transformed image are the same. A method for measuring and displaying 3-D features is also described.

Claims

1. A system configured to capture images, comprising:

an airborne or space-based movable imaging platform having a single sensor that is configured to capture images of a scene, each image comprising a plurality of pixels; and

an image processor configured to;

calculate one or more transformations based on a deterministic path determined from information received regarding movement of the imaging platform, viewing geometry of the single sensor, or both;

digitally transform captured images with respect to a common field of view (FOV), the one or more transformations compensating for the relative motion between the scene and the imaging platform such that the transformed images appear to be taken by a non-moving imaging platform, in which the pixel size and orientation of pixels are the same, and any motion remaining being residual motion; and

from the residual motion, identify and track apparent motion of one or more features of a stationary target in the scene, each of which moves, apparently, along a calculated deterministic path as a function of the imaging platform'"'"'s motion and location of the stationary target with respect to center of the field of view of the imaging platform.

2. The system according to claim 1, wherein, in tracking the one or more features, the image processor is configured to perform one or more of:

edge detection, texture detection, shape filtering, corner detection, local maxima detection, or frame difference detection.

3. The system according to claim 1, wherein the image processor is configured to receive information regarding the deterministic path from another source.

4. The system according to claim 1, wherein the image transformations are determined based on a relative motion of the imaging platform, a viewing geometry of the sensor with respect to the imaging platform, or both.

5. The system according to claim 1, wherein the transformations comprise eigenfunctions based on the known trajectory of the movable imaging platform relative to the scene being imaged.

6. The system according to claim 1, wherein the transformations include one or more of:

rotation, zoom, skew, parallax, or translation, when necessary.

7. The system according to claim 1, wherein the image processor is further configured to provide resolution enhancement of images.

8. The system according to claim 7, wherein, in performing the resolution enhancement, the image processor is configured to perform a super-resolution image enhancement process of images.

9. The system according to claim 1, wherein the image processor is configured to output a pair of stereoscopic images from one or more transformed images, and wherein the pair of stereoscopic images include two transformed images having a separation selected by the image processor.

10. The system according to claim 1, wherein the image processor is further configured to determine, from the transformed images, a height, an elevation, or three-dimensional information associated with the motion and/or lean of the tracked features of the stationary target.

11. The system according to claim 10, wherein the image processor is configured to output stereoscopic images when a determined height of the feature exceeds a predetermined height.

12. The system according to claim 11, wherein the image processor is further configured to annotate, encode information in, or both, in image frames based on the determined height, elevation, or 3-D information associated with the determined feature of the tracked target.

13. The system according to claim 1, wherein the image processor is configured to receive feedback from one or more user terminals to correct image misalignment.

27. The system according to claim 1, wherein the transformations comprise homography functions based on the known trajectory of the movable imaging platform relative to the scene being imaged.

14. A method for capturing images, the method comprising:

capturing images of a scene with a sensor on an airborne or space-based moving imaging platform having a single sensor, each image comprising a plurality of pixels;

calculating one or more transformations based on a deterministic path determined from information received regarding movement of the imaging platform, viewing geometry of the single sensor, or both;

digitally transforming captured images with respect to a common field of view (FOV), the one or more transformations compensating for the relative motion between the scene and the imaging platform such that the transformed images appear to be taken by a non-moving imaging platform, in which the pixel size and orientation of pixels are the same, and any motion remaining being residual motion; and

from the residual motion, identifying and tracking apparent motion of one or more features of a stationary target in the scene, each of which moves, apparently, along a calculated deterministic path as a function of the imaging platform'"'"'s motion and location of the stationary target with respect to center of the field of view of the imaging platform.

15. The method according to claim 14, wherein said tracking the one or more features comprises performing one or more of:

edge detection, texture detection, shape filtering, corner detection, local maxima detection, or frame difference detection.

16. The method according to claim 14, further comprising:

receiving information regarding the deterministic path from another source.

17. The method according to claim 14, wherein said image transformations are determined based on a relative motion of the imaging platform and a viewing geometry of the sensor with respect to the imaging platform.

18. The method according to claim 14, wherein the transformations comprise eigenfunctions based on the known trajectory of the movable imaging platform relative to the scene being imaged.

19. The method according to claim 14, wherein the transformations include one or more of:

rotation, zoom, skew, parallax, or translation, when necessary.

20. The method according to claim 14, further comprising:

performing resolution enhancement of images.

21. The method according to claim 20, wherein the resolution enhancement comprises:

performing a super-resolution image enhancement process of images.

22. The method according to claim 14, further comprising:

outputting a pair of stereoscopic images from one or more transformed images; and

selecting transformed images having a separation to be used as the pair of stereoscopic images.

23. The method according to claim 14, further comprising:

determining, from the transformed images, a height, an elevation, or 3-D information associated with the motion and/or lean of the tracked features of the stationary object.

24. The method according to claim 23, further comprising:

outputting stereoscopic images when a determined height of the feature exceeds a predetermined height.

25. The method according to claim 23, further comprising:

annotating, encoding information in, or both, in image frames based on the determined height, elevation, or 3-D information associated with the determined feature of the tracked target.

26. The method according to claim 14, further comprising:

receiving feedback from one or more user terminals, and correcting image misalignment based on said feedback.

28. The method according to claim 14, wherein the transformations comprise homography functions based on the known trajectory of the movable imaging platform relative to the scene being imaged.