Extended range image processing for electro-optical systems

• US 7,760,965 B2
• Filed: 04/18/2006
• Issued: 07/20/2010
• Est. Priority Date: 04/25/2001
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Alderson, Timothy, Tener, Gene D., Benton, Richard W. Sr.
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US11/405,557
• Publication Number:
  US 20060188169A1
• Time in Patent Office: 1,554 Days
• Field of Search: 382/266, 382/269, 382/284, 382/294, 382/295, 382298-300
• US Class Current: 382/294
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 25/63 applied to dark current
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A method and apparatus for processing imagery using images acquired via any known Electro-Optical (EO) system. In accordance with exemplary embodiments of the present invention, a first frame of data is selected as a template frame (e.g., a given frame). A second frame of data can be captured using the EO system. At least a portion of the second frame can be correlated with the template frame to generate a shift vector. The second frame can then be registered with the template frame by interpolating the second frame using the shift vector and re-sampling at least a portion of the second frame to produce a registered frame. The template frame can also be re-sampled. The registered frame and the re-sampled template frame can then be combined to generate an averaged frame. The averaged frame can be spatially filtered to enhance edges within the averaged frame.

Claims

1. A method of enhancing visual images, comprising the steps of:

(a) selecting a first frame of image data as a template frame;

(b) capturing a second frame of image data;

(c) aligning the second frame of image data with the template frame of image data by correcting line-of-sight deviations therebetween;

(d) spatially oversampling at least a portion of the aligned second frame of image data by a factor greater than one;

(e) integrating the spatially oversampled portion of the aligned second frame of image data with image data of the template frame to produce an output frame of image data; and

(f) repeating steps (b) through (e) a first predetermined number of times with the first frame of image data selected as the template frame to process at least one subsequently-captured second frame of image data into a continuous video stream of output frames produced in step (e), wherein the first and second frames of image data are based on an image data input from a sensor.

23. The method of claim 1, comprising:

(g) after step (f) of repeating steps (b)-(e) the first predetermined number of times, repeating steps (b)-(e) a second predetermined number of times with a different frame as the given frame of image data to process at least one subsequently-captured frame of image data received from the image sensor into the continuous video stream of output frames produced in step (e).

24. The method of claim 23, wherein the different frame is the most recently produced output frame in step (e).

25. The method of claim 23, wherein the first number of times is three and the second number of times is four.

2. A system for enhancing visual images, comprising:

a sensor for generating input data; and

a processor module coupled to the sensor, the processor module configured execute operations comprising;

(a) selecting a first frame of image data as a template frame;

(b) capturing a second frame of image data;

(c) aligning the second frame of image data with the template frame of image data to correct line-of-sight deviations therebetween;

(d) spatially oversampling at least a portion of the aligned second frame of image data by a factor greater than one;

(e) integrating the spatially oversampled portion of the aligned second frame of image data with image data of the template frame to produce an output frame of image data; and

(f) repeating operations (b) through (e) a first predetermined number of times with the first frame of image data selected as the template frame to process at least one subsequently-captured second frame of image data into a continuous video stream of output frames produced in step (e), wherein the first and second frames of image data are based on an image data input from the sensor.

3. A method for processing imagery, comprising:

(a) receiving a frame of image data captured by an image sensor;

(b) generating a measurement to determine an amount by which the received frame of image data is offset relative to a given frame of image data;

(c) aligning the received frame of image data with the given frame of image data to a sub-pixel level in accordance with the determined amount of offset;

(d) combining the aligned frame of image data with the given frame of image data to produce an output frame of image data; and

(e) repeating steps (a)-(d) a first predetermined number of times with the given frame of image data to process at least one subsequently-captured frame of image data into a continuous video stream of output frames produced in step (d).

4. The method of claim 3, comprising

capturing the received frame of image data along a line of sight.

5. The method of claim 4, comprising:

adding motion to the line of sight according to a random pattern to offset the received frame of image data relative to the given frame of image data.

6. The method of claim 3, wherein the received frame of image data is a frame of infrared image data captured by the image sensor.

7. The method of claim 3, wherein the given frame comprises a template frame.

8. The method of claim 7, wherein the template frame comprises a combination of a plurality of frames of image data.

9. The method of claim 3, wherein the generating in step (b) includes correlating at least a portion of the received frame of image data with at least a portion of the given frame of image data.

10. The method of claim 9, wherein the correlating comprises:

identifying the portion of the received frame of image data.

11. The method of claim 9, wherein the correlating comprises:

generating a shift vector.

12. The method of claim 3, wherein the aligning in step (c) comprises:

interpolating the received frame of image data.

13. The method of claim 3, wherein the aligning in step (c) comprises:

resampling the received frame of image data.

14. The method of claim 13, wherein the resampling comprises:

oversampling the received frame of image data so that the output frame of image data has a higher resolution than the received frame of image data.

15. The method of claim 3, wherein the aligning in step (c) comprises:

oversampling the captured frame of image data so that the output frame of image data has a higher resolution than the captured frame of image data.

16. The method of claim 3, wherein the combining in step (d) comprises:

combining the aligned frame of image data with the given frame of image data to reduce noise content.

17. The method of claim 3, wherein the combining in step (d) comprises:

temporally filtering the aligned frame of image data with the given frame of image data.

18. The method of claim 3, wherein the repeating in step (e) produces

stabilized output frames of image data for the continuous video stream.

19. The method of claim 9, comprising:

filtering the received frame of image data prior to the correlating.

21. The method of claim 3, wherein the generating in step (b) includes generating the measurement by an inertial motion measurement unit.

22. The method of claim 3, wherein the steps of aligning and combining are performed by a processor module.

26. The method of claim 3, comprising:

after step (e) of repeating steps (a)-(d) the first predetermined number of times, repeating steps (a)-(d) a second predetermined number of times with a different frame as the given frame of image data to process at least one subsequently-captured frame of image data into the stream of produced output frames.

27. The method of claim 26, wherein the different frame is the most recently produced output frame in step (d).

28. The method of claim 26, wherein the first number of times is three and the second number of times is four.

20. A system for processing image data, comprising:

an image capture device configured to capture a frame of image data; and

a processor module coupled to the image capture device, wherein the processor module is configured execute operations comprising;

(a) capturing a frame of image data;

(b) generating a measurement to determine an amount by which the captured frame of image data is offset relative to a given frame of image data;

(c) aligning the captured frame of image data with the given frame of image data to a sub-pixel level in accordance with the determined amount of offset;

(d) combining the aligned frame of image data with the given frame of image data to produce an output frame of image data; and

(e) repeating steps (a)-(d) a first predetermined number of times with the given frame of image data to process at least one subsequently-captured frame of image data into a continuous video stream of output frames produced in step (d).