Method of producing a high quality, high resolution image from a sequence of low quality, low resolution images that are undersampled and subject to jitter

• US 6,650,704 B1
• Filed: 10/25/1999
• Issued: 11/18/2003
• Est. Priority Date: 10/25/1999
• Status: Expired due to Term

• Current Assignee: Nytell Software LLC (Intellectual Ventures LLC)
• Original Assignee: Irvine Sensors Corp.
• Inventors: Carlson, Randolph S., Arnold, Jack L., Feldmus, Valentine G.
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US09/427,384
• Time in Patent Office: 1,485 Days
• Field of Search: H04/N 1.104, 375/240.01, 375/240.18, 375/240.16, 375/240.19, 375/240.11, 382/252, 382/300, 382/270, 382/303, 382/260, 382/284, 382/294, 382/236, 382/299, 348/699, 348/458, 348/409, 348/410, 348/411
• US Class Current: 375/240.01
• CPC Class Codes: G06T 3/4069 by subpixel displacements
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/14 Picture signal circuitry for video frequency region cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard

Abstract

A method of processing low resolution input frames containing undersampled views of an optically imaged scene to produce a higher quality, higher resolution output frame. This method operates by obtaining a sequence of low resolution input frames containing different undersampled views of an optically imaged scene. With regard to each new low resolution input frame, the method involves the further steps of measuring a displacement between a previous low resolution input frame and a new low resolution input frame to sub-pixel precision to produce a measured displacement; coarsely registering a high resolution working frame with the new low resolution input frame based on the measured displacement; finely registering the new low resolution input frame with the high resolution working frame by expanding the new low resolution input frame into a new high resolution input frame based on the measured displacement; and merging the new high resolution input frame into the high resolution working frame to produce an output frame.

Claims

1. A method of processing low resolution input frames containing undersampled views of an optically imaged scene to produce a higher quality, higher resolution output frame comprising the steps of:

obtaining a sequence of low resolution input frames containing different undersampled views of an optically imaged scene; and

for each new low resolution input frame;

measuring a displacement between a previous low resolution input frame and a new low resolution input frame to sub-pixel precision to produce a measured displacement;

coarsely registering a high resolution working frame with the new low resolution input frame based on the measured displacement;

finely registering the new low resolution input frame with the high resolution working frame by expanding the new low resolution input frame into a new high resolution input frame based on the measured displacement; and

merging the new high resolution input frame into the high resolution working frame to produce an output frame.

2. The method of claim 1 wherein the step of measuring a displacement between a previous low resolution input frame and a new low resolution input frame to sub-pixel precision to produce a measured displacement is performed in the X and Y directions only.

3. The method of claim 1 wherein the step of measuring a displacement between a previous low resolution input frame and a new low resolution input frame to sub-pixel precision to produce a measured displacement is accomplished by:

measuring the x gradient of a plurality of pixels in the new low resolution frame;

measuring the y gradient of the plurality of pixels in the new low resolution frame; and

measuring the difference in intensity between the plurality of pixels in the new low resolution frame and the same plurality of pixels in the previous low resolution frame.

4. The method of claim 1 further comprising the step of dividing the measured displacement into an integer displacement portion and a fractional displacement portion.

5. The method of claim 4 wherein the step of coarsely registering a high resolution working frame with the new low resolution input frame based on the measured displacement operates on the basis of the integer displacement portion.

6. The method of claim 5 wherein the step of coarsely registering a high resolution working frame with the new low resolution input frame based on the measured displacement that operates on the basis of the integer displacement portion comprises the substeps of:

providing a high resolution working frame; and

coarsely registering the high resolution working frame with the new low resolution input frame based on the integer displacement portion of the measured displacement.

7. The method of claim 4 wherein the step of finely registering the new low resolution input frame with the high resolution working frame by expanding the new low resolution input frame into a new high resolution input frame based on the measured displacement operates on the basis of the fractional displacement portion.

8. The method of claim 7 wherein the step of finely registering the new low resolution input frame with the high resolution working frame by expanding the new low resolution input frame into a new high resolution input frame based on the measured displacement that operates on the basis of the fractional displacement portion comprises the steps of:

providing a kernel;

fractionally adjusting the kernel based on the fractional displacement value to form a fractionally-shifted kernel; and

finely registering the new low resolution input frame with the high resolution working frame by expanding the new low resolution input frame into a new high resolution input frame using the fractionally-shifted kernel.

9. The method of claim 1 comprising the further steps of:

filtering the measured displacements associated with each new low resolution input frame to produce filter values that follow a smooth trajectory while removing jitter; and

de-shifting the output frames in accordance with the filter values.

10. The method of claim 1 wherein the low resolution input frames are cropped from higher resolution source frame.

11. The method of claim 1 wherein the output frames are resampled to form display frames of a different resolution.

12. A method of processing low resolution input frames containing undersampled views of an optically imaged scene to produce a higher quality, higher resolution output frames comprising the steps of:

obtaining a previous low resolution input frame containing a previous undersampled view of an optically imaged scene;

obtaining a new low resolution input frame containing a new undersampled view of the optically imaged scene due that is different from the previous undersampled view;

measuring a displacement between the new and previous low resolution input frames to sub-pixel precision to produce a measured displacement;

dividing the measured displacement into an integer displacement portion and a fractional displacement portion;

providing a high resolution working frame;

coarsely registering the high resolution working frame with the new low resolution input frame based on the integer displacement portion of the measured displacement;

providing a kernel;

fractionally adjusting the kernel based on the fractional displacement value to form a fractionally-shifted kernel;

finely registering the new low resolution input frame with the high resolution working frame by expanding the new low resolution input frame into a new high resolution input frame using the fractionally-shifted kernel; and

merging the new high resolution input frame into the high resolution working frame.

13. The method of claim 12 wherein the step of coarsely aligning the high resolution working frame with the new low resolution input frame based on the integer displacement portion of the measured displacement is accomplished by integer shifting the high resolution working frame based on the integer displacement value in order to move the high resolution working frame to within a fraction of a high resolution pixel of the new input frame.