High resolution image reconstruction

• US 7,596,284 B2
• Filed: 07/16/2003
• Issued: 09/29/2009
• Est. Priority Date: 07/16/2003
• Status: Active Grant

• Current Assignee: Dolby International AB (Dolby Laboratories Incorporated)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Samadani, Ramin
• Primary Examiner(s): Bella; Matthew C.
• Assistant Examiner(s): PERUNGAVOOR, SATHYANARAYA V
• Application Number:
  US10/620,937
• Publication Number:
  US 20050013509A1
• Time in Patent Office: 2,267 Days
• Field of Search: 382/299
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4023 based on decimating pixels or lines of pixels; based on inserting pixels or lines of pixels

Abstract

A technique of reconstructing a high resolution image from at least one image sequence of temporally related high and low resolution image frames wherein each of said high resolution image frames includes a low spatial frequency component and a high spatial frequency component is described. The high-resolution image reconstruction technique uses spatial interpolation to generate a low spatial frequency component from a low-resolution image frame of the image sequence. The technique is adapted to generate a high spatial frequency component from at least one high resolution image frame of the image sequence which is closely related to the low resolution image frame, and to remap the high spatial frequency component to a motion-compensated high spatial frequency component estimate of the low resolution image frame. The motion-compensated high spatial frequency component estimate is added to the generated low spatial frequency component to form a reconstructed high-resolution image of the low-resolution image frame.

Claims

1. A system for reconstructing a high resolution image from at least one image sequence of temporally related high and low resolution image frames, each of said high resolution image frames including a low spatial frequency component and a high spatial frequency component, said system comprising:

a tangible memory storing instructions; and

a processor coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising

generating a low spatial frequency component from a low resolution image frame of said at least one image sequence;

generating a high spatial frequency component from at least one high resolution image frame of said at least one image sequence, said at least one high resolution image frame being closely related to said low resolution image frame;

mapping said high spatial frequency component to a motion-compensated high spatial frequency component estimate of said low resolution image frame;

adding said motion-compensated high spatial frequency component estimate of said low resolution image frame to said generated low spatial frequency component of said low resolution image frame to form a reconstructed high resolution image of said low resolution image frame; and

controlling relative contributions of said motion-compensated high spatial frequency component estimate of said low resolution image frame and said generated low spatial frequency component of said low resolution image frame in the reconstructed high resolution image of said low resolution image frame based on measures of confidence in motion estimates used to map said high spatial frequency component to the motion-compensated high spatial frequency component estimate of said low resolution image frame.

2. The high resolution image reconstruction system of claim 1, wherein in the generating of the low spatial frequency component the processor performs operations comprising upsampling the low resolution image frame in accordance with a bicubic upsampling algorithm.

3. The high resolution image reconstruction system of claim 1, wherein in the generating of the low spatial frequency component the processor performs operations comprising upsampling the low resolution image frame in accordance with a bilinear upsampling algorithm.

4. The high resolution image reconstruction system of claim 1, wherein in the generating of the low spatial frequency component the processor performs operations comprising upsampling the low resolution image frame in accordance with a least squares error minimization algorithm.

5. The high resolution image reconstruction system of claim 1, wherein in the generating of the high spatial frequency component the processor performs operations comprising downsampling at least one high resolution image frame of said at least one image sequence.

6. A system for reconstructing a high resolution image from at least one image sequence of temporally related high and low resolution image frames, each of said high resolution image frames including a low spatial frequency component and a high spatial frequency component, said system comprising:

a tangible memory storing instructions; and

a processor coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising

generating a low spatial frequency component from a low resolution image frame of said at least one image sequence;

generating a high spatial frequency component from at least one high resolution image frame of said at least one image sequence, said at least one high resolution image frame being closely related to said low resolution image frame, wherein in the generating of the high spatial frequency component the processor performs operations comprising downsampling at least one high resolution image frame of said at least one image sequence and a subpixel motion processor for generating a motion vector field and a confidence scalar field from said downsampled high resolution image frame and said low resolution image frame of said at least one image sequence;

mapping said high spatial frequency component to a motion-compensated high spatial frequency component estimate of said low resolution image frame; and

adding said motion-compensated high spatial frequency component estimate of said low resolution image frame to said generated low spatial frequency component of said low resolution image frame to form a reconstructed high resolution image of said low resolution image frame.

7. The high resolution image reconstruction system of claim 6, wherein in the generating of the high spatial frequency component the processor performs operations comprising generating a low spatial frequency component from said downsampled high resolution image frame.

8. The high resolution image reconstruction system of claim 7, wherein in the generating of the high spatial frequency component the processor performs operations comprising subtracting said generated low spatial frequency component from said at least one high resolution image frame of said at least one image sequence to obtain a high spatial frequency component of said at least one high resolution image frame of said at least one image sequence.

9. A method of reconstructing a high-resolution image from at least one image sequence of temporally related high and low resolution image frames, each of said high-resolution image frames including a low spatial frequency component and a high spatial frequency component, the method comprising operating a processor to perform operations comprising:

spatially interpolating to generate a low spatial frequency component from a low-resolution image frame of said at least one image sequence;

generating a high spatial frequency component from at least one high resolution image frame of said at least one image sequence, said at least one high resolution image frame being closely related to said low resolution image frame;

remapping said high spatial frequency component to a motion-compensated high spatial frequency component estimate of said low resolution image frame;

adding said motion-compensated high spatial frequency component estimate of said low resolution image frame to said generated low spatial frequency component of said low resolution image frame to form a reconstructed high resolution image of said low resolution image frame; and

controlling relative contributions of said motion-compensated high spatial frequency component estimate of said low resolution image frame and said generated low spatial frequency component of said low resolution image frame in the reconstructed high resolution image of said low resolution image frame based on measures of confidence in motion estimates used to map said high spatial frequency component to the motion-compensated high spatial frequency component estimate of said low resolution image frame.

10. The method of claim 9, wherein said spatially interpolating is performed by bicubic upsampling.

11. The method of claim 9, wherein said spatially interpolating is performed by bilinear upsampling algorithm.

12. The method of claim 9, wherein said spatially interpolating is performed by utilizes a least squares error minimization algorithm.

13. The method of claim 9, wherein said high spatial frequency component generating further comprises downsampling at least one high resolution image frame of said at least one image sequence.

14. A method of reconstructing a high-resolution image from at least one image sequence of temporally related high and low resolution image frames, each of said high-resolution image frames including a low spatial frequency component and a high spatial frequency component, the method comprising operating a processor to perform operations comprising:

spatially interpolating to generate a low spatial frequency component from a low-resolution image frame of said at least one image sequence;

generating a high spatial frequency component from at least one high resolution image frame of said at least one image sequence, said at least one high resolution image frame being closely related to said low resolution image frame, wherein said high spatial frequency component generating further comprises downsampling at least one high resolution image frame of said at least one image sequence and subpixel motion processing for the purpose of generating a motion vector field and a confidence scalar field from said downsampled high resolution image frame and said low resolution image frame of said at least one image sequence;

remapping said high spatial frequency component to a motion-compensated high spatial frequency component estimate of said low resolution image frame; and

adding said motion-compensated high spatial frequency component estimate of said low resolution image frame to said generated low spatial frequency component of said low resolution image frame to form a reconstructed high resolution image of said low resolution image frame.

15. The method of claim 14, wherein the generating of said high spatial frequency component further comprises spatially interpolating to generate a low spatial frequency component from said downsampled high-resolution image frame.

16. The method of claim 15, wherein the generating of said high spatial frequency component further comprises subtracting said generated low spatial frequency component from said at least one high resolution image frame of said at least one image sequence to obtain a high spatial frequency component of said at least one high resolution image frame of said at least one image sequence.

17. A system for reconstructing a high resolution image from a mixed spatial resolution image sequence comprising temporally spaced-apart image frames having different respective spatial resolutions, the system comprising:

a tangible memory storing instructions; and

a processor coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising

generating a first spatial frequency component from a first image frame selected from the image sequence, the first image frame having a first spatial resolution;

generating a second spatial frequency component from a second image frame selected from the image sequence, the second image frame having a second spatial resolution higher than the first spatial resolution;

spatially aligning the first and second spatial frequency components based on estimates of motion between the first and second image frames; and

producing a high resolution image of the first image frame from a combination of the spatially aligned first and second spatial frequency components, wherein the combiner produces the high resolution image of the first image frame from the spatially aligned first and second spatial frequency components with relative contributions controlled by measures of confidence in motion estimates used to spatially align the first and second spatial frequency components.

18. The system of claim 17, wherein in the producing the processor performs operations comprising the high resolution image of the first image frame with relative contributions from the second frequency component that increase with increasing measures of confidence in the associated motion estimates.

19. A method of reconstructing a high resolution image from a mixed spatial resolution image sequence comprising temporally spaced-apart image frames having different respective spatial resolutions, the method comprising operating a processor to perform operations comprising:

generating a first spatial frequency component from a first image frame selected from the image sequence, the first image frame having a first spatial resolution;

generating a second spatial frequency component from a second image frame selected from the image sequence, the second image frame having a second spatial resolution higher than the first spatial resolution;

spatially aligning the first and second spatial frequency components based on estimates of motion between the first and second image frames; and

producing a high resolution image of the first image frame from a combination of the spatially aligned first and second spatial frequency components, wherein the producing comprises combining the spatially aligned first and second spatial frequency components based on measures of confidence in the estimates of motion.