Enhancing the resolution and quality of sequential digital images

• US 8,279,341 B1
• Filed: 02/26/2007
• Issued: 10/02/2012
• Est. Priority Date: 02/26/2007
• Status: Active Grant

• Current Assignee: Cubic Corporation
• Original Assignee: MotionDSP, Inc. (Cubic Corporation)
• Inventors: Bozinovic, Nikola
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): TEITELBAUM, MICHAEL E
• Application Number:
  US11/678,998
• Time in Patent Office: 2,045 Days
• Field of Search: None
• US Class Current: 348/452
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

In particular embodiments, a process relating to the creation of a high-resolution video from a low-resolution video. In a particular embodiment, the process receives as input a sequence of low-resolution video frames. The process first determines a matching score for consecutive frames in the sequence, where the matching score is based on a preliminary global transformation between consecutive frames. From the matching scores, the process determines a set of matching windows and relative-motion estimates and then uses the set and estimates to calculate more thorough global transformations and any residual relative-motion which can be explained using independent object motion and/or optical flow. The process uses the latter global transformation and any independent object motion and/or optical flow to create motion trajectories and to generate high-resolution frames by interpolating low-resolution frames at trajectory-defined points, using regular-to-irregular spatio-temporal interpolation. The process then restores the frames, using deblurring, and outputs them.

Claims

1. A method, comprising:

receiving as input a sequence of two or more low resolution video frames, wherein one of the low resolution video frames is a current frame;

calculating a global transformation, relative to the current frame, for the sequence of video frames;

determining whether unexplained relative motion remains following calculation of the global transformation;

responsive to identifying the unexplained relative motion, calculating an independent object motion, relative to the current frame, for the sequence of video frames;

determining whether unexplained relative motion remains following calculation of the global transformation and any independent object motion;

responsive to identifying the unexplained relative motion, calculating an optical flow, relative to the current frame, for the sequence of video frames;

creating pixel motion trajectories, relative to the current frame, for the sequence of video frames, based on the calculated global transformation, any calculated independent object motion and any calculated optical flow;

interpolating an enhanced resolution frame for the current frame from the pixel motion trajectories using spatio-temporal interpolation; and

outputting the enhance resolution frame.

2. The method of claim 1, further comprising the step of restoring the enhanced resolution frame.

3. The method of claim 1, wherein the spatio-temporal interpolation comprises regular-to-irregular spatial interpolation.

4. The method of claim 1, wherein a subset of the sequence of low resolution video frames are used for the calculation of global transformation, any calculated independent object motion, any calculated optical flow, and the creation of pixel motion trajectories.

5. The method of claim 1, wherein the calculation of the global transformation uses an eight-parameter projective motion model.

6. The method of claim 1, wherein the calculation of independent object motion uses an eight-parameter projective motion model.

7. The method of claim 2, wherein the restoration comprises deblurring.

8. The method claim 7, wherein the deblurring is based on a point spread function whose description is stored in a system library.

9. One or more non-transitory computer-readable media comprising computer-readable instructions for execution and when executed operable to cause one or more processors to:

receive as input a sequence of two or more low resolution video frames, wherein one of the low resolution video frames is a current frame;

calculate a global transformation, relative to the current frame, for the sequence of video frames;

determine whether unexplained relative motion remains following calculation of the global transformation;

responsive to identifying the unexplained relative motion, calculate an independent object motion, relative to the current frame, for the sequence of video frames;

determine whether unexplained relative motion remains following calculation of the global transformation and any independent object motion;

responsive to identifying the unexplained relative motion, calculate an optical flow, relative to the current frame, for the sequence of video frames;

create pixel motion trajectories, relative to the current frame, for the sequence of video frames, based on the calculated global transformation, any calculated independent object motion and any calculated optical flow;

interpolate an enhanced resolution frame for the current frame from the pixel motion trajectories, using spatio-temporal interpolation; and

output the enhanced resolution video frame.

10. The non-transitory computer-readable media of claim 9, further comprising executable logic to restore the high resolution frame.

11. The non-transitory computer-readable media of claim 9, wherein the spatio-temporal interpolation comprises regular-to-irregular spatial interpolation.

12. The non-transitory computer-readable media of claim 9, wherein a subset of the sequence of low resolution video frames are used for the calculation of global transformation, any calculated independent object motion, any calculated optical flow, and the creation of pixel motion trajectories.

13. The non-transitory computer-readable media of claim 9, wherein the calculation of the global transformation uses an eight-parameter projective motion model.

14. The non-transitory computer-readable media of claim 9, wherein the calculation of independent object motion uses an eight-parameter projective motion model.

15. The non-transitory computer-readable media of claim 10, wherein the restoration comprises deblurring.

16. The non-transitory computer-readable media claim 15, wherein the deblurring is based on a point spread function whose description is stored in a system library.

17. An apparatus, comprising:

a memory;

one or more processors; and

a non-transitory computer readable medium comprising computer readable instructions encoded thereon, the computer-readable instructions operative, when executed, to cause the one or more processors to;

display an enhanced resolution video, one or more frames of the enhanced resolution video output from an enhanced resolution module comprising;

one or more processors, and logic encoded in one or more computer-readable media for execution and when executed operable to;

receive as input a sequence of two or more low resolution video frames;

create pixel motion trajectories for the sequence of video frames by registering the frames;

perform regular-to-irregular spatial interpolation on the registered frames, wherein the results of this spatial interpolation are noisy observations;

perform temporal approximation on the noisy observations, wherein the results of this temporal approximation are high resolution video frames; and

output a sequence of enhanced resolution video frames.