Enhancing the resolution and quality of sequential digital images

• US 8,537,278 B1
• Filed: 09/10/2012
• Issued: 09/17/2013
• Est. Priority Date: 02/26/2007
• Status: Active Grant

• Current Assignee: Cubic Corporation
• Original Assignee: MotionDSP, Inc. (Cubic Corporation)
• Inventors: Bozinovic, Nikola
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): TEITELBAUM, MICHAEL E
• Application Number:
  US13/609,128
• Time in Patent Office: 372 Days
• Field of Search: 348/451, 348/452, 348/448, 348/701
• US Class Current: 348/452
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

In particular embodiments, a process relating to the creation of a high-resolution video from a low-resolution video. In a particular embodiment, the process receives as input a sequence of low-resolution video frames. The process first determines a matching score for consecutive frames in the sequence, where the matching score is based on a preliminary global transformation between consecutive frames. From the matching scores, the process determines a set of matching windows and relative-motion estimates and then uses the set and estimates to calculate more thorough global transformations and any residual relative-motion which can be explained using independent object motion and/or optical flow. The process uses the latter global transformation and any independent object motion and/or optical flow to create motion trajectories and to generate high-resolution frames by interpolating low-resolution frames at trajectory-defined points, using regular-to-irregular spatio-temporal interpolation. The process then restores the frames, using deblurring, and outputs them.

Claims

1. A method, comprising:

receiving as input a sequence of two or more low resolution video frames;

creating pixel motion trajectories for the sequence of video frames by registering the frames, the created pixel motion trajectories being relative to a current frame of the received sequence, and including;

calculating a global transformation, relative to the current frame, for the received sequence of video frames, such that the creating of the pixel motion trajectories is based on the calculated global transformation;

performing spatial interpolation on the registered frames from points that are not on a regular grid to points that are on a regular grid, wherein the results of the spatial interpolation comprise noisy observations on the regular grid;

performing temporal approximation on the noisy observations, wherein the results of the temporal approximation are high resolution video frames; and

outputting a sequence of high resolution video frames.

2. The method of claim 1, further comprising the step of restoring the enhanced resolution frame.

3. The method of claim 1, wherein the spatial interpolation comprises bicubic interpolation.

4. The method of claim 1, wherein the spatial interpolation comprises cubic spline interpolation.

5. The method of claim 1, wherein the spatial interpolation comprises steering kernel regression.

6. The method of claim 1, wherein the temporal approximation is based on geometric and statistical factors.

7. The method of claim 1,

wherein the creating the pixel motion trajectories further includes:

determining any unexplained relative motion remaining after application of the global transformation; and

in the case there is remaining unexplained relative motion, calculating an independent object motion, relative to the current frame, for the sequence of video frames,

and wherein the creating pixel motion trajectories is further based on the calculated independent object motion.

8. The method of claim 7,

wherein the creating the pixel motion trajectories further includes:

determining any further unexplained relative motion remaining after application of the global transformation and any independent object motion;

in the case there is even further remaining unexplained relative motion, calculating an optical flow, relative to the current frame, for the sequence of video frame,

and wherein the creating pixel motion trajectories is further based on any calculated optical flow.

9. The method of claim 1, further comprising the step of restoring the high resolution frames.

10. The method of claim 9, wherein the restoration comprises deblurring.

11. One or more non-transitory computer-readable media comprising computer-readable instructions for execution that when executed by one or more processors cause carrying out a method comprising:

receiving as input a sequence of two or more low resolution video frames;

creating pixel motion trajectories for the sequence of video frames by registering the frames, the creating pixel motion trajectories being relative to a current frame of the received sequence of video frames, and including;

calculating a global transformation, relative to the current frame, for the received sequence of video frames, such that the creating of the pixel motion trajectories is based on the calculated global transformation;

performing spatial interpolation on the registered frames from points that are not on a regular grid to points that are on a regular grid, wherein the results of the spatial interpolation comprise noisy observations on the regular grid;

performing temporal approximation on the noisy observations, wherein the results of the temporal approximation are high resolution video frames; and

outputting a sequence of high resolution video frames.

12. The non-transitory computer-readable media of claim 11, further comprising computer-readable instructions for execution that when executed by one or more processors cause restoring the high resolution frame.

13. The non-transitory computer-readable media of claim 11, wherein the spatial interpolation comprises bicubic interpolation.

14. The non-transitory computer-readable media of claim 11, wherein the spatial interpolation comprises cubic spline interpolation.

15. The non-transitory computer-readable media of claim 11, wherein the spatial interpolation comprises steering kernel regression.

16. The non-transitory computer-readable media of claim 11, wherein the temporal approximation is based on geometric and statistical factors.

17. The non-transitory computer-readable media of claim 11,

wherein the creating the pixel motion trajectories further includes:

determining any unexplained relative motion remaining after application of the global transformation; and

in the case there is remaining unexplained relative motion, calculating an independent object motion, relative to the current frame, for the sequence of video frames,

and wherein the creating pixel motion trajectories is further based on the calculated independent object motion.

18. The non-transitory computer-readable media of claim 17,

wherein the creating the pixel motion trajectories further includes:

determining any further unexplained relative motion remaining after application of the global transformation and of any independent object motion;

in the case there is even further remaining unexplained relative motion, calculating an optical flow, relative to the current frame, for the sequence of video frame,

and wherein the creating pixel motion trajectories is further and based on any calculated optical flow.

19. The non-transitory computer-readable media of claim 11, wherein the method further comprises the step of restoring the high resolution frames.