Video processing system and method for automatic enhancement of digital video

• US 7,502,522 B2
• Filed: 04/28/2006
• Issued: 03/10/2009
• Est. Priority Date: 06/28/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Soupliotis, Andreas, Anandan, Padmanabhan
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US11/380,728
• Publication Number:
  US 20060187346A1
• Time in Patent Office: 1,047 Days
• Field of Search: 382/103, 382/107, 382/215, 382/218, 382/254, 382260-264, 382/275, 382/276, 382/284, 348/169, 348/208.6, 348/208.99, 348/222.1, 348/239, 348/241, 348/252, 348/345, 348/348, 348/607, 348/625, 348/700, 250/332, 250/347, 250/208.1
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/012 Conversion between an interlaced and a progressive signal for graphics images G09G2310/0229

Abstract

An automatic video enhancement system and method for automatically enhancing video. The automated video enhancement method uses frame-to-frame motion estimation as the basis of the video enhancement. Motion estimation includes the computation of global motion (such as camera motion) and the computation of local motion (such as pixel motion). The automated video enhancement method includes generating global alignment transforms, generating optic flow vectors, and using these global alignment transforms and optic flow vectors to enhance the video. The invention also includes video processing and enhancement techniques that use the frame-to-frame motion estimation. These techniques include a deinterlace process, a denoise process, and a warp stabilization process that performs both damped and locked stabilization.

Claims

1. A computer-readable medium having computer-executable instructions for discovering blurry frames of a digital video, comprising:

inputting a frame from the digital video;

computing a Laplacian energy of the frame and neighboring frames surrounding the frame;

examining the Laplacian energy for the frame and the neighboring frames to determine whether the frame has an abrupt low energy value; and

designating the frame as blurry if the frame has an abrupt low energy value as compared to the neighboring frames.

2. The computer-readable medium as set forth in claim 1, further comprising:

ranking each of the frames according to a Laplacian energy of each frame.

3. The computer-readable medium as set forth in claim 2, further comprising discarding frames below approximately a tenth percentile representing a set of blurry frames.

4. The computer-readable medium as set forth in claim 3, further comprising synthesizing replacement frames for each of the discarded frames by using optic flow interpolation.

5. A method for removing blur from a video containing video frames, comprising:

using a computing device to perform the method of;

processing the video to obtain a denoised, deinterlaced, stabilized, full-frame video;

discovering a blurry frame within the video by examining a Laplacian energy of each frame of the video frame;

discarding the blurry frame;

obtaining a first frame that is previous in time to the blurry frame and not blurred;

obtaining a second frame that is later in time to the blurry frame and not blurred;

using optic flow interpolation to synthesize the new frame from the first frame and the second frame; and

replacing the blurry frame with the new frame.

6. The method as set forth in claim 5, further comprising examining a Laplacian energy of frames neighboring the blurry frame.

7. The method as set forth in claim 6, further comprising ranking the blurry frame and the frames neighboring the blurry frame based on the Laplacian energy of each frame.

8. The method as set forth in claim 7, further comprising discarding a frame that is in an approximately lower ten percent of the ranked frames.

9. A method for processing a digital video containing frames, comprising:

using a computing device to perform the process of;

inputting the digital video containing visual defects;

computing global alignment affine transforms at each of the frames;

computing optic flow vectors using the global alignment affine transforms and the digital video;

performing a motion-compensated deinterlacing on the digital video using the optic flow vectors;

denoising the digital video using a temporal median filter with flow outlier rejection;

stabilizing the digital video using the global alignment affine transforms;

filling in any missing pixels in a frame by borrowing pixels from neighboring frames to stitch together a mosaic; and

discarding blurry frames and replacing each discarded frame by interpolating between a previous frame and a subsequent frame of the discarded frame.

10. The method as set forth in claim 9, further comprising performing color correction on the digital video.

11. The method as set forth in claim 9, further comprising applying a linear sharpening filter to each of the frames.