Video processing system and method for automatic enhancement of digital video

• US 7,119,837 B2
• Filed: 06/28/2002
• Issued: 10/10/2006
• Est. Priority Date: 06/28/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Soupliotis, Andreas, Anandan, Padmanabhan
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US10/186,562
• Publication Number:
  US 20040001705A1
• Time in Patent Office: 1,565 Days
• Field of Search: 348/208.99
• US Class Current: 348/208.99
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/012 Conversion between an interlaced and a progressive signal for graphics images G09G2310/0229

Abstract

The present invention includes an automatic video enhancement system and method for automatically enhancing video. The automated video enhancement method uses frame-to-frame motion estimation as the basis of the video enhancement. Motion estimation includes the computation of global motion (such as camera motion) and the computation of local motion (such as pixel motion). The automated video enhancement method includes generating global alignment transforms, generating optic flow vectors, and using these global alignment transforms and optic flow vectors to enhance the video. The invention also includes video processing and enhancement techniques that use the frame-to-frame motion estimation. These techniques include a deinterlace process, a denoise process, and a warp stabilization process that performs both damped and locked stabilization.

Claims

1. A method for automatically enhancing a video having video frames and captured by a camera, comprising:

performing global motion computation to determine the motion of the camera and generate global alignment transforms;

performing local motion computation using the global alignment transforms to determine motion of each pixel in each of the video frames and generate optic flow vectors; and

performing denoising of the video using the optic flow vectors and a temporal median filter having flow outlier rejection.

2. The method as set forth in claim 1, wherein performing global motion computation further comprises computing pairwise affine transformations between the video frames.

3. The method as set forth in claim 2, wherein the global alignment transforms are global alignment affine transforms.

4. The method as set forth in claim 1, further comprising using the global alignment transforms to cancel out camera motion.

5. The method as set forth in claim 1, wherein performing local motion computation comprises using the video and the global alignment transforms to compute the optic flow vectors between each of the video frames.

6. The method as set forth in claim 1, further comprising performing motion-compensated deinterlacing using the video and the optic flow vectors.

7. The method as set forth in claim 1, further comprising performing warp stabilization using the global alignment transform.

8. The method as set forth in claim 7, further comprising:

canceling noisy pixels within a video frame; and

stitching a mosaic using pixels from nearby video frames to fill in the canceled noisy pixels.

9. The method as set forth in claim 1, wherein performing denoising of the video further comprises:

determining erroneous optic flow vectors by warping a given frame forward along an optic flow vector;

determining an accuracy score for each pixel signifying how close of a match there is between the warped frame and the next frame in time;

classifying each pixel based on the accuracy score.

10. The method as set forth in claim 9, further comprising:

classifying a first set of pixels as inaccurate pixels;

classifying a second set of pixels as accurate pixels;

performing temporal median filtering on the inaccurate pixels; and

performing motion-compensated temporal median filtering on the accurate pixels.

11. A computer-readable medium having computer-executable instructions for performing the method recited in claim 1.

12. A method for processing a video having a video frame sequence, comprising:

computing global alignment affine transforms using the video frame sequence to reduce camera motion;

computing optic flow vectors using the global alignment affine transforms and the video frame sequence; and

performing deinterlacing on the video frame sequence using the optic flow vectors and a temporal median filter with flow outlier rejection to generate a denoised, deinterlaced video.

13. The method as set forth in claim 12, further comprising performing a warp stabilization on the denoised, deinterlaced video.

14. The method as set forth in claim 13, wherein the warp stabilization is a locked stabilization.

15. The method as set forth in claim 14, further comprising performing image registration using the global alignment affine transforms.

16. The method as set forth in claim 13, wherein the warp stabilization is a damped stabilization to damp out jitter motion in the denoised, deinterlace video.

17. The method as set forth in claim 16, further comprising:

determining an actual motion of a camera that captured the video;

applying a smoothing operation to determine intended camera motion; and

subtracting the actual camera motion and the intended camera motion to determine the jitter motion.

18. The method as set forth in claim 13, further comprising building mosaics to replace pixels discarded due to the warp stabilization.

19. A method for processing a video containing a video frame sequence, comprising:

performing global motion computation to using an affine transformation to generate global alignment affine transforms;

performing warp stabilization on the video frame sequence using the global alignment affine transforms;

wherein performing warp stabilization further comprises;

discovering any blurry frames within the video frame sequence;

discarding the blurry frames;

synthesizing new frames using the global alignment affine transforms; and

replacing the discarded blurry frames with the new frames.

20. A method for processing a video containing a video frame sequence, comprising:

performing global motion computation to using an affine transformation to generate global alignment affine transforms;

performing warp stabilization on the video frame sequence using the global alignment affine transforms

using the global alignment affine transforms to determine moving pixels in a video frame of the video frame sequence; and

discarding the moving pixels within the video frame.

21. The method as set forth in claim 20, further comprising filling in the discarded moving pixels within the video frame with pixels borrowed from frames neighboring the video frame.

22. A method for processing a video containing a video frame sequence, comprising:

performing global motion computation to using an affine transformation to generate global alignment affine transforms;

performing warp stabilization on the video frame sequence using the global alignment affine transforms, wherein the warped stabilization is a damped stabilization, and performing the warp stabilization further comprises;

determining a curve corresponding to an actual motion of a camera that captured the video;

applying a smoothing operation to the actual camera motion to yield an intended camera motion; and

determining the difference between the intended camera motion and the actual camera motion to recover camera jitter.

23. A method for processing a video containing a video frame sequence, comprising:

performing global motion computation to using an affine transformation to generate global alignment affine transforms;

performing warp stabilization on the video frame sequence using the global alignment affine transforms, wherein the warned stabilization is a locked stabilization, and performing the warp stabilization further comprises;

determining a video frame from the video frame sequence to use as a reference frame; and

canceling out moving pixels in a video frame discovered by the global alignment affine transforms.

24. The method as set forth in claim 23, wherein the reference frame is a first frame of the video frame sequence.

25. The method as set forth in claim 23, further comprising filling in the canceled pixels by building a mosaic.

26. The method as set forth in claim 23, wherein building a mosaic further comprises using similar pixels from nearby video frames to fill in the canceled pixels.