Video enhancement

• US 20060257042A1
• Filed: 05/13/2005
• Published: 11/16/2006
• Est. Priority Date: 05/13/2005
• Status: Active Grant

• Current Assignee: Oxbotica Limited
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Matsushita, Yasuyuki
• Granted Patent: US 7,548,659 B2
• Time in Patent Office: Days
• US Class Current: 382/255
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 5/005 Retouching; Inpainting; Scratch removal detecting, correction, reducing or removing defects, e.g. non-responsive pixels of solid state image sensors H04N25/68, scratch removal for cinematographic films scanned by electronic image sensor H04N5/253
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations

Abstract

Video enhancement enable visually-apparent improvements to image frames of a video sequence. In an example implementation, stabilized or other image frames include missing image areas that are filled using a video completion technique. The missing image areas are filled in using a video completion technique that involves motion inpainting. Local motion data of a motion map from known image areas is propagated into the missing image areas to produce an extended motion map. Pixels of the missing image areas are then filled in by warping pixels from neighboring image frames into the missing image areas responsive to the extended motion map. In another example implementation, video deblurring reduces the blurriness of a current image frame. Sharper image pixels from neighboring image frames are transferred so as to replace blurry pixels of the current image frame. In yet another example implementation, video completion with motion inpainting enables amelioration of visual artifacts.

Claims

1. A method comprising:

propagating motion data of a motion map from a known image area of a current image frame to one or more missing image areas to produce an extended motion map; and

warping pixels from at least one neighboring image frame to the one or more missing image areas of the current image frame responsive to the extended motion map.

2. The method as recited in claim 1, further comprising:

determining multiple respective alignment errors between the current image frame and respective ones of multiple neighboring image frames;

selecting a particular respective neighboring image frame having a lowest respective alignment error to be the at least one neighboring image frame used in the warping; and

computing local motion with respect to the at least one neighboring image frame using common coverage areas between the current image frame and the at least one neighboring image frame;

wherein the computed local motion comprises the motion data that is propagated in the propagating.

3. The method as recited in claim 1, further comprising:

warping multiple pixels from multiple neighboring image frames to a corresponding pixel of the one or more missing image areas of the current image frame;

determining if the warped multiple pixels meet a consistency constraint;

if the warped multiple pixels are determined to meet a consistency constraint, accepting a value of the warped multiple pixels from a compatible frame as a value for the corresponding pixel; and

if the warped multiple pixels are not determined to meet a consistency constraint, performing the propagating motion data and the warping pixels for the corresponding pixel.

4. The method as recited in claim 1, further comprising:

estimating global motion for multiple image frames of a video sequence, the video sequence including the current image frame and the at least one neighboring image frame; and

smoothing the global motion of the multiple image frames by smoothing respective individual local displacements of respective individual ones of the multiple image frames.

5. The method as recited in claim 4, wherein the smoothing comprises:

computing a smoothing transformation for a given image frame to a motion-compensated image frame using transformation matrices of image frames that are neighbors of the given image frame while excluding transformation matrices of non-neighboring image frames.

6. The method as recited in claim 1, wherein the propagating comprises:

extrapolating the motion data by processing pixels of the one or more missing image areas in an ascending distance order away from an initial boundary between the known image area and the one or more missing image areas.

7. The method as recited in claim 1, wherein the propagating comprises:

producing the extended motion map using, for each individual pixel of the one or more missing image areas, a weighted average of a set of motion vectors that are associated with a set of pixels that comprise neighboring pixels for each individual pixel.

8. The method as recited in claim 7, wherein the weighted average reflects motion similarity by being based on respective similarities of color or intensity between each individual pixel and respective pixels of the set of pixels.

9. A method comprising:

determining blurriness measures for a current image frame and a predetermined number of neighboring image frames;

selecting, based on the ascertained blurriness measures, multiple neighboring image frames that are sharper than the current image frame; and

sharpening the current image frame using corresponding pixels from the selected multiple neighboring image frames.

10. The method as recited in claim 9, wherein the determining comprises:

ascertaining relative blurriness measures that can indicate if a particular neighboring image frame of the predetermined number of neighboring image frames is more or less sharp than the current image frame.

11. The method as recited in claim 9, wherein the determining comprises:

ascertaining respective blurriness measures with regard to respective common coverage areas between the current image frame and respective ones of the predetermined number of neighboring image frames.

12. The method as recited in claim 9, wherein the sharpening comprises:

sharpening the current image frame by transferring and interpolating corresponding pixels from the selected multiple neighboring image frames.

13. The method as recited in claim 12, wherein the sharpening comprises:

interpolating the corresponding pixels from the selected multiple neighboring image frames responsive to a weighting factor.

14. The method as recited in claim 13, wherein the weighting factor is based on at least one of (i) a pixel-wise alignment error between the current image frame and a respective one of the selected multiple neighboring image frames or (ii) a relative blurriness between the current image frame and the respective one of the selected multiple neighboring image frames.

15. One or more processor-accessible media including processor-executable instructions comprising a video enhancement module that is capable of ameliorating visual artifacts from a video sequence;

the video enhancement module adapted to remove a visual artifact from an image frame to thereby create a missing image area;

the video enhancement module further adapted to fill in pixels of the missing image area by performing a video completion process that includes motion inpainting.

16. The one or more processor-accessible media as recited in claim 15, wherein the motion inpainting involves extrapolation of motion data to the pixels of the missing image area from pixels located in another area of the image frame.

17. The one or more processor-accessible media as recited in claim 16, wherein the video completion process further includes pixel warping from at least one other image frame of the video sequence responsive to the extrapolated motion data.

18. The one or more processor-accessible media as recited in claim 15, wherein the video enhancement module is further capable of deblurring the video sequence by replacing relatively less sharp pixels of the image frame with relatively more sharp pixels of one or more other image frames of the video sequence.

19. The one or more processor-accessible media as recited in claim 15, wherein the visual artifact comprises at least one of:

(i) a mark resulting from an object on a lens of a video camera, (ii) a mark resulting from a defect of a charge coupled device (CCD) of a video camera, or (iii) alphanumeric characters and/or a logo that overlays a portion of the image frame.

20. The one or more processor-accessible media as recited in claim 15, wherein the video enhancement module is further adapted (i) to identify the visual artifact automatically based on one or more characteristics of pixels that form the visual artifact and/or (ii) to identify the visual artifact responsive to manual input indicating a pixel area of the visual artifact.