Spatial and temporal alignment of video sequences

• US 9,530,220 B2
• Filed: 01/25/2011
• Issued: 12/27/2016
• Est. Priority Date: 11/02/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Souchard, Christophe
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US13/013,802
• Publication Number:
  US 20110116767A1
• Time in Patent Office: 2,163 Days
• Field of Search: 386/278, 386/280, 386/284
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.). Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a “first” image) of a first video sequence and another image (i.e., a “second” image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.

Claims

1. A non-transitory machine readable medium storing a computer program for a video editing application, the computer program for execution by a processor, the computer program comprising sets of instructions for:

aligning an underexposed first video image and a second video image;

generating a monochrome image of the first video image; and

blending the first and second images by using the monochrome image as a mask to generate a high dynamic range video image.

2. The non-transitory machine readable medium of claim 1, wherein the computer program further comprises a set of instructions for transforming, before aligning the first and second images, one of said first and second images.

3. The non-transitory machine readable medium of claim 2, wherein the set of instructions for aligning the first and second images comprises a set of instructions for aligning the transformed image to the non-transformed image.

4. The non-transitory machine readable medium of claim 1, wherein the set of instructions for generating a monochrome image of the first image comprises a set of instructions for applying a low pass filter to the underexposed first image to create a blurred single color version of the first image.

5. The non-transitory machine readable medium of claim 1, wherein the second image is overexposed.

6. The non-transitory machine readable medium of claim 1, wherein the dynamic range is a range between a lightest region and a darkest region of an image.

7. The non-transitory machine readable medium of claim 1, wherein the set of instructions for blending comprises a set of instructions for changing a blend factor that weights pixel values of the first and second video images to adjust an exposure level of the high dynamic range video image.

8. The non-transitory machine readable medium of claim 1, wherein the set of instructions for generating a monochrome image of the first image comprises a set of instructions for applying, to the underexposed image, a low pass filter that blocks out high spatial frequency components of the underexposed image.

9. The non-transitory machine readable medium of claim 1, wherein the computer program further comprises a set of instructions for specifying a mask region to remove an uneven boundary when the first and second images do not completely overlap after aligning the first and second images.

10. The non-transitory machine readable medium of claim 9, wherein the mask region excludes a plurality of pixels of the first image from generation of the monochrome image.

11. A method comprising:

aligning an underexposed first video image and a second video image, the first and second video images each comprising a plurality of pixel values;

based on at least one of the first and second video images, generating a mask comprising a plurality of grayscale pixel values, each grayscale pixel value corresponding to a first pixel value of the first video image and a second pixel value of the second video image; and

blending the corresponding first and second pixel values of the first and second images based on their corresponding grayscale pixel value to generate a high dynamic range video image.

12. The method of claim 11 further comprising, before aligning the first and second images, transforming one of said first and second images.

13. The method of claim 12, wherein aligning the first and second images comprises aligning the transformed image to the non-transformed image.

14. The method of claim 11, wherein generating a mask comprises applying a low pass filter to the underexposed first image to create a blurred single color version of the first image.

15. The method of claim 11, wherein the second image is overexposed.

16. The method of claim 11, wherein the dynamic range is a range between a lightest region and a darkest region of an image.

17. The method of claim 11, wherein blending further comprises changing a blend factor that weights pixel values of the first and second video images to adjust an exposure level of the high dynamic range video image.

18. The method of claim 11, wherein generating a mask comprises applying, to the underexposed image, a low pass filter that blocks out high spatial frequency components of the underexposed image.

19. The method of claim 11 further comprising specifying a mask region to remove an uneven boundary when the first and second images do not completely overlap after aligning the first and second images.

20. The method of claim 19, wherein the mask region excludes a plurality of pixels of the first image from generation of the mask.