Spatial and temporal alignment of video sequences

• US 9,998,685 B2
• Filed: 12/18/2016
• Issued: 06/12/2018
• Est. Priority Date: 11/02/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Souchard, Christophe
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US15/382,696
• Publication Number:
  US 20170099442A1
• Time in Patent Office: 541 Days
• Field of Search: 386278, 386280, 386284
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.). Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a “first” image) of a first video sequence and another image (i.e., a “second” image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.

Claims

1. A method of generating a high dynamic range image, the method comprising:

aligning an underexposed first image and a second image;

generating a blurred single color image by applying a low pass filter to the underexposed first image to block out high spatial frequency components of the underexposed first image; and

blending the first and second images by using the blurred single color image as a mask to generate a high dynamic range image.

2. The method of claim 1 further comprising:

defining a blend factor that weights pixel values of the first and second images; and

adjusting an exposure level of the high dynamic range image by changing the blend factor.

3. The method of claim 1, wherein the low pass filter is a Gaussian filter.

4. The method of claim 1 further comprising:

defining a set of transform operations based on a difference between the underexposed first image and the second image,

wherein the aligning comprises applying the set of transform operations to one of the first and second images.

5. The method of claim 4, wherein defining the set of transform operations comprises:

defining a set of motion functions, each motion function for expressing a motion between the underexposed first image and the second image;

defining a set of objective functions based on the set of motion functions;

optimizing the set of objective functions to compute a set of optimal motion functions;

and defining the set of transform operations based on the set of optimal motion functions.

6. The method of claim 5, wherein each objective function in the set of objective functions expresses a difference between the first and second images.

7. The method of claim 5, wherein optimizing the set of objective functions comprises minimizing the set of objective functions based on a set of constraints.

8. The method of claim 1 further comprising specifying a mask region to remove an uneven boundary when the first and second images do not completely overlap after aligning the first and second images.

9. The method of claim 8, wherein the mask region excludes a plurality of pixels of the first image from generation of the monochrome image.

10. The method of claim 1, wherein the second image is an overexposed image.

11. The method of claim 1, wherein the dynamic range is a range between a lightest region and a darkest region of an image.

12. A non-transitory machine readable medium storing a computer program for generating a high dynamic range image, the computer program executable by a processor, the computer program comprising sets of instructions for:

aligning an underexposed first image and a second image by applying a set of transform operations defined based on a difference between the underexposed first image and the second image to one of the first and second images;

generating a blurred single color image by applying a low pass filter to the underexposed first image to block out high spatial frequency components of the underexposed first image; and

blending the first and second images by using the blurred single color image as a mask to generate a high dynamic range image.

13. The non-transitory machine readable medium of claim 12, the program further comprising sets of instructions for:

defining a blend factor that weights pixel values of the first and second images; and

adjusting an exposure level of the high dynamic range image by changing the blend factor.

14. The non-transitory machine readable medium of claim 12, wherein the low pass filter is a Gaussian filter.

15. The non-transitory machine readable medium of claim 12, the program further comprising sets of instructions for:

defining a set of motion functions, each motion function for expressing a motion between the underexposed first image and the second image;

defining a set of objective functions based on the set of motion functions; and

optimizing the set of objective functions to compute a set of optimal motion functions,

wherein the set of transform operations are defined based on the set of optimal motion functions.

16. The non-transitory machine readable medium of claim 15, wherein each objective function in the set of objective functions expresses a difference between the first and second images.

17. The non-transitory machine readable medium of claim 15, wherein the set of instructions for optimizing the set of objective functions comprises a set of instructions for minimizing the set of objective functions based on a set of constraints.

18. The non-transitory machine readable medium of claim 12, the program further comprising a set of instructions for specifying a mask region to remove an uneven boundary when the first and second images do not completely overlap after aligning the first and second images.

19. The non-transitory machine readable medium of claim 18, wherein the mask region excludes a plurality of pixels of the first image from generation of the monochrome image.

20. The non-transitory machine readable medium of claim 12, wherein the second image is an overexposed image.