Spatial and temporal alignment of video sequences

• US 20070097266A1
• Filed: 11/02/2005
• Published: 05/03/2007
• Est. Priority Date: 11/02/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Computer Incorporated (Apple Inc.)
• Inventors: Souchard, Christophe
• Granted Patent: US 7,912,337 B2
• Time in Patent Office: Days
• US Class Current: 348/571
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.). Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a “first” image) of a first video sequence and another image (i.e., a “second” image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.

Claims

1. A method for editing video, said method comprising:

a) selecting a first set of video images through a video editing application;

b) selecting a second set of video images through the video editing application; and

c) aligning the first and second sets of video images to produce a third set of video images through the video editing application, wherein the third set of video images is a set of panoramic video images.

2. The method of claim 1, wherein before aligning the first and second sets of video images, transforming the first set of video images.

3. The method of claim 2, wherein transforming the first set of images comprises:

a) defining an objective function;

b) optimizing the objective function to compute a motion function; and

c) applying the motion function to the first set of images to transform the first set of images,

4. The method of claim 3, wherein the objective function expresses the difference between two frames.

5. The method of claim 3, wherein optimizing the objective function comprises minimizing the objective function based on a set of constraints.

6. The method of claim 1, wherein the selected first and second video images comprises selecting a set of pixels in the first and second video images.

7. The method of claim 1, wherein selecting the set of pixels comprises selecting the set of pixels that include a spatial frequency above a particular value.

8. The method of claim 1, wherein aligning the first and second sets of video images comprises:

aligning a first video image in the first set of images to a first video image in the second set of images,

wherein the first set of images comprises a plurality of images that are arranged in a first sequence,

wherein the second set of images comprises a plurality of images that are arranged in a second sequence,

wherein the position of the first video image in the first sequence is different than the position of the first video image in the second sequence.

9. The method of claim 8, wherein aligning the first and second sets of video images further comprises:

aligning a second video image in the first set of images to a second video image in the second set of images,

wherein the position of the second video image in second sequence is before the position of the first image in the second sequence.

10. The method of claim 9, wherein the position of the second image in the first sequence is before the position of the first image in the first sequence.

11. The method of claim 8, wherein aligning the first and second sets of video images further comprises:

aligning a second video image in the first set of images to a second video image in the second set of images,

wherein the position of the second video image in the first sequence is after the position of the first image in the first sequence.

12. The method of claim 11, wherein the position of the second image in the second sequence is after the position of the first image in the second sequence.

13. The method of claim 2 further comprising specifying a mask before transforming the first set of video images.

14. The method of claim 13, wherein the mask excludes a plurality of pixels in the first set of video images from being transformed.

15. A method for editing video, said method comprising:

a) selecting a first set of video images through a video editing application;

b) selecting a second set of video images through the video editing application; and

c) aligning the first and second sets of video images to produce a third set of video images through the video editing application, wherein the third set of video images is a set of high dynamic range video images.

16. The method of claim 15, wherein before aligning the first and second sets of video images, transforming the first set of video images.

17. The method of claim 16, wherein transforming the first set of video images comprises:

a) computing a monochrome transform of the first set of video images; and

b) computing a low pass transform of the first set of video images, wherein the low pass transform excludes a set of pixels that have a spatial frequency above a particular value in the first set of video images.

18. The method of claim 17, wherein aligning the first and second sets of video images comprises aligning the transformed first set of video images to the second set of video images.

19. The method of claim 15, wherein the first of video images is underexposed.

20. The method of claim 15, wherein the second set of video images is overexposed.