SPATIAL AND TEMPORAL ALIGNMENT OF VIDEO SEQUENCES

• US 20110116767A1
• Filed: 01/25/2011
• Published: 05/19/2011
• Est. Priority Date: 11/02/2005
• Status: Active Grant

• Current Assignee: Christophe Souchard
• Original Assignee: Christophe Souchard
• Inventors: Souchard, Christophe
• Granted Patent: US 9,530,220 B2
• Time in Patent Office: Days
• US Class Current: 386/278
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.). Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a “first” image) of a first video sequence and another image (i.e., a “second” image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.

Claims

1-20. -20. (canceled)

21. A computer readable medium storing a computer program for a video editing application, the computer program for execution by a processor, the computer program comprising sets of instructions for:

aligning an underexposed first video image and a second video image;

generating a monochrome image of the first video image; and

blending the first and second images by using the monochrome image as a mask to generate a high dynamic range video image.

22. The computer readable medium of claim 21, wherein the computer program further comprises a set of instructions for transforming, before aligning the first and second images, one of said first and second images.

23. The computer readable medium of claim 22, wherein the set of instructions for aligning the first and second images comprises a set of instructions for aligning the transformed image to the non-transformed image.

24. The computer readable medium of claim 21, wherein the set of instructions for generating a monochrome image of the first image comprises a set of instructions for applying a low pass filter to the underexposed first image to create a blurred single color version of the first image.

25. The computer readable medium of claim 21, wherein the second image is overexposed.

26. The computer readable medium of claim 21, wherein the dynamic range is a range between a lightest region and a darkest region of an image.

27. The computer readable medium of claim 21, wherein the set of instructions for blending comprises a set of instructions for changing a blend factor that weights pixel values of the first and second video images to adjust an exposure level of the high dynamic range video image.

28. The computer readable medium of claim 21, wherein the set of instructions for generating a monochrome image of the first image comprises a set of instructions for applying, to the underexposed image, a low pass filter that blocks out high spatial frequency components of the underexposed image.

29. The computer readable medium of claim 21, wherein the computer program further comprises a set of instructions for specifying a mask region to remove an uneven boundary when the first and second images do not completely overlap after aligning the first and second images.

30. The computer readable medium of claim 29, wherein the mask region excludes a plurality of pixels of the first image from generation of the monochrome image.

31. A method comprising:

aligning an underexposed first video image and a second video image;

generating a monochrome image of the first video image; and

blending the first and second images by using the monochrome image as a mask to generate a high dynamic range video image.

32. The method of claim 31 further comprising, before aligning the first and second images, transforming one of said first and second images.

33. The method of claim 32, wherein aligning the first and second images comprises aligning the transformed image to the non-transformed image.

34. The method of claim 31, wherein generating a monochrome image of the first image comprises applying a low pass filter to the underexposed first image to create a blurred single color version of the first image.

35. The method of claim 31, wherein the second image is overexposed.

36. The method of claim 31, wherein the dynamic range is a range between a lightest region and a darkest region of an image.

37. The method of claim 31, wherein blending further comprises changing a blend factor that weights pixel values of the first and second video images to adjust an exposure level of the high dynamic range video image.

38. The method of claim 31, wherein generating a monochrome image of the first image comprises applying, to the underexposed image, a low pass filter that blocks out high spatial frequency components of the underexposed image.

39. The method of claim 31 further comprising specifying a mask region to remove an uneven boundary when the first and second images do not completely overlap after aligning the first and second images.

40. The method of claim 39, wherein the mask region excludes a plurality of pixels of the first image from generation of the monochrome image.