Smoothing and/or locking operations in video editing

• US 8,750,636 B2
• Filed: 06/20/2011
• Issued: 06/10/2014
• Est. Priority Date: 04/16/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Souchard, Christophe
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): BROOKS, JULIAN D
• Application Number:
  US13/164,614
• Publication Number:
  US 20110311202A1
• Time in Patent Office: 1,086 Days
• Field of Search: 382/107, 382/254, 382/260, 382/275, 382/309, 348/208.99, 348/208.4, 348/208.13, 348/606, 348/607, 348/620, 348/700, 348/701
• US Class Current: 382/254
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Some embodiments allow a video editor to remove unwanted camera motion from a sequence of video images (e.g., video frames). Some embodiments are implemented in a video editing application. Some of these embodiments distinguish unwanted camera motion from the intended underlying motion of a camera (e.g., panning and zooming) and/or motion of objects within the video sequence.

Claims

1. A non-transitory computer readable medium storing a video editing application which when executed by at least one processing unit edits a sequence of video images, the video editing application comprising sets of instructions for:

receiving a selection of a set of video images;

in a graphical user interface (GUI) of the video editing application, displaying a set of at least three movement types that are individually selectable;

concurrently receiving a user selection of a subset of at least two movement types, said subset having less movement types than the set of movement types;

receiving input specifying a mask region in a subset of one or more of the video images;

removing unwanted movement corresponding to said subset of at least two movement types from the set of video images without removing unwanted movement corresponding to at least one of the two movement types from the mask region in the subset of video images.

2. The non-transitory computer readable medium of claim 1, wherein said unwanted movement comprises a camera movement.

3. The non-transitory computer readable medium of claim 2, wherein said unwanted camera movement comprises a translation motion.

4. The non-transitory computer readable medium of claim 2, wherein said unwanted camera movement comprises a camera zoom operation.

5. A non-transitory computer readable medium, storing a video editing application having a graphical user interface (“

GUI”

), the video editing application executable by at least one processing the GUI comprising;

a display area for displaying a video clip, the video clip comprising a plurality of video images;

a movement selection tool for displaying a plurality of individually selectable movement types;

a masking tool for specifying a masked region in a set of one or more video images from the plurality of video images; and

a selectable GUI item for activating a smoothing operation that;

when only one particular movement type is selected through the movement selection tool, removes the particular movement type from only a portion of the video clip that does not include the masked region; and

when a plurality of movement types are selected through the movement selection tool, concurrently removes the plurality of selected movement types from only the same portion of the video clip.

6. The non-transitory computer readable medium of claim 5, wherein the movement types comprise camera movement types.

7. The non-transitory computer readable medium of claim 6, wherein the camera movement types comprise at least a translation motion and a zoom operation.

8. A non-transitory computer readable medium storing a video editing application which when executed by at least one processing unit edits a sequence of video images, the video editing application having a graphical user interface (“

GUI”

), the GUI comprising;

a display area for displaying a video clip, the video clip comprising a plurality of video images;

a movement selection tool for displaying a set of at least three movement types that are individually selectable;

concurrently receiving a user selection of a subset of at least two movement types, said subset having less movement types than the set of movement types;

a masking tool for specifying a mask region in a set of one or more of video images in the plurality of video images; and

removing unwanted movement corresponding to said subset of at least two movement types from the plurality of video images without removing unwanted movement corresponding to at least one of the two movement types from the mask region in the set of video images.

9. The non-transitory computer readable medium of claim 8, wherein the selected subset of movement types includes a first translation motion in a first direction and a second translation motion in a second different direction.

10. The non-transitory computer readable medium of claim 8, wherein said unwanted movement comprises a camera zoom operation.

11. A non-transitory computer readable medium storing a computer program for editing a sequence of video images, the computer program executable by at least one processor, said computer program comprising sets of instructions for:

defining a plurality of motion types, each motion type representing a different camera movement;

generating a first set of a plurality of user adjustable curves, each of said first set curves representing a camera movement corresponding to one motion type of the plurality of motion types;

receiving manual adjustments of a subset of one or more curves from the first set curves in response to user input;

generating a second set of a plurality of curves, each of said second set curves being a smoothed version of a corresponding user adjustable first set curve, wherein a first subset of the second set curves corresponds to the subset of manually adjusted curves while a second subset of the second set curves does not correspond to the subset of manually adjusted curves;

defining, for each motion type in said plurality of motion types, a transform operation from the user adjustable first set curve and the corresponding second set curve; and

removing unwanted camera movements from the sequence of video images using said transform operations defined for the plurality of motion types.

12. The non-transitory computer readable medium of claim 11, wherein said unwanted camera movements comprise two camera translation motions along two different directions.

13. The non-transitory computer readable medium of claim 11, wherein said unwanted camera movements comprise a camera rotation motion and a camera zoom motion.

14. The non-transitory computer readable medium of claim 11, wherein the set of instructions for said generating the second set curves comprises a set of instructions for removing jagged edges from each first set curve.

15. The non-transitory computer readable medium of claim 11, wherein the set of instructions for said generating the second set curves comprises a set of instructions for using a heat diffusion model to smooth out each first set curve.

16. The non-transitory computer readable medium of claim 11, wherein the set of instructions for said removing comprises sets of instructions for:

selecting a set of pixels to track between at least two images;

generating a motion function that expresses a motion of the set of pixels between said at least two images; and

using the generated motion function to remove the unwanted camera movements.

17. The non-transitory computer readable medium of claim 16, wherein the set of instructions for using the generated motion function comprises sets of instructions for:

generating a function that expresses a difference between said at least two images based on the motion function; and

finding a solution for the generated function.

18. A computer-implemented method for editing a sequence of video images, the method comprising:

defining a plurality of motion types, each motion type representing a different camera movement;

at a computer, generating a first set of a plurality of user adjustable curves, each of said first set curves representing a camera movement corresponding to one motion type of the plurality of motion types;

receiving manual adjustments of a subset of one or more curves from the first set curves in response to user input;

generating a second set of a plurality of curves, each of said second set curves being a smoothed version of a corresponding user adjustable first set curve, wherein a first subset of the second set curves corresponds to the subset of manually adjusted curves while a second subset of the second set curves does not correspond to the subset of manually adjusted curves;

defining, for each motion type in said plurality of motion types, a transform operation from the user adjustable first set curve and the corresponding second set curve; and

removing unwanted camera movements from the sequence of video images using said transform operations defined for the plurality of motion types.

19. The method of claim 18, wherein said unwanted camera movements comprise a camera translation motion, a camera rotation motion, and a camera zoom motion.

20. The method of claim 18, wherein said removing comprises:

selecting a set of pixels to track between at least two images;

generating a motion function that expresses a motion of the set of pixels between said at least two images; and

using the generated motion function to remove the unwanted camera movements.

21. The method of claim 20 further comprising:

generating a function that expresses a difference between said at least two images based on the motion function; and

finding a solution for the generated function.

22. A non-transitory computer readable medium storing a video editing application having a graphical user interface (“

GUI”

), the video editing application executable by at least one processing unit, the GUI comprising;

a display area for displaying a video clip;

a movement selection tool for displaying a plurality of individually selectable movement types;

a sensitivity tool for specifying in a sensitivity value for each individually selectable movement type from a plurality of sensitivity values associated with the plurality of selectable movement types; and

a selectable GUI item for activating a smoothing operation that;

when only one particular movement type is selected through the movement selection tool, removes the particular movement type from at least a portion of the video clip; and

when a plurality of movement types are selected through the movement selection tool, concurrently removes the plurality of selected movement types from at least the same portion of the video clip.

23. The non-transitory computer readable medium of claim 22, wherein the GUI further comprises a range selection tool for specifying the portion of the video clip for the smoothing operation to analyze.

24. The non-transitory computer readable medium of claim 22, wherein the GUI further comprises a motion tracking display area for displaying, for each selected movement type, a curve that tracks camera motion over a set of video images of the video clip.