JUXTAPOSING STILL AND DYNAMIC IMAGERY FOR CLIPLET CREATION

• US 20130229581A1
• Filed: 03/05/2012
• Published: 09/05/2013
• Est. Priority Date: 03/05/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Joshi, Neel Suresh, Cohen, Michael F., Drucker, Steven M., Hoppe, Hugues, Uyttendaele, Matthieu, Mehta, Sisil Sanjeev
• Granted Patent: US 8,872,850 B2
• Time in Patent Office: Days
• US Class Current: 348/584
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Various technologies described herein pertain to juxtaposing still and dynamic imagery to create a cliplet. A first subset of a spatiotemporal volume of pixels in an input video can be set as a static input segment, and the static input segment can be mapped to a background of the cliplet. Further, a second subset of the spatiotemporal volume of pixels in the input video can be set as a dynamic input segment based on a selection of a spatial region, a start time, and an end time within the input video. Moreover, the dynamic input segment can be refined spatially and/or temporally and mapped to an output segment of the cliplet within at least a portion of output frames of the cliplet based on a predefined temporal mapping function, and the output segment can be composited over the background for the output frames of the cliplet.

Claims

1. A method of juxtaposing still and dynamic imagery to generate a cliplet, comprising:

setting a first subset of a spatiotemporal volume of pixels in an input video as a static input segment from the input video;

mapping the static input segment to a background of the cliplet, wherein the background is repeated across output frames of the cliplet;

setting a second subset of the spatiotemporal volume of pixels in the input video as a dynamic input segment from the input video based on a selection of a spatial region within the input video, a start time within the input video, and an end time within the input video;

refining the dynamic input segment at least one of spatially or temporally;

mapping, based on a predefined temporal mapping function, the dynamic input segment as refined to an output segment of the cliplet defined within at least a portion of the output frames of the cliplet, wherein the predefined temporal mapping function is one of a play temporal mapping function, a loop temporal mapping function, or a mirror temporal mapping function; and

compositing the output segment over the background for the output frames of the cliplet.

2. The method of claim 1, wherein the first subset of the spatiotemporal volume of pixels in the input video set as the static input segment includes pixels from a single frame of the input video.

3. The method of claim 1, wherein the static input segment is at least one of a panoramic image stitched from a pan in the input video or a clean plate image with one or more items removed from the input video.

4. The method of claim 1, further comprising pre-processing the input video prior to setting the static input segment and the dynamic input segment to globally align frames of the input video based on features included in the input video.

5. The method of claim 1, wherein refining the dynamic input segment further comprises deforming the spatial region of the dynamic input segment over a time interval from the start time to the end time to minimize optical flow of pixels in a region along a boundary of the spatial region of the dynamic input segment over the time interval.

6. The method of claim 1, further comprising:

detecting a pair of frames from the input video where differences in pixel color and optical flow in the spatial region between the pair of frames are minimized when the predefined temporal mapping function is the loop temporal mapping function; and

at least one of;

suggesting the pair of frames as the start time within the input video and the end time within the input video;

or

automatically selecting the pair of frames as the start time within the input video and the end time within the input video.

7. The method of claim 1, wherein refining the dynamic input segment further comprises temporally feathering, using optical flow, the dynamic input segment across frames within a first temporal region that includes the start time and a second temporal region that includes the end time when the predefined temporal mapping function is the loop temporal mapping function.

8. The method of claim 1, further comprising:

detecting a first frame and a second frame from the input video where differences in optical flow in the spatial region between the first frame and frames temporally adjacent to the first frame and between the second frame and frames temporally adjacent to the second frame, respectively, are minimized when the predefined temporal mapping function is the mirror temporal mapping function; and

at least one of;

suggesting the first frame and the second frame as the start time within the input video and the end time within the input video;

or

automatically selecting the first frame and the second frame as the start time within the input video and the end time within the input video.

9. The method of claim 1, wherein refining the dynamic input segment further comprises retiming the dynamic input segment to gradually slow towards the start time and the end time when the predefined temporal mapping function is the mirror temporal mapping function.

10. The method of claim 1, wherein compositing the output segment over the background for the set of output frames of the cliplet further comprises spatially blending the output segment and the background.

11. The method of claim 1, further comprising:

setting a third subset of the spatiotemporal volume of pixels in the input video as a second dynamic input segment from the input video based on a selection of a second spatial region within the input video, a second start time within the input video, and a second end time within the input video;

refining the second dynamic input segment at least one of spatially or temporally;

mapping, based on a second predefined temporal mapping function, the second dynamic input segment as refined to a second output segment of the cliplet defined within at least a second portion of the output frames of the cliplet, wherein the second predefined temporal mapping function is one of the play temporal mapping function, the loop temporal mapping function, or the mirror temporal mapping function; and

compositing the second output segment over the output segment and the background for the output frames of the cliplet.

12. The method of claim 1, further comprising receiving input via a user interface, wherein the input identifies the predefined temporal mapping function, the selection of the spatial region within the input video, the start time within the input video, and the end time within the input video.

13. The method of claim 1, further comprising automatically generating a gallery of potential cliplets from the input video, wherein the potential cliplets are differing variations generated from the input video, and wherein the gallery includes the cliplet.

14. The method of claim 1, configured for execution by a processor of a device that captures the input video.

15. A system that generates a cliplet from an input video, comprising:

a segmentation component that sets respective subsets of a spatiotemporal volume of pixels in the input video as input segments based on selections of spatial regions within the input video, start times within the input video, and end times within the input video;

a map component that maps the input segments to output segments using a set of predefined temporal mapping functions, wherein each input segment is mapped using a corresponding predefined temporal mapping function selected for the input segment from the set of predefined temporal mapping functions, wherein the set of predefined temporal mapping functions comprises at least three of a still temporal mapping function, a play temporal mapping function, a loop temporal mapping function, or a mirror temporal mapping function; and

a composite component that composites the cliplet from the output segments.

16. The system of claim 15, wherein the input video is an informally captured handheld video, and wherein the system is configured for execution by a processor of a device that captures the informally captured handheld video.

17. The system of claim 15, further comprising a spatial refinement component that deforms at least one of the spatial regions corresponding to at least one of the input segments over time to minimize optical flow of pixels in a boundary region of the at least one of the spatial regions.

18. The system of claim 15, further comprising a temporal refinement component that temporally feathers, using optical flow, at least one of the input segments across frames.

19. The system of claim 15, further comprising a loop detection component that suggests at least one of the start times and at least one of the end times corresponding to at least one of the input segments.

20. A computer-readable storage medium including computer-executable instructions that, when executed by a processor, cause the processor to perform acts including:

setting a first subset of a spatiotemporal volume of pixels in an input video as a static input segment from the input video based on a selection of a time within the input video;

mapping the static input segment to a background of a cliplet, wherein the background is repeated across output frames of the cliplet;

setting a second subset of the spatiotemporal volume of pixels in the input video as a dynamic input segment from the input video based on a selection of a spatial region within the input video, a start time within the input video, and an end time within the input video;

identifying a selection of a predefined temporal mapping function for the dynamic input segment, wherein the predefined temporal mapping function is selected from a set that comprises a play temporal mapping function, a loop temporal mapping function, and a mirror temporal mapping function;

refining the dynamic input segment at least one of spatially or temporally;

mapping, based on the selected predefined temporal mapping function, the dynamic input segment as refined to an output segment of the cliplet defined within at least a portion of the output frames of the cliplet; and

compositing the output segment over the background for the output frames of the cliplet