Juxtaposing still and dynamic imagery for cliplet creation

• US 8,872,850 B2
• Filed: 03/05/2012
• Issued: 10/28/2014
• Est. Priority Date: 03/05/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Joshi, Neel Suresh, Mehta, Sisil Sanjeev, Cohen, Michael F., Drucker, Steven M., Hoppe, Hugues, Uyttendaele, Matthieu
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US13/411,651
• Publication Number:
  US 20130229581A1
• Time in Patent Office: 967 Days
• Field of Search: 345/629
• US Class Current: 345/629
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Various technologies described herein pertain to juxtaposing still and dynamic imagery to create a cliplet. A first subset of a spatiotemporal volume of pixels in an input video can be set as a static input segment, and the static input segment can be mapped to a background of the cliplet. Further, a second subset of the spatiotemporal volume of pixels in the input video can be set as a dynamic input segment based on a selection of a spatial region, a start time, and an end time within the input video. Moreover, the dynamic input segment can be refined spatially and/or temporally and mapped to an output segment of the cliplet within at least a portion of output frames of the cliplet based on a predefined temporal mapping function, and the output segment can be composited over the background for the output frames of the cliplet.

Claims

1. A method of juxtaposing still and dynamic imagery to generate a cliplet, the method executed by a processor of a computing device, the method comprising:

setting a first subset of a spatiotemporal volume of pixels in an input video as a static input segment;

mapping the static input segment to a background of the cliplet, wherein the background is repeated across output frames of the cliplet;

setting a second subset of the spatiotemporal volume of pixels in the input video as a dynamic input segment based on a selection of a spatial region within the input video, a start time within the input video, and an end time within the input video;

refining the dynamic input segment, the refining further comprises at least one of;

spatially refining the dynamic input segment based upon;

a first region within the spatial region over a time interval from the start time to the end time, the first region being along a spatial boundary of the spatial region; and

a second region within the static in input segment that abuts the spatial region;

or

temporally refining the dynamic input segment based upon;

the spatial region in frames within a first temporal region of the dynamic input segment that includes the start time; and

the spatial region in frames within a second temporal region of the dynamic input segment that includes the end time;

mapping, based on a predefined temporal mapping function, the dynamic input segment as refined to an output segment of the cliplet defined within at least a portion of the output frames of the cliplet, wherein the predefined temporal mapping function is one of a play temporal mapping function, a loop temporal mapping function, or a mirror temporal mapping function; and

compositing the output segment over the background for the output frames of the cliplet.

2. The method of claim 1, wherein the first subset of the spatiotemporal volume of pixels in the input video set as the static input segment includes pixels from a single frame of the input video.

3. The method of claim 1, wherein the static input segment is at least one of a panoramic image stitched from a pan in the input video or a clean plate image with one or more items removed from the input video.

4. The method of claim 1, further comprising pre-processing the input video prior to setting the static input segment and the dynamic input segment, the pre-processing globally aligns frames of the input video based on features included in the input video.

5. The method of claim 1, wherein spatially refining the dynamic input segment further comprises deforming the spatial region of the dynamic input segment over the time interval, the spatial region being deformed to minimize optical flow of pixels in the first region within the spatial region along the spatial boundary over the time interval, and align with the second region within the static input segment.

6. The method of claim 1, wherein the temporal mapping function is the loop temporal mapping function, the method further comprising:

detecting a pair of frames from the input video where differences in pixel color and optical flow in the spatial region between the pair of frames are minimized; and

at least one of;

suggesting the pair of frames as the start time within the input video and the end time within the input video;

or

automatically selecting the pair of frames as the start time within the input video and the end time within the input video.

7. The method of claim 1, wherein temporally refining the dynamic input segment further comprises temporally feathering, using optical flow, the spatial region across the frames within the first temporal region that includes the start time and the frames within the second temporal region that includes the end time, wherein the predefined temporal mapping function is the loop temporal mapping function.

8. The method of claim 1, wherein the predefined mapping function is the mirror temporal mapping function, the method further comprising:

detecting a first frame and a second frame from the input video where differences in optical flow in the spatial region between the first frame and frames temporally adjacent to the first frame and between the second frame and frames temporally adjacent to the second frame, respectively, are minimized; and

at least one of;

suggesting the first frame and the second frame as the start time within the input video and the end time within the input video;

or

automatically selecting the first frame and the second frame as the start time within the input video and the end time within the input video.

9. The method of claim 1, wherein temporally refining the dynamic input segment further comprises retiming the dynamic input segment to gradually slow the frames within the first temporal region towards the start time and the frames within the second temporal region towards the end time, wherein the predefined temporal mapping function is the mirror temporal mapping function.

10. The method of claim 1, wherein compositing the output segment over the background for the set of output frames of the cliplet further comprises spatially blending the output segment and the background.

11. The method of claim 1, further comprising:

setting a third subset of the spatiotemporal volume of pixels in the input video as a second dynamic input segment based on a selection of a second spatial region within the input video, a second start time within the input video, and a second end time within the input video;

refining the second dynamic input segment at least one of spatially or temporally;

mapping, based on a second predefined temporal mapping function, the second dynamic input segment as refined to a second output segment of the cliplet defined within at least a second portion of the output frames of the cliplet, wherein the second predefined temporal mapping function is one of the play temporal mapping function, the loop temporal mapping function, or the mirror temporal mapping function; and

compositing the second output segment over the output segment and the background for the output frames of the cliplet.

12. The method of claim 1, further comprising receiving input via a user interface, wherein the input identifies the predefined temporal mapping function, the spatial region within the input video, the start time within the input video, and the end time within the input video.

13. The method of claim 1, further comprising automatically generating a gallery of potential cliplets from the input video, wherein the potential cliplets are differing variations generated from the input video, and wherein the gallery includes the cliplet.

14. The method of claim 1, the computing device being a device that captures the input video.

15. A system that generates a cliplet from an input video, comprising:

a processor; and

a memory that comprises components that are executable by the processor, the components comprising;

a segmentation component that sets respective subsets of a spatiotemporal volume of pixels in the input video as input segments based on selections of spatial regions within the input video, start times within the input video, and end times within the input video;

a spatial refinement component that spatially refines a first input segment, the first input segment having a first spatial region and a time interval from a start time to an end time, the spatial refinement component deforms the first spatial region over the time interval to minimize optical flow of pixels in a portion of the first spatial region along a spatial boundary over the time interval and align with a portion of a second spatial region within a second input segment, the portion of the second spatial region abuts the portion of the first spatial region;

a map component that maps the input segments as refined to output segments using a set of predefined temporal mapping functions, wherein each input segment is mapped using a corresponding predefined temporal mapping function selected for the input segment from the set of predefined temporal mapping functions, wherein the set of predefined temporal mapping functions comprises at least three of a still temporal mapping function, a play temporal mapping function, a loop temporal mapping function, or a mirror temporal mapping function; and

a composite component that composites the cliplet from the output segments.

16. The system of claim 15, wherein the input video is an informally captured handheld video, and wherein the system is configured for execution by a device that captures the informally captured handheld video.

17. The system of claim 15, the components further comprising a temporal refinement component that temporally feathers, using optical flow, the first spatial region across frames within a first temporal region of the first input segment that includes the start time and frames within a second temporal region of the first input segment that includes the end time.

18. The system of claim 15, the components further comprising a loop detection component that suggests at least one of the start times and at least one of the end times corresponding to at least one of the input segments.

19. A computer-readable storage medium including computer-executable instructions that, when executed by a processor, cause the processor to perform acts including:

setting a first subset of a spatiotemporal volume of pixels in an input video as a static input segment based on a selection of a time within the input video;

mapping the static input segment to a background of a cliplet, wherein the background is repeated across output frames of the cliplet;

setting a second subset of the spatiotemporal volume of pixels in the input video as a dynamic input segment based on a selection of a spatial region within the input video, a start time within the input video, and an end time within the input video;

identifying a selection of a predefined temporal mapping function for the dynamic input segment, wherein the predefined temporal mapping function is selected from a set that comprises a play temporal mapping function, a loop temporal mapping function, and a mirror temporal mapping function;

temporally refining the dynamic input segment based upon;

the spatial region in frames within a first temporal region of the dynamic input segment that includes the start time; and

the spatial region in frames within a second temporal region of the dynamic input segment that includes the end time;

mapping, based on the selected predefined temporal mapping function, the dynamic input segment as refined to an output segment of the cliplet defined within at least a portion of the output frames of the cliplet; and

compositing the output segment over the background for the output frames of the cliplet.

20. The computer-readable storage medium of claim 19, the computer-executable instructions, when executed by the processor, further cause the processor to perform acts including:

spatially refining the dynamic input segment based upon;

a first region within the spatial region over a time interval from the start time to the end time, the first region being along a spatial boundary of the spatial region; and

a second region within the static input segment that abuts the spatial region.