Stylization of video

• US 7,657,060 B2
• Filed: 03/31/2004
• Issued: 02/02/2010
• Est. Priority Date: 03/31/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shum, Heung-Yeung, Cohen, Michael F., Wang, Jue, Xu, Ying-Qing
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): Bitar; Nancy
• Application Number:
  US10/814,851
• Publication Number:
  US 20050226502A1
• Time in Patent Office: 2,134 Days
• Field of Search: 382/173, 382/103, 382/284, 382/128, 382/299, 382/167, 382/164, 382/201, 382/205, 382/162, 382/165, 382/305, 382/176, 382/203, 382/202, 382/224, 382/275, 345/473, 708/490, 707/6, 378/62, 378/87
• US Class Current: 382/103
• CPC Class Codes: G06T 15/02 Non-photorealistic rendering
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

The techniques and mechanisms described herein are directed to a system for stylizing video, such as interactively transforming video to a cartoon-like style. Briefly stated, the techniques include determining a set of volumetric objects within a video, each volumetric object being a segment. Mean shift video segmentation may be used for this step. With that segmentation information, the technique further includes indicating on a limited number of keyframes of the video how segments should be merged into a semantic region. Finally, a contiguous volume is created by interpolating between keyframes by a mean shift constrained interpolation technique to propagate the semantic regions between keyframes.

Claims

1. A method for styling vides video utilizing a processor and a memory, comprising:

performing a spatio-temporal segmentation analysis on the video to identify three dimensional volumes of contiguous pixels having a similar color;

receiving an interactive user input identifying a group of the three dimensional volumes of contiguous pixels, wherein the inactive user input comprises outlining a plurality of the three dimensional volumes of contiguous pixels,

wherein the outlining comprises a user manually drawing loop boundaries that physically encircle the three dimensional volumes of contiguous pixels, the three dimensional volumes of contiguous pixels extending forward and backward in time, the outlining being performed on a number of keyframes of the video, the number of keyframes being fewer than a total number of frames of the video,

and additional three dimensional volumes of contiguous pixels on frames of the video other than keyframes are identified by determining a relationship of the additional three dimensional volumes of contiguous pixels to the three dimensional volumes of contiguous pixels outlined on the keyframes; and

identifying the group of three dimensional volumes of contiguous pixels as a single semantic region;

deriving a set of two-dimensional edge sheets that represent the surface of the single three-dimensional semantic region, the edge sheets being derived from constituent surface representations of the three-dimensional semantic region, the constituent surface representations being annotated with measurable properties, the edge sheets being derived based on a value of the measurable properties, wherein the edge sheets are sliced at a frame time to extract a curved line configured to be rendered with the stylized video,

and associating the edge sheets with the single three-dimensional semantic region, wherein a thickness of the edge sheets is determined based on a user-input parameter in combination with criteria associated with the single three-dimensional semantic region, the criteria comprising a position of the edge sheet relative to an arclength of the edge sheet.

2. The method of claim 1, wherein the spatio-temporal segmentation analysis comprises an anisotropic kernel mean shift segmentation procedure.

3. The method of claim 1, wherein the relationship comprises at least a portion of the additional three dimensional volumes of contiguous pixels being enclosed by one or more of the three dimensional volumes of contiguous pixels outlined on the keyframes.

4. The method of claim 3, wherein the at least a portion comprises at least a majority of pixels of the additional three dimensional volumes of contiguous pixels.

5. The method of claim 1, further comprising applying a stylization to the single semantic region.

6. The method of claim 5, wherein the stylization comprises a mean shift technique.

7. A computer storage medium having computer-executable instructions for stylizing video stored thereon, the instructions comprising:

performing a spatio-temporal segmentation analysis on the video to identify three dimensional volumes of contiguous pixels having a similar color;

receiving an interactive user input identifying a group of the three dimensional volumes, wherein the interactive user input comprises manually outlining a plurality of three dimensional volumes of contiguous pixels;

identifying the group of three dimensional volumes as a single three-dimensional semantic region; and

deriving a set of two-dimensional edge that represent the surface of the single three-dimensional semantic region, the edge sheets being derived from constituent surface representation of the three-dimensional semantic region, the constituent surface representation being annotated with measurable properties, the edge sheets being derived based on a value of the measurable properties, wherein the edge sheets are sliced at a frame time to exact a curved line configured to be rendered with the stylized video,

and associating the edge sheets with the single three-dimensional semantic region, wherein a thickness of the edge sheets is determined based on a user-input parameter in combination with criteria associated with the single three-dimensional semantic region, the criteria comprising a position of the edge sheet relative to an arclength of the edge sheet.

8. The computer storage medium of claim 7, further comprising rendering the edge sheets as a curve between the single three-dimensional semantic region and another portion of the video.

9. The computer storage medium of claim 7, wherein the criteria comprises a duration of existence of the single three-dimensional semantic region in the video.

10. The computer storage medium of claim 7, wherein the criteria comprises a movement of the single three-dimensional semantic region in the video.