Digital video effects

• US 8,026,931 B2
• Filed: 08/28/2006
• Issued: 09/27/2011
• Est. Priority Date: 03/16/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Weiwei, Wang, Qiang, Sun, Jian, Shum, Heung-Yeung, Tang, Xiaoou
• Primary Examiner(s): Tung; Kee M
• Assistant Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US11/467,859
• Publication Number:
  US 20070216675A1
• Time in Patent Office: 1,856 Days
• Field of Search: 345/632, 345/633
• US Class Current: 345/632
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation

Abstract

Digital video effects are described. In one aspect, a foreground object in a video stream is identified. The video stream comprises multiple image frames. The foreground object is modified by rendering a 3-dimensional (3-D) visual feature over the foreground object for presentation to a user in a modified video stream. Pose of the foreground object is tracked in 3-D space across respective ones of the image frames to identify when the foreground object changes position in respective ones of the image frames. Based on this pose tracking, aspect ratio of the 3-D visual feature is adaptively modified and rendered over the foreground object in corresponding image frames for presentation to the user in the modified video stream.

Claims

1. A method at least partially implemented by a computing device, the method comprising:

identifying a foreground object in a video stream comprising multiple image frames;

rendering a three-dimensional (3-D) visual feature over a portion of the foreground object to add a digital video effect to the video stream and generate a modified foreground object;

tracking each pose of the foreground object in 3-D space across the multiple image frames by;

iteratively refining estimations associated with a first and a second conditional distributions in a joint distribution of a Bayesian network that models the foreground object, the first conditional distribution comprises a distribution of a relative pose given correspondences between 3-D model points and two-dimensional (2-D) features of the foreground object, the second conditional distribution comprises a distribution of matching features of the visual object between two frames of the video sequence given the 3-D model points and a relative pose estimation associated with the first conditional distribution; and

using a Bayesian fusion of the iteratively refined estimations to obtain a current pose of the foreground object, wherein the iteratively refined estimations include an iteratively refined relative pose estimation and an iteratively refined feature matching estimation; and

maintaining rendered aspect ratios of the 3-D visual feature in correspondence with aspect ratios of a remaining portion of the foreground object as the foreground object changes pose in respective ones of the image frames.

2. The method of claim 1, wherein the video stream is associated with a communication session between at least two users.

3. The method of claim 1, wherein the method is a networked service provided to one or more users of one or more remote computing devices.

4. The method of claim 1, wherein tracking the pose further comprises tracking facial features to identify rotational and translational vectors across respective ones of the image frames;

and wherein the method further comprises rendering the 3-D visual feature over the portion of the foreground object based on the rotational and translational vectors.

5. The method of claim 1, further comprising presenting a user interface to a user for the user to select at least the 3-D visual feature.

6. The method of claim 1, further comprising presenting a modified video stream comprising the modified foreground object to one or more users.

7. The method of claim 1, further comprising:

identifying a background from one or more of the image frames;

modifying the background to generate a modified background, the modifying being independent of any modification to the foreground object; and

wherein the modifying adaptively modifies the background responsive to one or more of background changes and detected camera shake across respective ones of the image frames.

8. The method of claim 7, further comprising presenting a user interface to a user for the user to select a modification to apply to the background.

9. The method of claim 8, wherein modifying the background comprises selectively blurring the background, replacing the background with an image, or replacing the background with an animation.

10. The method of claim 8, further comprising presenting a modified video stream to one or more users, the modified video stream comprising the modified background and the modified foreground object.

11. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

generating a video stream comprising a 3-D image of a first person involved in a video communication session with a second person using a remote computing device;

separating a foreground object representing the 3-D image from a background of the video stream based on differences between color and contrast attributes of pixels of the foreground object and color and contrast attributes of pixels of the background;

refining boundaries between the foreground object and the background by adaptively attenuating a background contrast while preserving contrasts across the boundaries;

adaptively rendering a 3-D feature over particular ones of multiple video frames that comprise the foreground object in multiple translational and rotational poses to generate a modified video stream, the 3-D feature being rendered over a portion of the 3-D facial features of the first person such that aspect ratios of the 3-D feature are maintained in correspondence with aspect ratios of a remaining portion of 3-D facial features in view of the translational and rotational poses; and

communicating the modified video stream to the remote computing device for presentation to the second person.

12. The computing device of claim 11, further comprising determining the 3-D feature from a set of pre-configured user preferences.

13. The computing device of claim 11, further comprising:

presenting a user interface to the first person, the user interface comprising a video effects area that provides one or more selectable costume options to the first person;

responsive to the first person selecting a particular one option of the one or more selectable costume options, setting the 3-D feature to the particular one option.

14. The computing device of claim 13, wherein the user interface further comprises a first display area for presenting an image of the first person and a second display area for presenting an image of the second person, the user interface being used by the first and second person for real-time communications between the first and the second person, the image of the first person being a modified image overlain with the 3-D feature responsive to user input, the modified image representing what is viewed by the second person during the real-time communications.

15. The computing device of claim 11, further comprising:

dynamically altering the background to adapt to changes to the background across respective ones of the frames and translational and rotational movement of the foreground object, the dynamic altering generating modified background frames; and

wherein the modified video stream comprises the modified background frames.

16. The computing device of claim 15, further comprising:

presenting a user interface to the first person, the user interface comprising a video effects area that provides one or more selectable background modification options to the first person; and

responsive to the first person selecting a particular one option of the background modification options, applying the particular one option to adaptively modify the background across respective ones of the frames that represent the background, the modified video stream comprising such adaptive background modifications.

17. The computing device of claim 16, wherein the user interface further comprises a first display area for presenting, responsive to user input, the modified video stream, the modified video stream representing what is viewed by the second person during the real-time communications.

18. A tangible computer-readable storage medium comprising computer-program instructions executable by a processor, the computer-program instructions, when executed by the processor, for performing operations comprising:

providing a user with one or more video stream background modification options;

presenting the user with one or more costume overlay options;

responsive to selection by the user of a particular background modification option of the video stream background modification options, adaptively modifying background of a video stream using the particular background modification option;

responsive to selection by the user of a particular costume overlay option of the costume overlay options, adaptively rendering a 3-D visual feature associated with the costume overlay option over a portion of a foreground object in frames that comprise the video stream, the portions being parts of facial features;

maintaining aspect ratios of the 3-D visual features in correspondence with aspect ratios of a remaining portion of the foreground object as the foreground object changes translational or rotational position in the frames; and

communicating the video stream to a remote computing device for presentation to a different user.

19. The tangible computer-readable storage medium of claim 18, wherein the computer-program instructions further comprise instructions for presenting a different video stream to the user, the different video stream showing an image of the different user that has been altered with one or more 3-D visual features that follows 3-D translational and rotational movement of the different user across respective frames of the different video stream.

20. The tangible computer-readable storage medium of claim 19, wherein the different video stream further comprises an adaptively altered background that is blurred or replaced as per input from the different user, the altered background being adapted to allow for translational and rotational movement of the different user across respective frames of the different video stream.