MODIFYING VIDEO REGIONS USING MOBILE DEVICE INPUT

• US 20120327172A1
• Filed: 06/22/2011
• Published: 12/27/2012
• Est. Priority Date: 06/22/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: El-Saban, Motaz, Lau, James Kai Yu, Kaheel, Ayman, Shawky, Mohamed
• Granted Patent: US 9,153,031 B2
• Time in Patent Office: Days
• US Class Current: 348/14.02
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20096 Interactive definition of curve of interest
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 2007/145 Handheld terminals

Abstract

Apparatus and methods are disclosed for modifying video based on user input and or face detection data received with a mobile device to generate foreground regions (e.g., to separate a user image from background in the video). According to one disclosed embodiment, a method comprises receiving user input and/or face regions generated with a mobile device, producing an initial representation for segmenting input video into a plurality of portions based on the user input, where the initial representation includes probabilities for one or more regions of the input video being designated as foreground regions or background regions. Based on the initial representation, input video is segmented by designating one or more of the regions of the input video as foreground regions or background regions.

Claims

1. A method, comprising:

receiving input generated with a mobile device for positioning an edge segmentation template;

producing an initial representation for segmenting input video into a plurality of portions, the initial representation based on the positioned edge segmentation template, the initial representation including weights for one or more regions of the input video to be designated as foreground regions or background regions; and

based on the initial representation, segmenting the input video by designating one or more of the portions of the input video as foreground regions or background regions.

2. The method of claim 1, wherein the weights are based on at least one or more of the following:

a color likelihood term, a spatial prior term, a temporal prior term, or a motion likelihood term.

3. The method of claim 1, wherein the weights are based on a superpixel-based term that includes superpixel neighbor effects.

4. The method of claim 1, wherein the initial representation represents a plurality of pixels of the input video as a single superpixel.

5. The method of claim 1, wherein the input for positioning the edge segmentation template is generated by:

displaying a template superimposed over the input video on a display coupled to the mobile device;

receiving user input for positioning the template relative to the displayed input video; and

generating the positioned edge segmentation template based on the user input.

6. The method of claim 1, wherein the input for positioning the edge segmentation template is generated by:

generating one or more face regions for one or more regions of the input video using a camera coupled to the mobile device; and

positioning the edge segmentation template based on the face regions.

7. The method of claim 6, wherein:

the face regions include a foreground face region and one or more background face regions;

the foreground face region is assigned one or more weights associated with being designated as a portion of the foreground region; and

the background face regions are assigned one or more weights associated with being designated as a portion of the background regions.

8. The method of claim 1, further comprising generating modified video, the modified video comprising at least some of the designated foreground regions but not comprising at least some portions of the input video not designated as foreground regions.

9. The method of claim 8, further comprising transmitting the modified video for use with a video call application.

10. The method of claim 1, wherein the initial representation is based on a first frame of the input video, and wherein the segmenting comprises designating one or more regions of other frames of the input video, the method further comprising:

periodically repeating the act of producing the initial representation for a different first frame of the input video to produce an updated initial representation; and

based on the updated initial representation, repeating the act of segmenting for different other frames of the input video, the different other frames not including the different first frame.

11. The method of claim 10, wherein:

the initial representation represents a subset of one or more but not all pixels of one or more frames of the input video.

12. One or more computer-readable storage media storing computer-readable instructions that when executed by a computer, cause the computer to perform the method of claim 1.

13. A method of designating background regions in a first image selected from a sequence of images, the method comprising:

generating one or more energy terms for the first image based on an edge segmentation template positioned using a mobile device;

based on the energy terms, designating one or more regions of the first image as background regions; and

replacing one or more of the designated background regions of the image with corresponding regions from a different image or video than the first image to produce a modified sequence of images.

14. The method of claim 13, wherein the edge segmentation template is positioned using user input received with the mobile device.

15. The method of claim 13, wherein the edge segmentation template is positioned by detecting one or more faces in at least a portion of input video generated with the mobile device.

16. The method of claim 13, further comprising:

repeating the generating one or more energy terms for a second respective image in the sequence of images, wherein the energy terms are based on a subset of pixels in the second respective image; and

repeating the designating, the replacing, and the displaying for a different image than the second respective image.

17. The method of claim 13, wherein:

one or more of the energy terms are based on seed regions for a second image in the sequence of images, the seed regions designating one or more pixels of the second image as either a foreground region or a background region based on user input received with the mobile device;

the replacing the background regions comprises applying a filter to the designated background regions of the image to produce the modified sequence of images; and

the modified sequence of images comprises one or more regions of the image designated as foreground portions; and

the method further comprises displaying the modified sequence of images on a display coupled to the mobile device.

18. The method of claim 13, wherein:

the image is comprised of a plurality of pixels; and

the acts of generating the seed regions and designating the foreground and background regions are performed using superpixels, each of the superpixels representing two or more of the plurality of pixels, wherein the number of pixels each of the represents is determined based at least in part on usage of a computing resource associated with the mobile device.

19. A mobile device, comprising:

a video camera operable to produce input video;

a touch screen display for receiving touch screen input and displaying video;

at least one processor coupled to the video camera and the touch screen display;

a communication interface coupled to the at least one processor; and

one or more computer-readable storage media storing computer-readable instructions executable by the at least one processor for transmitting modified video based on the input video, the instructions comprising;

instructions for receiving touch screen input from the touch screen display and designating one or more regions of the input video based on the touch screen input,

instructions for designating one or more regions of a portion of the input video using a representation of a conditional random field (CRF),

instructions for replacing one or more of the designated regions in the input video with corresponding regions from a second image or video to produce the modified video, wherein the replacing is based at least in part on the designated regions, and

instructions for transmitting the modified video using the communication interface.

20. The mobile device of claim 19, wherein the instructions further comprise:

instructions for displaying the input video in a first portion of the touch screen display; and

instructions for displaying the modified video in a second portion of the display, wherein the instructions for displaying the input video and the instructions for displaying the modified video can be executed to display the input video and the modified video concurrently.