Learning object cutout from a single example

• US 20080304735A1
• Filed: 06/05/2007
• Published: 12/11/2008
• Est. Priority Date: 06/05/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tang, Xiaoou, Wen, Fang, Yang, Qiong
• Granted Patent: US 8,644,600 B2
• Time in Patent Office: Days
• US Class Current: 382/164
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20121 Active appearance model [AAM]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]

Abstract

Systems and methods are described for learning visual object cutout from a single example. In one implementation, an exemplary system determines the color context near each block in a model image to create an appearance model. The system also learns color sequences that occur across visual edges in the model image to create an edge profile model. The exemplary system then infers segmentation boundaries in unknown images based on the appearance model and edge profile model. In one implementation, the exemplary system minimizes the energy in a graph-cut model where the appearance model is used for data energy and the edge profile is used to modulate edges. The system is not limited to images with nearly identical foregrounds or backgrounds. Some variations in scale, rotation, and viewpoint are allowed.

Claims

1. A method, comprising:

receiving a first image; and

segmenting a second image based on contextual colors and edge profiles in the first image.

2. The method as recited in claim 1, wherein the first image is segmented into a foreground object and a background.

3. The method as recited in claim 1, further comprising:

in the first image, creating an appearance model of a foreground object and a background based on contextual colors in the first image, wherein the appearance model is based on local color patterns of the first image;

creating an edge profile model of the first image based on learned color contexts near edges in the first image; and

wherein the segmenting includes optimizing a segmentation boundary between a foreground object of the second image and a background of the second image using at least in part the appearance model and the edge profile model obtained from the first image.

4. The method as recited in claim 3, wherein creating the appearance model of the foreground object includes representing a likelihood energy of the foreground object, the likelihood energy used in optimizing the segmentation boundary of the second image.

5. The method as recited in claim 3, wherein creating the edge profile model includes creating edge profiles for modulating a smoothness energy used in optimizing the segmentation boundary of the second image.

6. The method as recited in claim 3, wherein optimizing the segmentation boundary includes optimizing a binary labeling process by minimizing a total energy via a min-cut/max-flow technique that includes a likelihood energy component and a smoothness energy component;

wherein at least part of the appearance model represents the likelihood energy; and

wherein at least part of the edge profile model modulates the smoothness energy.

7. The method as recited in claim 6, wherein the min-cut/max flow technique comprises

$E$

( X ) = ∑

p ∈

V 

E data 

( x p ) + α



∑

( p , q ) ∈

ɛ



E smooth 

( x p , x q ) ,

wherein E_datax_p comprises the likelihood energy, encoding a cost when the label of a pixel p in the first image is x_p, and E_smooth(x_p, x_q) comprises the smoothness energy, denoting a cost when the labels of adjacent pixels p and q in the first image are labeled x_p and x_q, respectively, and α

comprises a parameter to balance between the likelihood energy and the smoothness energy.

8. The method as recited in claim 3, wherein creating the appearance model of the foreground object and the background based on contextual colors in the first image includes:

partitioning the first image into segments;

partitioning each segment into blocks;

selecting multiple directions to proceed from each block;

for each block;

proceeding in each of the directions until a sample color along the direction differs by a contrast threshold from a color of the block; and

recording the sample color and the color of the block as a contextual color pair.

9. The method as recited in claim 8, further comprising recording a number of the contextual color pairs for each block, the number equal, to the number of the directions selected.

10. The method as recited in claim 8, wherein at least one of the segments represents a foreground object.

11. The method as recited in claim 8, further comprising pre-segmenting the first image via a mean-shift technique.

12. The method as recited in claim 1, further comprising:

in the first image, creating an appearance model of a background and a foreground object having a likelihood energy, based on contextual colors in the first image;

in the second image creating an appearance likelihood model of a likely foreground and a likely background of the second image, including a cumulative likelihood energy of the likely foreground;

in the first image, creating an edge profile model of the first image based on learned color contexts near edges in the first image;

in the second image, creating an edge profile likelihood model of the second image based on learned color contexts of edges in the second image, including a likely segmentation boundary;

optimizing the likely segmentation boundary in the second image by minimizing a total energy via a min-cut/max-flow technique that includes a likelihood energy component and a smoothness energy component.

13. The method as recited in claim 12, wherein the likelihood energies are calculated via contextual color pairs, the contextual color pairs for the first and second images being obtained by:

partitioning the first and second images into segments;

partitioning each segment into blocks;

selecting multiple directions to proceed from each block;

for each block;

proceeding in each of the directions until a sample color along the direction differs by a contrast threshold from a color of the block;

recording the sample color and the color of the block as a contextual color pair; and

recording a number of the contextual color pairs for each block, the number equal to the number of the directions selected.

14. The method as recited in claim 1, further comprising applying the segmenting to one of:

automatically segmenting images similar to the first image;

automatically segmenting images that have a different foreground or a different background than the first image;

automatically cutting out a video object moving between frames of a video sequence;

automatically detecting a facial view;

automatically searching a database during content-based image retrieval.

18. The method as recited in claim 1, further comprising applying the propagating object cutout to one of:

automatically segmenting images similar to the first image;

automatically segmenting images that have a different foreground or a different background than the first image;

automatically cutting out a video object moving between frames of a video sequence;

automatically detecting a facial view;

automatically searching a database during content-based image retrieval.

15. A method of propagating object cutout from a model image, comprising:

dividing the model image into blocks;

sampling contextual colors around each block into an appearance model that is based on learning local color patterns of the model image;

learning edge profiles for edges in the model image;

segmenting a different image based on both the appearance model and the edge profiles.

16. The method as recited in claim 15, wherein sampling contextual colors around each block further includes:

selecting multiple directions to proceed from each block;

for each block;

proceeding in each of the directions until a sample color of a different color mode is detected,;

recording the sample color and the color of the block as a contextual color pair; and

recording a number of the contextual color pairs for each block, the number equal to the number of the directions selected.

17. The method as recited in claim 15, further comprising:

deriving a likelihood energy from the appearance model;

deriving a smoothness energy from the edge profiles;

optimizing binary labeling of each pixel in the different image by minimizing a total energy that includes the likelihood energy and the smoothness energy;

wherein the likelihood energy denotes the cost when labeling the pixel as either foreground object or background and modulated smoothness energy handicaps two adjacent pixels that are given different labels to force the segmentation boundary to follow a true object boundary.

19. A system, comprising:

means for sampling a local color pattern around each block of a first image;

means for learning edges in the first image; and

means for segmenting a second image based on the local color patterns and the learned edges in the first image.

20. The system as recited in claim 19, wherein the means for segmenting includes:

means for optimizing a segmentation boundary in the second image, which further includes;

means for minimizing an energy comprising;

a likelihood energy derived from the local color patterns of the first image and local color patterns extracted from the second image; and

a smoothness energy of edges of the second image modulated by the learned edges of the first image.