Interactive image matting

• US 8,386,964 B2
• Filed: 07/21/2010
• Issued: 02/26/2013
• Est. Priority Date: 07/21/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sun, Jian, He, Kaiming, Liu, Jiangyu
• Primary Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US12/841,094
• Publication Number:
  US 20120023456A1
• Time in Patent Office: 951 Days
• Field of Search: 715/863, 715/856
• US Class Current: 715/863
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

A user interface enables interactive image matting to be performed on an image The user interface may provide results including an alpha matte as feedback in real time. The user interface may provide interactive tools for selecting a portion of the image, and an unknown region for alpha matte processing may be automatically generated adjacent to the selected region. The user may interactively refine the alpha matte as desired to obtain a satisfactory result.

Claims

1. A method comprising:

receiving an indication of a brush stroke made on an image displayed in a user interface, wherein the user interface provides a plurality of brush types, including a first brush type for identifying a foreground region of the image and a second brush type for identifying an unknown region of the image;

identifying an area of the image as a foreground region based at least in part on a location of a brush stroke of the first brush type on the image;

generating a boundary area on a border of the foreground region, the boundary area containing an unknown region;

identifying the unknown region of the image based at least in part on a location of a brush stroke of the second brush type on the image;

solving, by a processor, the unknown region to provide an alpha matte that distinguishes a foreground portion from a background portion in the unknown region; and

displaying the alpha matte for the unknown region in real time in response to receiving the indication of a brush stroke of a one of the plurality of brush types.

2. The method according to claim 1, wherein the brush stroke is made with the first brush type to indicate selection of the foreground region, and further comprising:

receiving an indication of a second brush stroke made with the second brush type to indicate an expanded unknown region;

solving the expanded unknown region to provide a modified alpha matte; and

displaying the modified alpha matte for the expanded unknown region.

3. The method according to claim 1, wherein the boundary area is initially generated having a predetermined width specified through the user interface.

4. The method according to claim 1, further comprising an automated brush selection mechanism causing the first brush type to be automatically selected when a cursor of an input device is located over the foreground region and the second brush type to be automatically selected when the cursor is located over the unknown region.

5. The method according to claim 4, further comprising a third brush type for identifying a background region, the automated brush selection mechanism causing the third brush type to be automatically selected when the cursor is located over the background region.

6. The method of claim 1, wherein solving the unknown region provides the alpha matte for a portion of the image based at least in part on a location of a most recent brush stroke of a one of the plurality of brush types.

7. The method of claim 1, wherein solving the unknown region provides the alpha matte for a portion of the image based at least in part upon a radius of the brush of the most recent brush stroke of the one of the plurality of brush types.

8. The method of claim 1, wherein solving the unknown region is processed in a predetermined number of iterations.

9. The method of claim 1, further comprising placing the foreground region on a new background as a composite image.

10. A method comprising:

receiving a user input as a first brush stroke made using a first brush type, the first brush stroke identifying a portion of an image as a foreground region;

in response to the first brush stroke, identifying, by a processor, an unknown region adjacent to the foreground region;

displaying an alpha matte that distinguishes foreground elements from background elements in the unknown region;

receiving a second user input as a second brush stroke made using a second brush type, the second brush stroke identifying a portion of a background region to be included in the unknown region;

in response to the second brush stroke, providing a modified alpha matte including an updated portion of the unknown region;

displaying the modified alpha matte in real time.

11. The method according to claim 10, wherein

additional portions of the foreground region are automatically identified as part of the foreground region based at least in part on statistically determined color similarities with the identified portion of the foreground region;

and

the unknown region is comprises a boundary of a specified width generated between the foreground region and the background region.

12. The method according to claim 10, further comprising automatically switching between the first brush type and the second brush type, wherein

when a cursor of an input device is located over the foreground region, the first brush type is automatically selected;

and

when the cursor is located over the unknown region, the second brush type is automatically selected.

13. The method according to claim 10, further comprising solving the alpha matte by:

employing a plurality of rectangles covering the unknown region, the plurality of rectangles having a size based at least in part on a number of unknown pixels in a corresponding plurality of segments of the unknown region; and

solving for the alpha matte over the plurality of rectangles using an integral images technique.

14. The method according to claim 10, further comprising solving the alpha matte by sub-sampling the image into a coarse resolution used to solve the unknown area, and subsequently perform bilinear interpolation to generate the alpha matte at a finer resolution based at least in part on a resolution of the image.

15. The method according to claim 10, further comprising calculating foreground colors in parallel with calculating the alpha matte.

16. A system comprising:

a processor in communication with computer-readable storage media;

a user interface component, maintained in the computer-readable storage media and executed on the processor, to receive a first user input comprising a first brush stroke made with a first brush type indicating a foreground region of an image, the user interface component generating an unknown region of a fixed width adjacent to the foreground region indicated by the first brush stroke, and further to receive a second user input comprising a second brush stroke made with a second brush type for identifying portions of the image for inclusion in the unknown region; and

a matting component to provide an alpha matte that distinguishes a foreground portion from a background portion in the unknown region.

17. The system according to claim 16, wherein the matting component further comprises a variable kernel size component to solve the unknown region of the image using a kernel size selected relative to a size of the unknown region.

18. The system according to claim 16, the matting component further comprising a localized integral images component to employ a plurality of local windows covering the unknown region to solve for the alpha matte using an integral images technique.

19. The system according to claim 16, the matting component further comprising a coarse-to-fine approximation component to sub-sample the image into a coarse resolution used to solve the unknown area, and subsequently perform bilinear interpolation to generate the alpha matte at a finer resolution based at least in part on a resolution of the image.

20. The system according to claim 16, the matting component further comprising a foreground estimation component to estimate foreground colors and the alpha matte in parallel.