Automatic surveillance video matting using a shape prior

• US 10,147,021 B2
• Filed: 02/29/2016
• Issued: 12/04/2018
• Est. Priority Date: 11/07/2011
• Status: Active Grant

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Yu, Ting, Tu, Peter Henry, Liu, Xiaoming, Lim, Ser Nam
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US15/056,669
• Publication Number:
  US 20160180199A1
• Time in Patent Office: 1,009 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A novel technique for performing video matting, which is built upon a proposed image matting algorithm that is fully automatic is disclosed. The disclosed methods utilize a PCA-based shape model as a prior for guiding the matting process, so that manual interactions required by most existing image matting methods are unnecessary. By applying the image matting algorithm to these foreground windows, on a per frame basis, a fully automated video matting process is attainable. The process of aligning the shape model with the object is simultaneously optimized based on a quadratic cost function.

Claims

1. A method comprising:

computing an initial alpha matte from an input image; and

automatically updating the initial alpha matte, wherein automatically updating comprises;

selecting one or more shape models from a shape model database, wherein the shape model database comprises a plurality of principal component analysis-based shape models trained using eigen analysis;

incorporating the one or more shape models into the initial alpha matte;

incorporating one or more alignment parameters into the initial alpha matte to align the one or more shape models in the initial alpha matte; and

optimizing the initial alpha matte to obtain an optimized alpha matte, wherein optimizing the initial alpha matte to obtain the optimized alpha matte comprises iteratively solving a quadratic cost function to determine the one or more alignment parameters that produce a minimum value of the quadratic cost function.

2. The method of claim 1, wherein incorporating the one or more alignment parameters into the initial alpha matte comprises:

calculating an updated alpha matte;

calculating a shape basis coefficient; and

calculating a transformation parameter.

3. The method of claim 2, wherein calculating the updated alpha matte, calculating the shape basis coefficient, and calculating the transformation parameter occur substantially simultaneously.

4. The method of claim 1, wherein iteratively solving the quadratic cost function to determine the one or more alignment parameters that produce a minimum value of the quadratic cost function comprises iteratively solving the quadratic cost function less than 20 times.

5. The method of claim 1, wherein calculating the initial alpha matte comprises extracting a foreground from a background of the input image.

6. The method of claim 5, wherein the input image comprises a video frame.

7. The method of claim 1, wherein calculating the initial alpha matte from the input image comprises calculating a plurality of alpha mattes from a plurality of images by extracting a foreground from a background of each of the plurality of images, wherein each of the plurality of images correspond to a frame of a video.

8. The method of claim 7, wherein the video comprises over 300 frames.

9. The method of claim 1, wherein the optimized alpha matte comprises an alignment error of less than 1.5 pixels along an x-axis of the alpha matte and less than 2.5pixels along a y-axis of the alpha matte.

10. The method of claim 1, wherein the shape model database comprises at least 150 principal component analysis-based shape models.

11. The method of claim 1, wherein computing the initial alpha matte from the input image and automatically updating the initial alpha matte are performed without human interaction.

12. The method of claim 1, wherein automatically updating the initial alpha matte does not include manually marking a foreground region or a background region.

13. A method for automatic video matting comprising:

obtaining a plurality of images from a video, where each of the plurality of video images correspond to a frame of the video;

utilizing one or more shape models from a shape model database to estimate an alpha matte for each of the plurality of images, wherein the shape model database comprises a plurality of principal component analysis-based shape models trained using eigen analysis; and

aligning each of the plurality of images with the one or more shape models to estimate the alpha matte for each of the plurality of images, wherein aligning each of the plurality of images with the one or more shape models to estimate the alpha matte for each of the plurality of images comprises iteratively solving a quadratic cost function to determine one or more alignment parameters that produce a minimum value of the quadratic cost function.

14. The method of claim 13, wherein each of the plurality of video images is obtained from a person detector.

15. The method of claim 13, wherein utilizing the one or more shape models from the shape model database to estimate the alpha matte for each of the plurality of images and aligning each of the plurality of images are performed simultaneously.

16. A method, comprising:

computing an initial alpha matte from an input image;

selecting one or more shape models from a shape model database, wherein the shape model database comprises a plurality of principal component analysis-based shape models trained using eigen analysis;

inputting the one or more shape models into the initial alpha matte; and

determining one or more alignment parameters of the initial alpha matte that align the one or more shape models with the input image to generate an updated alpha matte, wherein determining the one or more alignment parameters of the initial alpha matte that align the one or more shape models with the input image to generate an updated alpha matte comprises iteratively solving a quadratic cost function to determine the one or more alignment parameters that produce a minimum value of the quadratic cost function.

17. The method of claim 16, wherein iteratively solving the quadratic cost function to determine the one or more alignment parameters that produce a minimum value of the quadratic cost function comprises iteratively solving the quadratic cost function less than 20 times.

18. The method of claim 16, wherein calculating the initial alpha matte comprises automatically extracting a foreground from a background of the input image, and wherein the input image comprises a video frame.