Real-Time Chromakey Matting Using Image Statistics

• US 20100277471A1
• Filed: 04/01/2010
• Published: 11/04/2010
• Est. Priority Date: 04/01/2009
• Status: Active Grant

• Current Assignee: University of Central Florida Research Foundation Inc. (State University System of Florida)
• Original Assignee: University of Central Florida Research Foundation Inc. (State University System of Florida)
• Inventors: Yamazawa, Kazumasa, Zhang, Yunjun, Hughes, Charles E., Beato, Nicholas, Colbert, Mark
• Granted Patent: US 8,477,149 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

A method, system and computer readable media for real-time chromakey matting using image statistics. To identify the chroma key spectrum, the system/method executes in three stages. In an off-line training stage, the system performs semi-automatic calibration of the chroma key parameterization. In the real-time classification stage, the system estimates the alpha matte on a GPU. Finally, an optional error minimization stage improves the estimated matte, accounting for misclassifications and signal noise. Given the resulting matte, standard alpha blending composites the virtual scene with the video feed to create the illusion that both worlds coexist.

Claims

1. A method for generating an alpha matte based on chromakey information in real-time, comprising:

(a) off-line training comprising calibration of chroma key parameterization;

(b) real-time classification via a processor comprising estimating an alpha matte;

(c) outputting a resulting alpha matte to allow for alpha blending to composite a virtual scene with a video feed.

2. The method of claim 1 further comprising the step of error minimization for improving the alpha matte comprising accounting for misclassifications and signal noise prior to outputting the resulting alpha matte.

3. The method of claim 1 wherein the off-line training comprises:

(a) randomly sampling chroma color from several viewing angles and positions using a target camera to produce N triplet data points representing a data set of pixels;

(b) running principle components analysis (PCA) using the N triplet data points as input to calculate eigenvectors having nine values and three three-dimensional (3D) vectors, eigenvalues having three values, and means having three values of the data set;

(c) constructing a whitening function from a mean vector (μ

), column-major matrix of eigenvectors (E=[e₁ e₂ e₃]), and diagonal matrix of eigenvalues (Λ

=diag (λ

₁, λ

₂, λ

₃)) comprising

M(I_x)=Λ

<sup>−

1/2</sup>E^T(I_x−

μ

); and

(d) outputting a triplet of a value.

4. The method of claim 3 wherein the real-time classification comprises:

(a) for all pixels, p, in a video frame;

(i) calculate z=M(p), wherein z is a triplet representing standard deviations in three independent dimensions of pixel p in chroma color distribution;

(ii) receive a selection of a classifier function, C, and compute α

=C(z), where α

is the alpha channel of the pixel p;

(b) outputting the alpha matte.

5. The method of claim 4 wherein the classifier comprises a sphere or a cube.

6. The method of claim 4 wherein the classifier comprises a sphere wherein data distribution is a 3D Gaussian, comprising:

(a) calculating a 3D distance to a mean in a transformed color space wherein d=magnitude(z);

(b) using the data distribution to find a lower and upper bound on single dimension z-score wherein d is used with the upper and lower bounds; and

(c) calculating the alpha by linearly interpolating d between the lower and upper bound.

7. The method of claim 6 wherein for a given transformed pixel, I′

_x, using Euclidean distance from an origin with linear interpolation between an inner and outer radii, r_in and r_out calculating opacity, α

_x, as;

8. The method of claim 4 wherein the classifier comprises a cube having each dimension of the data distribution follows a Gaussian distribution, comprising:

(a) calculating each 1D distance to a mean in a transformed color space for each dimension (d_i=abs(z_i));

(b) assuming the worst z score (d_i) encloses the data distribution best, reducing dimensionality by taking a maximum distance across the first two or all three dimensions;

(c) using the data distribution, finding a lower and upper bound on single dimension z-score; and

(d) calculating the alpha by linearly interpolating d_max between the lower and upper bound.

9. The method of claim 8 wherein, for a given transformed pixel, I′

_x, given an inner and outer axis-aligned bounding cube of sizes d_in and d_out, use a projected Euclidean distance from each plane to calculate an opacity, α

_x, as;

10. The method of claim 1 wherein the processor is a graphics processing unit (GPU) operating at high frame rates.

11. The method of claim 10 further comprising:

given four images of foreground, background, video, and matte, compositing the images using a three layer blending function.

20. A computer system for an alpha matte based on the chromakey information in real-time configured to cause one or more computer processors to perform the steps recited in claim 1.

12. A system for generating an alpha matte based on chromakey information in real-time, said system comprising:

a display device;

a memory; and

a processor with hardware and software components for;

(a) off-line training comprising calibration of chroma key parameterization;

(b) real-time classification via a processor comprising estimating an alpha matte;

(c) outputting a resulting alpha matte to allow for alpha blending to composite a virtual scene with a video feed.

13. The system of claim 12 further comprising a software component for error minimization for improving the alpha matte comprising accounting for misclassifications and signal noise prior to outputting the resulting alpha matte.

14. The system of claim 12 wherein the off-line training comprises:

(a) randomly sampling chroma color from several viewing angles and positions using a target camera to produce N triplet data points representing a pixel in a data set;

(b) running principle components analysis (PCA) using the N triplet data points as input to calculate eigenvectors having nine values and three three-dimensional (3D) vectors, eigenvalues having three values, and means having three values of the data set;

(c) constructing a whitening function from a mean vector (μ

), column-major matrix of eigenvectors (E=[e₁ e₂ e₃]), and diagonal matrix of eigenvalues (Λ

=diag (λ

₁, λ

₂, λ

₃)) comprising

M(I_x)=Λ

<sup>−

1/2</sup>E^T(I_x−

μ

); and

(d) outputting a triplet of a non-negative value.

15. The system of claim 14 wherein the real-time classification comprises:

(a) for all pixels, p, in a video frame;

(i) calculate z=M(p), wherein z is a triplet representing standard deviations in three independent dimensions of pixel p in chroma color distribution;

(ii) receive a selection of a classifier function, C, and compute α

=C(z), where α

is the alpha channel of the pixel p;

(b) outputting the alpha matte.

16. An article of manufacture comprising:

a computer usable medium having computer readable code means embodied therein for;

(a) off-line training comprising calibration of chroma key parameterization;

(b) real-time classification via a processor comprising estimating an alpha matte;

(c) outputting a resulting alpha matte to allow for alpha blending to composite a virtual scene with a video feed.

17. The article of manufacture of claim 16 further comprising computer readable code means for error minimization for improving the alpha matte comprising accounting for misclassifications and signal noise prior to outputting the resulting alpha matte.

18. The article of manufacture of claim 16 wherein the computer readable code means for off-line training comprises:

(a) randomly sampling chroma color from several viewing angles and positions using a target camera to produce N triplet data points representing a pixel in a data set;

(b) running principle components analysis (PCA) using the N triplet data points as input to calculate eigenvectors having nine values and three three-dimensional (3D) vectors, eigenvalues having three values, and means having three values of the data set;

(c) constructing a whitening function from a mean vector (μ

), column-major matrix of eigenvectors (E=[e₁ e₂ e₃]), and diagonal matrix of eigenvalues (Λ

=diag (λ

₁, λ

₂, λ

₃)) comprising

M(I_x)=Λ

<sup>−

1/2</sup>E^T(I_x−

μ

); and

(d) outputting a triplet of a value.

19. The article of manufacture of claim 18 wherein the computer readable code means for real-time classification comprises:

(a) for all pixels, p, in a video frame;

(i) calculate z=M(p), wherein z is a triplet representing standard deviations in three independent dimensions of pixel p in chroma color distribution;

(ii) receive a selection of a classifier function, C, and compute α

=C(z), where α

is the alpha channel of the pixel p;

(b) outputting the alpha matte.