Method for removing from an image the background surrounding a selected object

• US 6,134,346 A
• Filed: 01/16/1998
• Issued: 10/17/2000
• Est. Priority Date: 01/16/1998
• Status: Expired due to Term

• Current Assignee: iMatte, Inc.
• Original Assignee: iMatte, Inc.
• Inventors: Dadourian, Arpag, Vlahos, Paul, Berman, Arie
• Primary Examiner(s): Tran, Phuoc
• Assistant Examiner(s): Mariam, Daniel G.
• Application Number:
  US09/008,270
• Time in Patent Office: 1,005 Days
• Field of Search: 382/162, 382/163, 382/167, 348/577, 348/586, 348/587, 358/519, 358/521, 345/114, 345/336
• US Class Current: 382/163
• CPC Class Codes: H04N 1/3872 Repositioning or masking
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

A computer implemented method to extract a selected subject from its background, by removing the background, including that portion of the background visible through semi transparent areas of the subject, and generating a matte signal containing a record of background levels outside of and within semitransparent subject areas. The observed RGB signal levels of a pixel in the semitransparent transition between a subject and its background, are a mixture of color contributed by the subject, and by the background. The estimated subject color, and the estimated background color, and the observed color of a transition pixel (pixRGB), may be shown as three points in a three dimensional color space.

Claims

1. A method for extracting a selected subject from an image, including those portions of the subject that are semitransparent, by removing all elements of the background surrounding the subject, and also removing those background elements visible through semitransparent areas of the subject, where said method comprises the steps of;

a) drawing, on a display of said image, an inner boundary line on the subject near to an edge of said subject,

b) drawing an outer boundary line on said display, on the background near to the subject, but spaced at a distance sufficient to avoid inclusion of any element of said subject,

c) generating a Clear Frame by filling the area inside said outer boundary line with RGB levels generated by interpolation and/or extrapolation of the RGB levels existing on and/or outside said outer boundary line,

d) generating an Expanded Subject Frame by replacing the RGB levels in the area bounded by said inner and outer boundary lines with RGB levels generated by extrapolation of the RGB levels existing on and/or inside said inner boundary line,

e) calculating an estimated Percent Background RGB Contribution (Matte Signal) to each pixel in the bounded area using equation;

space="preserve" listing-type="equation">percentage background contribution=(F(pixRGB)-F(subRGB)) / (F(bgRGB)-F(subRGB)),

where;

F is any linear function of the form, F(RGB)=k₁ R+k₂ G+k₃ B,

pixRGB is observed RGB at a pixel at a given address in the bounded area,

subRGB is RGB obtained from the Expanded Subject Frame at said given address,

bgRGB is RGB obtained from the Clear Frame at said given address,

f) generating a Matte Frame by assigning a matte value of 1.0 to all pixels outside said outer boundary line, and assigning a matte value of 0.0 to all pixels inside said inner boundary line, and including the bounded area matte signal of step `e` to complete the full Matte Frame signal,

g) multiplying RGB levels at each pixel in the Clear Frame by the matte signal at the corresponding pixel in the Matte Frame to form their product,

h) subtracting said product from the RGB levels of the original image to result in a Processed Foreground image of the subject, where said subject appears to have been photographed against a black backing, with no residual elements of the background remaining in the background nor in semitransparent areas of the subject.

2. The method of claim 1 in which the generation of the Expanded Subject Frame is limited to the bounded area.

3. The method of claim 1 in which the generation of said Clear Frame is limited to that portion of the frame within the bounded area.

4. The method of claim 1 in which the function F(RGB) employed in calculating the percent contribution of the background is one or more of the following:

F₁ (RGB)=B,

F₂ (RGB)=G

F₃ (RGB)=R

F₄ (RGB)=B-G

F₅ (RGB)=B-R

F₆ (RGB)=G-R.

5. The method of claim 1 in which the percentage contribution of the background to a pixel in a transition area is obtained from a ratio, and said ratio is computed for the functions

F₁ (RGB)=B,

F₂ (RGB)=G

F₃ (RGB)=R

F₄ (RGB)=B-G

F₅ (RGB)=B-R

F₆ (RGB)=G-R

and said percentage background contribution is a weighted sum of said ratios, and said weighting increases as a absolute difference of the denominator of said ratio increases.

6. The method of claim 1 in which the Matte signal in the bounded area is that calculation that results in a largest denominator when calculating percent contribution of the background.

7. The method of claim 1 in which said semitransparent subject area is defined as those pixels within the bounded area where said percent contribution of the background is greater than zero and less than 1.0.

8. The method of claim 1 in which said estimated foreground and background colors at corresponding positions in the Clear Frame and Expanded Subject Frame are assigned by an operator.

9. The method of claim 8 in which the assigned foreground and assigned background colors are those existing at the inner and outer boundary lines where a scan line crosses said boundary lines and also crosses said given pixel in the bounded area.

10. A method for extracting a selected subject from an image, including those portions of the subject that are semitransparent, by removing all elements of the background surrounding the subject, and also removing those background elements visible through semitransparent areas of the subject, when said background is busy, detailed, and in sharp focus, where said method comprises the steps of;

a) drawing, on a display of said image, an inner boundary line on the subject near to a edge of said subject,

b) drawing an outer boundary line on said display, on the background near to the edge of said subject but spaced at a distance sufficient to avoid inclusion of any element of said subject,

c) generating a Clear Frame by filling the area inside said outer boundary line with RGB levels generated by interpolation and/or extrapolation of the RGB levels existing on and/or outside said outer boundary line,

d) generating an Expanded Subject Frame by replacing the RGB levels in the area bounded by said inner and outer boundary lines with RGB levels generated by extrapolation of the RGB levels existing on and/or inside said inner boundary line,

e) calculating M1, an estimated Percent Background RGB Contribution to each pixel in the bounded area using the following equation;

space="preserve" listing-type="equation">percentage background contribution=(F(pixRGB)-F(subRGB)) / (F(bgRGB)-F(subRGB)),

where;

percentage background contribution is defined as matte signal M1,

F is any linear function of the form,

F(RGB)=k₁ R+k₂ G+k₃ B,

pixRGB is observed RGB at a pixel at a given address in the bounded area,

subRGB is RGB obtained from the Expanded Subject Frame at said given address,

bgRGB is RGB obtained from the Clear Frame at said given address,

f) subtracting said matte signal M1 of step "e" from 1.0 to generate a Reversed Matte signal (1-M1),

g) generating Processed Foreground 1, (PrFg1), where, PrFg1=(1-M1)*subRGB

h) generating Processed Background 1, (PrBg1), where PrBg1=pixRGB-PrFg1

i) generating the reversed Clear Frame (bgRGB2), where reversed Clear Frame bgRGB 2=(1-M1)*bgRGB

j) generating the Reversed Clear Frame bgRGB 3, where bgRGB 3=PrBg1+bgRGB 2

k) generating the Final Matte M2, where ##EQU6## l) generating Processed Foreground 2PrFg 2, where PrFg 2=pixRGB-(M2*bgRGB3)

and said Processed Foreground 2 is the subject as though photographed against a black backing, with no remaining elements of a detailed, sharply focused background surrounding the subject nor any residual background in semitransparent areas of the subject.

11. The method of claim 10, in which the reversed matte (1-M1) is calculated directly by using the equation:

##EQU7##

12. A method for determining, in an image, an estimated contribution of background color to an observed color at a given pixel in a semitransparent area of a foreground subject, when the color of the background near said pixel in said semitransparent area is known, declared, or can be estimated, and the color of the subject near said pixel in said semitransparent area is known, declared, or can be estimated, said method comprising the steps of:

a) representing said known, declared, or estimated background color RGB levels (bgRGB) as a point (9) in a three dimensional color space,

b) representing said known, declared or estimated foreground color RGB levels (subRGB) as a second point (11) in said color space,

c) representing said given pixel observed color RGB levels (pixRGB) as a third point (10) in said color space,

d) calculating an estimate of the percent contribution of the background color to the observed color of said given pixel by determining the ratio of the distance, in color space, from pixRGB to subRGB, divided by the distance, in color space, from bgRGB to subRGB, where said calculation of distance utilizes the numerical values of at least one linear functions of F(RGB), where F(RGB)=k₁ R+k₂ G+k₃ B.

13. The method of claim 12 in which the function F(RGB) is at least one of the following:

F₁ (RGB)=B,

F₂ (RGB)=G

F₃ (RGB)=R

F₄ (RGB)=B-G

F₅ (RGB)=B-R

F₆ (RGB)=G-R.

14. The method of claim 12 in which a percentage contribution of the background to an observed pixel in the transition area is obtained from a ratio, and said ratio is computed for the functions:

F₁ (RGB)=B,

F₂ (RGB)=G

F₃ (RGB)=R

F₄ (RGB)=B-G

F₅ (RGB)=B-R

F₆ (RGB)=G-R.

and said percentage background contribution is a weighted sum of said ratios, and said weighting increases as a absolute difference of a denominator of said ratio increases.

15. The method of claim 12 in which said semitransparent subject area is defined as those pixels, anywhere in the image frame, whose said percent contribution of the background is determined to be greater than zero and less than 1.0.

16. The method of claim 12 in which said estimated foreground and background colors near said semitransparent area are assigned by an operator.

17. The method of claim 12 in which the Matte signal in the bounded area is that calculation that results in a largest denominator when calculating percent contribution of the background.

18. The method of claim 12 in which the bgRGB, pixRGB and subRGB are represented as points in a two dimensional color space.

19. The method of claim 18 in which the two dimensional color space is a Commision internationale de l'"'"'Eclairage (CIE) chromaticity diagram.