Comprehensive method for removing from an image the background surrounding a selected subject

• US 6,134,345 A
• Filed: 08/28/1998
• Issued: 10/17/2000
• Est. Priority Date: 08/28/1998
• Status: Expired due to Term

• Current Assignee: iMatte, Inc.
• Original Assignee: Ultimatte Corporation (Blackmagic Design Pty Limited)
• Inventors: Dadourian, Arpag, Vlahos, Paul, Berman, Arie
• Primary Examiner(s): Tran, Phuoc
• Application Number:
  US09/141,703
• Time in Patent Office: 781 Days
• Field of Search: 382/162, 382/163, 382/167, 382/282, 348/584, 348/586, 348/587, 348/590, 348/591, 348/592, 348/650, 345/114, 345/473
• US Class Current: 382/162
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

In most cases, the estimated subject color and/or the estimated background color for a given pixel will have a certain amount of error, which could result in either not enough background removal, or removal of all of the background and some of the foreground. By manually or automatically altering the estimated subject color and/or background color, the errors in matte calculation and subsequent removal of the background, can be minimized or eliminated. Manual alteration of a subject or background color is achieved by using a cursor to select a true color from the background, for example, and inserting it in one or more areas in the background area being partially obscured by the subject. Automatic alteration of one of the estimated subject or background colors involves its computation as a projection in color space of the observed transition pixel color, onto a plane defined by the subject and background reference colors or other defined planes. By treating the shadow as a semitransparent subject, a matte (alpha channel) may be generated to represent the shadow'"'"'s transition densities, thus permitting the shadow to be regenerated over the new background.

Claims

1. A method to enable adjustment of estimates of subject and background colors used to compute the contribution of a background color to pixels in semitransparent and translucent subject areas, said method comprising the steps of:

a) receiving visually identified semitransparent and translucent subject areas where a first estimated background color contribution, when subtracted from an image, does not fully remove the background, or removes all of the background and some of the subject,

b) receiving a color selected from said subject or background based on a judgment by an operator as to whether at least one of;

i) a poor estimate of the background color, and

ii) a poor estimate of the subject color, is primarily responsible for generating an inaccurate matte in said identified semitransparent and translucent subject areas and storing said selected color in memory,

c) injecting said stored selected color, based on an input from the operator, into at least one point in at least one of

i) said judged subject, and

ii) said judged background, which when interpolated/extrapolated, provides an improved estimate of the selected subject or background color contributing to the color in a transition pixel.

2. The method of claim 1 in which selected colors are injected into both background and subject frames.

3. The method of claim 1 in which a color selected for injection may first be adjusted in luminance, saturation and hue, by adjusting its RGB levels.

4. The method of claim 1 in which a selected color is injected into a translucent area of the subject, perceived to represent an incorrect luminance, hue or saturation.

5. The method of claim 1 in which a color selected by the operator for injection may be selected from anywhere in the image.

6. The method of claim 1 in which a selected background color is injected into an extrapolated/interpolated background frame.

7. The method of claim 1 in which a selected subject color is injected into an extrapolated/interpolated subject frame.

8. The method of claim 1 in which selected colors are injected into selected areas prior to any computed estimates of subject reference, background reference, or matte, said selected colors and areas selected by visually identifying semitransparent and translucent subject areas where the anticipated estimated background color contribution, when subtracted from the image, is not expected to fully remove the background, or else is expected to remove all of the background and some of the subject.

9. A method for adjusting a reference color R of specified proportions of background and subject colors by enabling selection of, as the adjusted reference color of the same specified proportions, the projection of an observed color point on to a plane in color space which passes through the initial reference color point R and is perpendicular to the vector connecting that reference color point R with a reference color point S of other specified proportions.

10. The method of claim 9, but in which said reference color R is the background reference color and said reference color S is the subject reference color.

11. The method of claim 9, but in which said reference color R is the subject reference color and said reference color S is the background reference color.

12. The method of claim 9, but in which said projection of said observed color point is an orthogonal projection.

13. The method of claim 9, but in which said projection of said observed color point is a "straight" projection, i.e. it is a point on said plane which is equal to the observed color point in two of the dimensions or color components.

14. A method for adjusting a reference color R of specified proportions by selecting, as the adjusted reference color of the same specified proportions, a point on the ray emanating from a reference color point S of other specified proportions, and passing through the observed color point, and is further than the observed color point from the reference color point S.

15. The method of claim 14, but in which said reference color R is the background reference color and said reference color S is the subject reference color.

16. The method of claim 14, but in which said reference color R is the subject reference color and said reference color S is the background reference color.

17. The method of claim 14, but in which said adjusted reference color is the point of intersection of said ray and a surface enclosing all possible colors.

18. The method of claim 14, but in which said adjusted reference color is the point of intersection of said ray and the plane passing through the initial said reference color point R and perpendicular to the vector connecting that initial reference color point R with said reference color point S.

19. A method for extracting a shadow from an image, said method comprising the steps of:

a. receiving as an input a region of the image designated by an operator as the interior of the shadow, the fully shaded or shadowed region;

b. receiving as an input a region of the image designated by an operator as being clear of shadow, and being a pure background region;

c. estimating an interior of shadow color for each point in the image by interpolating/extrapolating colors from the designated interior of shadow region into the rest of the image area, optionally excluding the clear of shadow area;

d. estimating a clear of shadow color for each point in the image by interpolating/extrapolating colors from the designated clear of shadow region into the rest of the image area, and selectively excluding the interior of shadow area;

e. for each point or pixel position in the image, estimating a matte, or percentage of background color present in the pixel, by using an original observed color v, the designated or estimated interior of shadow color vs, and the designated or estimated clear of shadow color vc, by selecting from or combining matte estimates of the form

h(v-vs)/h(vc-vs) where h is a linear function.

20. The method of claim 19 further comprising the step of remapping the matte, or shadow, levels.

21. The method of claim 20, wherein said mapping is a linear mapping.

22. The method of claim 21, wherein said linear mapping effects a gain on the difference from 100%.

23. The method of claim 22, wherein the amount of gain is estimated from the original image as the complement of the ratio of the brightness of pixels in the interior region of the shadow to the brightness of neighboring pixels in the clear of shadow region.

24. The method of claim 19, wherein the image presented for shadow extraction is a clear background image produced from a prior extraction of another subject.

25. The method of claim 19, further comprising the step of removing the estimated background, or clear of shadow color, from each pixel, in accordance with, or in proportion indicated by, said estimated matte.