Method and apparatus for the automatic generation of subject to background transition area boundary lines and subject shadow retention

• US 6,667,774 B2
• Filed: 11/02/2001
• Issued: 12/23/2003
• Est. Priority Date: 11/02/2001
• Status: Expired due to Term

• Current Assignee: iMatte, Inc.
• Original Assignee: iMatte, Inc.
• Inventors: Dadourian, Arpag, Vlahos, Paul, Berman, Arie
• Primary Examiner(s): Lee, Michael H.
• Assistant Examiner(s): DESIR, JEAN WICEL
• Application Number:
  US10/008,868
• Publication Number:
  US 20030086018A1
• Time in Patent Office: 781 Days
• Field of Search: 348/586, 348/584, 348/587, 348/598, 348/592, 348/590, 348/650, 348/252, 348/577, 348/625, 348/715, 382/167, 382/162, 382/163, 358/518, 358/520
• US Class Current: 348/584
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/275 Generation of keying signals
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

Changes in a current image frame from variations in room light level, and automatic camera adjustments, are determined and incorporated into a clear (reference) frame, thereby resulting in a zero difference in the background region when the clear frame and a current frame are subtracted. Any deviation from zero defines the beginning edge of a transition from background to subject. The location of the outer edge of a transition area is further refined by subtracting derivatives of the RGB signals in each frame, and by the use of a fourth color channel when available. The difference of derivitives (d_cu-d_cl) identifies the subject to background transition area, whose edges define the inner and outer boundary lines. Shadow areas on the backing are retained and may be transferred to the inserted background scene. Small areas on the backing having the same color and luminance as the subject are prevented from printing through the subject by a second “derivative” alpha channel for the small areas.

Claims

1. A method for compositing motion picture and video images by automatically generating inner and outer boundary lines on either side of a subject-to-background transition area comprising the steps of:

a) storing in a first memory the signal levels of each signal channel comprising the signals at each pixel location in a clear image frame with a subject absent,

b) storing in a second memory the signal levels of each signal channel comprising the signals at each pixel location in a current image frame with said subject in place,

c) subtracting the signal levels of pixels in said clear image frame from signal levels of corresponding pixels in said current image frame to obtain their difference,

d) storing in a third memory a 0.0 at all pixel addresses where said difference approximates zero, and storing a 1.0 for all remaining pixels to form a field of zeros in the background area and a field of ones in the subject and transition areas,

e) defining an outer boundary line as the addresses of those pixels forming a continuous line of pixels having been assigned a 0.0 and located adjacent to pixels having been assigned a 1.0, thereby forming an outer boundary line on the background adjacent to the edge of the subject-to-background transition area,

f) shrinking said outer boundary line inward to form an inner boundary line on the subject at a sufficient distance from said subject'"'"'s edge to avoid including elements of the background, thereby automatically generating said inner and outer boundary lines.

2. The method of claim 1 in which the signal channels comprising the signals being stored in said clear frame and said current frame are red, green and blue.

3. The method of claim 1 in which the signal channels comprising the signals being stored for said clear frame and said current frame are at least one red, green, blue, infrared red and ultra violet.

4. The method of claim 1 in which signal levels of pixels in dark areas in the background of the current frame are subtracted from the signal levels of corresponding pixels in the clear frame to obtain a correction to the clear frame that will reduce to zero any black level offset introduced by automatic camera adjustments or changes in scene illumination.

5. The method of claim 4 in which said dark areas are pixels having signal levels below a predetermined level and are identified by counters and accumulators seeking the largest areas having consistent differences, with the average of said differences becoming said clear frame corrections.

6. The method of claim 1 in which RGB signal levels in high luminance areas in the background of the current frame are divided by RGB signal levels from corresponding pixels in the clear frame to obtain clear frame correction factors that will reduce to zero any difference in the RGB signal levels between pixels in the background of the current frame and corresponding pixels in the clear frame thus removing the effects of luminance and color changes caused by changes in scene illumination or automatic camera adjustments.

7. The method of claim 6 in which pixels in high luminance areas are defined as having an R,G or B signal above a predetermined level, and are identified by a histograming process that seeks the most consistent ratio values, which are then averaged to reduce the effect of random signal noise.

8. The method of claim 1 in which said inner boundary line on the opaque subject is spaced from the outer boundary line by the width of the transition area.

9. The method of claim 8 in which the width of the transition area at any point along the edge of the subject is the positive difference of the derivative of the clear frame subtracted from the derivative of the current frame, d_cu-d_cl, which defines the width and location of the transition area.

10. The method of claim 9 in which said derivatives of said clear frame and current frame signals are determined by substituting for the signals at the central pixel of a matrix of nine pixels the signal difference between the lowest and the highest signal levels of the nine pixels.

11. The method of claim 1 in which the difference obtained by subtraction of clear frame signals from current frame signals is considered to be zero when said difference is less than a selected threshold and where said threshold approximates the combined noise of the camera and digitizing process.

12. The method of claim 11 in which white specks (1.0) in a field of black (0.0), representing noise peaks exceeding said noise threshold, are removed by substituting the smallest value of a 3×

3 matrix of nine pixels for the existing value of the current pixel, which is at the center of the matrix.

13. The method of claim 12 in which a one pixel shrinkage of the subject area, that occurred when removing said noise peaks, may be expanded by one pixel while simultaneously removing black specks in a white subject field by substituting for the central pixel of a 3×

3 matrix a 1.0 if any of the nine matrix pixels is not zero.

14. The method of claim 1 in which the colors in a clear frame are conformed to the colors in the background of a current frame, including the background color component of transition areas in the current frame, by multiplying each pixel in the clear frame by a correction factor computed for each pixel position, said correction factor being a ratio of a pixel color from a synthetic clear frame I_cu to a corresponding pixel color from an interpolated clear frame I_cl.

15. The method of claim 14 in which a representation of the luminance of each pixel in synthetic clear frame Icu is divided by a corresponding representation of luminance of a corresponding pixel in interpolated clear frame Icl, forming an estimate of shadow luminance for each pixel, said shadow luminance limited to 1.0.

16. The method of claim 15 in which said shadow area and density, Sh, is obtained by one of equations 8, 9 as follows;

17. The method of claim 14 in which said clear (cl) frame is multiplied by the ratio (I _cu/I_cl) to form a corrected clear frame that includes shadows extending behind a subject'"'"'s transition area, and said corrected clear frame constitutes a background reference frame.

18. The method of claim 1 in which the subtraction in step 1.c) and the assignments in step 1.d) are replaced by alternate methods of comparing the clear frame and the current frame to determine the foreground region, such as by computing a preliminary alpha and processed foreground and assigning a 1.0 to any pixel for which the preliminary processed foreground exceeds some low threshold, or for which the preliminary alpha is below another low threshold, and assigning a 0.0 to all other pixels, and where said preliminary alpha and said computation of said processed foreground and preliminary alpha comprise the following steps,

a) estimating the similarity of colors between corresponding pixels in the clear frame and the current frame, on a scale ranging from 0.0 to 1.0,

b) subtracting the product of the preliminary alpha and the clear frame color from the current frame color, forming said preliminary processed foreground.

19. A system for compositing motion picture and video images by automatically generating inner and outer boundary lines on either side of a subject-to-background transition area comprising:

a) a first memory for storing the signal levels of each signal channel comprising the signals at each pixel location in a clear image frame with a subject absent,

b) a second memory for storing the signal levels of each signal channel comprising the signals at each pixel location in a current image frame with said subject in place,

c) a comparator coupled to said first memory and said second memory for subtracting the signal levels of pixels in said clear image frame from signal levels of corresponding pixels in said current image frame to obtain their difference,

d) a third memory coupled to said comparator for storing a 0.0 at all pixel addresses where said difference approximates zero, and storing a 1.0 for all remaining pixels to form a field of zeros in the background area and a field of ones in the subject and transition areas,

e) means for defining an outer boundary line as the addresses of those pixels forming a continuous line of pixels having been assigned a 0.0 and located adjacent to pixels having been assigned a 1.0, thereby forming an outer boundary line on the background adjacent to the edge of the subject-to-background transition area,

f) means for shrinking said outer boundary line inward to form an inner boundary line on the subject at a sufficient distance from said subject'"'"'s edge to avoid including elements of the background, thereby automatically generating said inner and outer boundary lines.