Method and apparatus for extracting and treating digital images for seamless compositing

• US 5,630,037 A
• Filed: 05/18/1994
• Issued: 05/13/1997
• Est. Priority Date: 05/18/1994
• Status: Expired due to Fees

• Current Assignee: Schindler Imaging, Inc.
• Original Assignee: Schindler Imaging, Inc.
• Inventors: Schindler, Keith W.
• Primary Examiner(s): Zimmerman, Mark K.
• Application Number:
  US08/245,377
• Time in Patent Office: 1,091 Days
• Field of Search: 395/131, 348/578, 348/591, 348/592, 348/594, 348/598, 382/163, 382/164, 382/168, 382/171, 382/173, 382/282, 382/283
• US Class Current: 345/592
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof

Abstract

An image processing system for extracting and treating a digitized color subject image for seamless compositing against an arbitrary background includes the generation of a final control image from the image of a subject in front of a screen consisting of a range of colors. A sample of the background screen is captured and analyzed using filter kernels to determine a range of red, green and blue values which are stored in separate histograms. A first control image is generated by comparing mean histogram data with the subject image. A final control image is generated by dividing the first control image into separate background, foreground and fringe regions. A treated subject image is generated by leaking color from the local background region into the fringe regions of the subject image using filter kernels. The treated image is suitable for compositing against an arbitrary background image. The composite image may be touched up by using a software brush which on each pass of the brush expands the background region by eliminating immediately adjacent fringe pixels, causes other nearby fringe pixels to increase in translucence, and allows the boundary of the fringe region to encroach on the foreground region. Simultaneously, the brush may also cause fringe pixels to absorb some of the color of neighboring foreground pixels.

Claims

1. A method for extracting a digitized foreground image for compositing with an arbitrary background image, said method comprising the steps of:

acquiring an image comprising a foreground subject and a background,

obtaining data representing coloration of said background, said data comprising a plurality of values corresponding to color intensities,

generating from said data a statistical range of color intensities for each of a selected group of colors,

deriving an indication of average background color from said statistical range of color intensities,

differentiating said foreground subject from said background by comparing color intensity values of said foreground subject with said indication of average background color,

generating a control image comprised of values representing a distance in color of each color intensity value in said foreground subject from said average background color, and

generating a second control image by dividing said first control image into a background region, a fringe region of variable width, and a foreground region, and mixing color into foreground pixels at locations corresponding to said fringe region of said second control image from nearby background pixels at locations corresponding to said background region of said second control image,

wherein said second control image comprises a plurality of values corresponding to pixels at specific locations, and wherein said step of mixing color into foreground pixels at locations corresponding to said fringe region of said second control image from nearby background pixels at locations corresponding to said background region of said second control image comprises the steps of (a) identifying a pixel in said fringe region bordered by pixels from said background region, (b) changing a color of the corresponding pixel in said foreground image to reflect an average color of a plurality of background pixels in locations corresponding to said background region near said pixel, (c) associating said pixel no longer with said fringe region but instead with said background region, and (d) repeating steps (a) through (c) for a plurality of pixels in said fringe region.

2. The method of claim 1 wherein said plurality of pixels in step (d) comprises all of the pixels in the fringe region.

3. The method of claim 1 wherein said step of changing a color of the corresponding pixel in said foreground image comprises the step of calculating an average color of background pixels in a predefined square region around said pixel.

4. The method of claim 1 further comprising the step of generating a treated image by removing color pixels whose values correspond to the background coloration from said foreground subject, determining neighboring color pixel values for selected remaining portions of said foreground subject, and mixing local background coloration into said selected remaining portions on a pixel-by-pixel basis.

5. The method of claim 4 wherein said step of mixing color into foreground pixels at locations corresponding to said fringe region of said second control image from nearby background pixels at locations corresponding to said background region of said second control image comprises the step of generating an intermediate image by mixing color with selected fringe pixels from neighboring pixel locations, and wherein said step of removing color pixels whose values correspond to the background coloration from said foreground subject comprises the step of calculating a difference between fringe pixels in said intermediate image and fringe pixels in said image.

6. The method of claim 4 further comprising the step of generating a composite image by combining said treated image with a selected background image.

7. The method of claim 6 wherein said second control image comprises an alpha channel, and wherein said step of combining said treated image with a selected background image comprises the step of mixing portions of said treated image and said selected background image in a ratio determined by said alpha channel.

8. The method of claim 6 wherein said step of combining said treated image with a selected background image results in said composite image being the same as said selected background image in regions where said second control image has a minimum value, the same as said treated image in regions where said second control image has a maximum value, and a mixture of said selected background image and said treated image in regions where said second control image has a value between said minimum value and said maximum value.

9. A method for touching up a digitally composited image comprising the steps of:

selecting a region of a composited image, said composited image comprising a foreground image and a background image bordered by a fringe region of variable width, wherein said step of selecting a region comprises the step of interactively placing a brush on a screen displaying said composited image,

in a portion of said selected region corresponding to said fringe region and immediately bordered at least in part by said foreground region, replacing some of the color pixel values in said portion with color pixel values that are closer in proximity, within a color space, to said foreground region, and

mixing a remaining color in said portion with a color of said foreground region.

10. The method of claim 9 wherein said step of placing a brush is carried out with a stylus, pen or mouse.

11. A method for processing a digitized subject image, said method comprising the steps of:

acquiring an image of a subject in front of a background, said image comprising a plurality of color values at each of a plurality of pixel locations,

measuring the color of said background by obtaining a plurality of color values at each of a plurality of pixel locations in said background,

generating a plurality of histograms, one for each color of a select group of colors, from said color values of said background,

calculating an average background color from said plurality of histograms,

generating a first control image by comparing said average background color and said color values of said image,

dividing said first control image into a plurality of regions so as to generate a second control image, said second control image defining a background region, a fringe region, and a foreground region, and

modifying color values in a portion of said image corresponding to said fringe region,

wherein each of said histograms comprises a plurality of average color values, said step of generating a plurality of histograms comprising the step of deriving each of said average color values by averaging the color values at a particular pixel location with the color values of a predefined group of surrounding pixels.

12. A method for processing a digitized subject image, said method comprising the steps of:

acquiring an image of a subject in front of a background, said image comprising a plurality of color values at each of a plurality of pixel locations,

measuring the color of said background by obtaining a plurality of color values at each of a plurality of pixel locations in said background,

generating a plurality of histograms, one for each color of a select group of colors, from said color values of said background,

calculating an average background color from said plurality of histograms,

generating a first control image by comparing said average background color and said color values of said image,

dividing said first control image into a plurality of regions so as to generate a second control image, said second control image defining a background region, a fringe region, and a foreground region,

modifying color values in a portion of said image corresponding to said fringe region, and

adjusting a location of a first frontier between said background region and said fringe region, and a location of a second frontier between said fringe region and said foreground region.

13. A method for processing a digitized subject image, said method comprising the steps of:

acquiring an image of a subject in front of a background, said image comprising a plurality of color values at each of a plurality of pixel locations,

measuring the color of said background by obtaining a plurality of color values at each of a plurality of pixel locations in said background,

generating a plurality of histograms, one for each color of a select group of colors, from said color values of said background,

calculating an average background color from said plurality of histograms,

generating a first control image by comparing said average background color and said color values of said image,

dividing said first control image into a plurality of regions so as to generate a second control image, said second control image defining a background region, a fringe region, and a foreground region, and

modifying color values in a portion of said image corresponding to said fringe region, said step of modifying a portion of said image comprising the step of generating an intermediate image by adding color from said background region to said fringe region,

wherein said step of generating said intermediate image comprises the steps of (a) identifying a pixel in said fringe region bordered by at least one pixel in said background region, (b) changing a color of said fringe pixel to reflect an average color of a plurality of pixels in said background region, (c) associated said fringe pixel no longer with said fringe region but instead with said background region, and (d) repeating steps (a) through (c) for a plurality of pixels in said fringe region.

14. The method of claim 13 wherein said plurality of pixels in step (d) comprises all of the pixels in said fringe region.

15. The method of claim 13 wherein the step of changing a color of said fringe pixel comprises the step of calculating an average color of background pixels in a predefined square region around said fringe pixel.

16. A method for processing a digitized subject image, said method comprising the steps of:

acquiring an image of a subject in front of a background, said image comprising a plurality of color values at each of a plurality of pixel locations,

measuring the color of said background by obtaining a plurality of color values at each of a plurality of pixel locations in said background,

generating a plurality of histograms, one for each color of a select group of colors, from said color values of said background,

calculating an average background color from said plurality of histograms,

generating a first control image by comparing said average background color and said color values of said image,

dividing said first control image into a plurality of regions so as to generate a second control image, said second control image defining a background region, a fringe region, and a foreground region,

modifying color values in a portion of said image corresponding to said fringe region, said step of modifying a portion of said image comprising the step of generating an intermediate image by adding color from said background region to said fringe region, and

generating a treated image by removing at least a portion of said background color from said image, determining neighboring color pixel values in the remaining portions of said image, and leaking a local background color from adjacent background pixels into boundary pixels of each of said remaining portions.

17. The method of claim 16 wherein said step of removing at least a portion of the background color comprises the step of calculating a difference between values of said intermediate image and values of said image.

18. The method of claim 16 further comprising the step of generating a composite image by combining said treated image with a selected background image.

19. The method of claim 18 wherein said second control image comprises an alpha channel, and wherein said step of combining said treated image with a selected background image comprises the step of mixing portions of said treated image and said selected background image in a ratio determined by said alpha channel.

20. The method of claim 18 wherein said step of combining said treated image with a selected background image results in said composite image being the same as said selected background image in regions where said second control image has a minimum value, the same as said treated image in regions where said second control image has a maximum value, and a mixture of said selected background image and said treated image in regions where said second control image has value between said minimum value and said maximum value.

21. The method of claim 18 further comprising the step of touching up said composite image.