Apparatus and method for removing background on visual

• US 6,996,272 B2
• Filed: 07/11/2002
• Issued: 02/07/2006
• Est. Priority Date: 04/23/2002
• Status: Expired due to Fees

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Liu, Cheng-Yi, Chen, I-Yen, Liu, Jau-Fu, Chen, Jiann-Jone
• Primary Examiner(s): Bali, Vikkram
• Assistant Examiner(s): LAROSE, COLIN M
• Application Number:
  US10/193,393
• Publication Number:
  US 20030198382A1
• Time in Patent Office: 1,307 Days
• Field of Search: 382/103, 382/162, 382/163, 382/164, 382/173, 382/282, 382/283, 358/538, 345/634, 345/635
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

A background removal apparatus comprises a color normalization module that normalizes an original image, and an image segmentation module that segments the normalized image into several segmented regions. A background mesh generation module simulates the variation of pixel colors in background and generates an interpolated background mesh. A comparison module compares the normalized image and the interpolated background mesh to form a background mask by extracting the coherent regions between them. A refinement module uses a refined rule to determine a final background mask, and a background removal module through which a pure foreground image is obtained. The apparatus combines the efficiency of both color and spatial clustering, and improves the capabilities of current image segmentation method to perform background removal.

Claims

1. An apparatus for removing the background of visual content, said apparatus comprising:

a color normalization module normalizing an original image;

an image segmentation module segmenting said normalized image into a segmented image;

a background mesh generation module simulating the variation of pixel colors in the background of said normalized image and generating an interpolated background mesh;

a comparison module comparing said normalized image with said interpolated background mesh to generate a background mask, said background mask being formed by identifying and extracting coherent regions between said normalized image and said interpolated background mesh;

a refinement module generating a refined background mask according to said segmented image and said background mask; and

a background removal module generating a foreground image by masking out background from said original image according to said refined background mask;

wherein said background mesh generation module further comprises;

a resolution reduction sub-module reducing the resolution of said normalized image to form a reduced image, each pixel in said reduced image being regarded as one block;

a border color averaging sub-module computing an average color vector for border blocks of said reduced image;

a comparison sub-module locating initial background blocks on said reduced image based on a first decision rule;

a growing sub-module repeatedly extending said initial background blocks towards neighboring pixels on said reduced image to generate final background blocks based on a second decision rule; and

an interpolation sub-module generating an interpolated background mesh according to said reduced image and said final background blocks which serve as anchored blocks for interpolation.

2. A method for removing the background of visual content, said method comprising the steps of:

(a) normalizing the color of an original image to form a normalized image;

(b) segmenting said normalized image to form a segmented image;

(c) simulating the variation of pixel colors in the background of said original image and generating an interpolated background mesh;

(d) generating a background mask by comparing said normalized image and said interpolated background mesh, said background mask being formed by locating and extracting coherent regions between said normalized image and said interpolated background mesh;

(e) generating a refined background mask according to said background mask and said segmented image; and

(f) generating a foreground image by masking out background from said original image according to said refined background mask;

wherein said segmented image comprises a plurality of segmented regions and for each segmented region, if more than half of all pixels in said segmented region are covered by said background mask, then said refined background mask includes all pixels in said segmented region;

otherwise all pixels in said segmented region are excluded from said refined background mask.

3. The method for removing the background of visual content as claimed in claim 2, wherein said step (a) rescales the intensities of red, green and blue components of said original image so that said intensities of red, green and blue components have a minimum value 0 and a maximum value 255 after normalization.

4. The method for removing the background of visual content as claimed in claim 2, wherein said step (a) rescales the intensities of red, green and blue components of said original image according to the following linear equations:

$r_n<br><br>\left(i,j\right)=\frac{r_o<br><br>\left(i,j\right)-r_{\min }}{r_{\max }-r_{\min }}\times <br><br>255,\mathrm{where}<br><br><br><br>r_{\max }=\underset{i,j}{\max }<br><br>\left(r_o<br><br>\left(i,j\right)\right)<br><br><br><br>\mathrm{and}<br><br><br><br>r_{\min }=\underset{i,j}{\min }<br><br>\left(r_o<br><br>\left(i,j\right)\right),<br><br>g_n<br><br>\left(i,j\right)=\frac{g_o<br><br>\left(i,j\right)-g_{\min }}{g_{\max }-g_{\min }}\times <br><br>255,\mathrm{where}<br><br><br><br>g_{\max }=\underset{i,j}{\max }<br><br>\left(g_o<br><br>\left(i,j\right)\right)<br><br><br><br>\mathrm{and}<br><br><br><br>g_{\min }=\underset{i,j}{\min }<br><br>\left(g_o<br><br>\left(i,j\right)\right),<br><br>b_n<br><br>\left(i,j\right)=\frac{b_o<br><br>\left(i,j\right)-b_{\min }}{b_{\max }-b_{\min }}\times <br><br>255,\mathrm{where}<br><br><br><br>b_{\max }=\underset{i,j}{\max }<br><br>\left(b_o<br><br>\left(i,j\right)\right)<br><br><br><br>\mathrm{and}<br><br><br><br>b_{\min }=\underset{i,j}{\min }<br><br>\left(b_o<br><br>\left(i,j\right)\right),$

wherein r_o(i, j), g_o(i, j), and b_o(i, j) denote the intensities of R, G, B components in pixel (i, j) of said original image respectively, r_n(i, j), g_n(i, j), and b_n(i, j) denote the intensities of R, G, B components in pixel (i, j) of said normalized image respectively, and min and max denote maximum and minimum functions respectively.

5. The method for removing the background of visual content as claimed in claim 2, wherein in said step (d) a pixel belongs to said background mask if the distance between color vectors of said pixel in said normalized image and a corresponding pixel in said background mesh is equal to or less than a threshold T₃;

otherwise said pixel belongs to foreground;

wherein said threshold T₃ is a predefined parameter, or a value determined by statistics from image analysis of said normalized image.

6. The method for removing the background of visual content as claimed in claim 2, wherein said foreground image in step (f) is generated by assigning a white color to a pixel in said foreground image if said pixel is covered by said refined background mask, and assigning the color vector of a corresponding pixel on said original image if said pixel is not covered by said refined background mask.

7. A method for removing the background of visual content, said method comprising the steps of:

(a) normalizing the color of an original image to form a normalized image;

(b) segmenting said normalized image to form a segmented image;

(c) simulating the variation of pixel colors in the background of said original image and generating an interpolated background mesh;

(d) generating a background mask by comparing said normalized image and said interpolated background mesh, said background mask being formed by locating and extracting coherent regions between said normalized, image and said interpolated background mesh;

(e) generating a refined background mask according to said background mask and said segmented image; and

(f) generating a foreground image by masking out background from said original image according to said refined background mask;

wherein said step (c) further comprises the steps of;

(c1) forming a reduced image by reducing the resolution of said normalized image, each pixel in said reduced image being regarded as one block;

(c2) computing an average color vector for border blocks of said reduced image;

(c3) locating initial background blocks on said reduced image based on a first decision rule;

(c4) repeatedly extending said initial background blocks towards neighboring pixels on said reduced image to generate final background blocks based on a second decision rule; and

(c5) generating an interpolated background mesh according to said reduced image and said final background blocks which serve as anchored blocks for interpolation.

8. The method for removing the background of visual content as claimed in claim 7, said reduced image in said step (c1) being generated in accordance with a uniformly distributed pixel mapping defined as:

{overscore (c_r)}(i′

, j′

)={overscore (c_n)}(i, j), for i′

=1, 2, . . . , h′

, j′

=1, 2, . . . , w′

, i=1, 2, . . . , h, j=1, 2, . . . , w,

wherein $i=\left[1+\frac{h-1}{h^{\mathrm{\prime <br><br>}}-1}<br><br>\left(i^{\mathrm{\prime <br><br>}}-1\right)\right],j=\left[1+\frac{w-1}{w^{\mathrm{\prime <br><br>}}-1}<br><br>\left(j^{\mathrm{\prime <br><br>}}-1\right)\right],$

[x] means rounding x to a nearest integer, {right arrow over (c_n)}(i, j) is a color vector of said normalized image at pixel (i, j) and {right arrow over (c_r)}(i′

, j′

) is a color vector of said reduced image at pixel (i′

, j′

).

9. The method for removing the background of visual content as claimed in claim 7, said reduced image in said step (c1) being generated in accordance with a block division and color averaging method in which an average color of all pixels in a block of said normalized image is assigned to a corresponding pixel in said reduced image.

10. The method for removing the background of visual content as claimed in claim 7, wherein said average color vector for border blocks of said reduced image in step (c2) is a mean of color vectors of border blocks of said reduced image.

11. The method for removing the background of visual content as claimed in claim 7, wherein said average color vector for border blocks of said reduced image in step (c2) is a median of color vectors of border blocks of said reduced image.

12. The method for removing the background of visual content as claimed in claim 7, wherein said average color vector for border blocks of said reduced image in step (c2) is a mode of color vectors of border blocks of said reduced image.

13. The method for removing the background of visual content as claimed in claim 7, wherein said first decision rule in step (c3) defines a block (i′

, j′

) as a background block if said block (i′

, j′

) is a border block and a vector norm ∥

{overscore (c_r)}(i′

, j′

)−

{overscore (c_b)}∥

is equal to or less than a threshold T₁;

otherwise said block (i′

, j′

) is defined as a foreground block;

wherein {overscore (c_b)} is said average color vector for border blocks in said reduced image, {overscore (c_r)}(i′

, j′

) is a color vector for block (i′

, j′

) in said reduced image, and said threshold T₁ is a predefined parameter, or a value determined by statistics from image analysis of said normalized image.

14. The method for removing the background of visual content as claimed in claim 7, wherein said second decision rule in step (c4) defines a block (i′

, j′

) as a background block at k^th iteration if said block (i′

, j′

) is a background block at k−

1^th iteration, or said block (i′

, j′

) has a neighboring pixel which is a background block at k−

1^th iteration and the distance between color vectors of said block (i′

, j′

) and said neighboring pixel is equal to or less than a threshold T₂;

otherwise said block (i′

, j′

) is defined as a foreground block at k^th iteration;

wherein said threshold T₂ is a predefined parameter, or a value determined by statistics from image analysis of said normalized image.

15. The method for removing the background of visual content as claimed in claim 7, said interpolation in step (c5) being a triangle-based interpolation.

16. The method for removing the background of visual content as claimed in claim 7, said interpolation in step (c5) being a nearest neighbor interpolation.

17. The method for removing the background of visual content as claimed in claim 7, wherein said step (a) rescales the intensities of red, green and blue components of said original image so that said intensities of red, green and blue components have a minimum value 0 and a maximum value 255 after normalization.

18. The method for removing the background of visual content as claimed in claim 7, wherein said step (a) rescales the intensities of red, green and blue components of said original image according to the following linear equations:

$r_n<br><br>\left(i,j\right)=\frac{r_o<br><br>\left(i,j\right)-r_{\min }}{r_{\max }-r_{\min }}\times <br><br>255,\mathrm{where}<br><br><br><br>r_{\max }=\underset{i,j}{\max }<br><br>\left(r_o<br><br>\left(i,j\right)\right)<br><br><br><br>\mathrm{and}<br><br><br><br>r_{\min }=\underset{i,j}{\min }<br><br>\left(r_o<br><br>\left(i,j\right)\right),<br><br>g_n<br><br>\left(i,j\right)=\frac{g_o<br><br>\left(i,j\right)-g_{\min }}{g_{\max }-g_{\min }}\times <br><br>255,\mathrm{where}<br><br><br><br>g_{\max }=\underset{i,j}{\max }<br><br>\left(g_o<br><br>\left(i,j\right)\right)<br><br><br><br>\mathrm{and}<br><br><br><br>g_{\min }=\underset{i,j}{\min }<br><br>\left(g_o<br><br>\left(i,j\right)\right),<br><br>b_n<br><br>\left(i,j\right)=\frac{b_o<br><br>\left(i,j\right)-b_{\min }}{b_{\max }-b_{\min }}\times <br><br>255,\mathrm{where}<br><br><br><br>b_{\max }=\underset{i,j}{\max }<br><br>\left(b_o<br><br>\left(i,j\right)\right)<br><br><br><br>\mathrm{and}<br><br><br><br>b_{\min }=\underset{i,j}{\min }<br><br>\left(b_o<br><br>\left(i,j\right)\right),$

wherein r_o(i, j), g_o(i, j), and b_o(i, j) denote the intensities of R, G, B components in pixel (i, j) of said original image respectively, r_n(i, j), g_n(i, j), and b_n(i, j) denote the intensities of R, G, B components in pixel (i, j) of said normalized image respectively, and min and max denote maximum and minimum functions respectively.

19. The method for removing the background of visual content as claimed in claim 7, wherein in said step (d) a pixel belongs to said background mask if the distance between color vectors of said pixel in said normalized image and a corresponding pixel in said background mesh is equal to or less than a threshold T₃;

otherwise said pixel belongs to foreground;

wherein said threshold T₃ is a predefined parameter, or a value determined by statistics from image analysis of said normalized image.

20. The method for removing the background of visual content as claimed in claim 7, wherein said foreground image in step (f) is generated by assigning a white color to a pixel in said foreground image if said pixel is covered by said refined background mask, and assigning the color vector of a corresponding pixel on said original image if said pixel is not covered by said refined background mask.