METHOD AND APPARATUS FOR IDENTIFYING VISUAL CONTENT FOREGROUNDS

• US 20100045663A1
• Filed: 07/14/2009
• Published: 02/25/2010
• Est. Priority Date: 08/25/2008
• Status: Active Grant

• Current Assignee: National Taiwan University of Science and Technology
• Original Assignee: National Taiwan University of Science and Technology
• Inventors: Su, Chun-rong, Lien, Chun-hao, Chen, Jiann-jone
• Granted Patent: US 8,203,554 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/155 involving morphological operators
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

A method and apparatus for identifying visual content foregrounds, the method comprises steps of: determining a 3-D opening-by-reconstruction modest structure element (B_O) and a 3-D closing-by-reconstruction modest structure element (B_C); comparing an original image with an image obtained by performing MSOR operation to the original image with B_O and an image obtained by performing MSCR operation to the original image with B_C so as to generate an enhanced top-hat image and an enhanced bottom-hat image; and locating an overlap region between the enhanced top-hat image and the enhanced bottom-hat image. The overlap region forms a foreground identifying screen which is capable of identifying and extracting—refined for obtaining delicate foregrounds.

Claims

1. A foreground identification apparatus, comprising:

a modest structure element (SE) determination module, for performing multi-scale opening by reconstruction (MSOR) and multi-scale closing by reconstruction (MSCR) operations, respectively, to an original image by using plural values of three-dimensional (3-D) structure element for determining a 3-D opening-by-reconstruction modest structure element and a 3-D modest closing-by-reconstruction structure element;

an enhanced top-hat image and bottom-hat image generation module, for comparing the original image and an image obtained by performing MSOR operation with the 3-D opening-by-reconstruction modest structure element to the original image so as to generate an enhanced top-hat image, and comparing the original image and an image obtained by performing MSCR operation with the 3-D closing-by-reconstruction modest structure element to the original image so as to generate an enhanced bottom-hat image;

a foreground identification screen generation module, for locating an overlap region between the enhanced top-hat image and the enhanced bottom-hat image, where the overlap region forms a foreground identification screen;

a background mesh generation module, for simulating the variation of pixel colors in background areas extracted by the foreground identification screen for generating an interpolated background mesh;

an image segmentation module, for dividing the original image into a plurality of regions and indexing the plural regions so that an image with indexed regions is obtained;

a refinement module, for comparing the interpolated background mesh and the image with indexed regions for determining a refined foreground identification screen according to a refinement calculation; and

a foreground identification module, for identifying and extracting foregrounds from the original image by utilizing the refined foreground identification screen.

2. A foreground identification method, comprising steps of:

(a) respectively performing MSOR and MSCR operations to an original image by using plural values of 3-D structure element for determining a 3-D opening-by-reconstruction modest structure element and a 3-D modest closing-by-reconstruction structure element;

(b) comparing the original image and an image obtained by performing MSOR operation with the 3-D opening-by-reconstruction modest structure element to the original image so as to generate an enhanced top-hat image, and comparing the original image and an image obtained by performing MSCR operation with the 3-D modest closing-by-reconstruction structure element to the original image so as to generate an enhanced bottom-hat image;

(c) locating an overlap region between the enhanced top-hat image and the enhanced bottom-hat image, where the overlap region forms a foreground identification screen;

(d) simulating the variation of pixel colors in background areas extracted by the foreground identification screen for generating an interpolated background mesh;

(e) dividing the original image into a plurality of regions and indexing the plural regions so that an image with indexed regions is obtained;

(f) comparing the interpolated background mesh and the image with indexed regions for determining a refined foreground identification screen according to a refinement calculation; and

(g) identifying and extracting foregrounds from the original image by utilizing the refined foreground identification screen.

3. The foreground identification method of claim 2, wherein the step (a) of determining the 3-D opening-by-reconstruction modest structure element comprises sub-steps of:

(a1) setting a first structure element with a first value and a second structure element with a second value, where the second value is larger than the first value;

(a2) calculating the difference between all pixels in the original image and an image obtained by performing MSOR operation to the original image by using the first structure element, for obtaining a first difference value;

(a3) calculating the difference between all pixels in the original image and an image obtained by performing MSOR operation to the original image by using the second structure element, for obtaining a second difference value;

(a4) dividing the difference between the first difference value and the second difference value by the difference between the first structure element and the second structure element, for obtaining a slope; and

(a5) adopting the first structure element as the 3-D opening-by-reconstruction modest structure element if the slope is substantially equal to zero, otherwise, increasing the second value and setting the increased second value as a new value of the second structure element, setting the original second value as a new value of the first structure element, and repeating the steps (a2) to (a4) by using the new values of the first and the second structure elements.

4. The foreground identification method of claim 3, wherein the original image is resized to a small one and the resized original image is used to determine the 3-D opening-by-reconstruction modest structure element through the steps (a1) to (a5).

5. The foreground identification method of claim 2, wherein MSOR operation is represented as

I_B^OR=(I{tilde over (∘

)}nB)=δ

_B^m+1(I∘

nB,I)=min(δ

_B^m(I∘

nB,I)[+]nB,I)

where I is the original image, B is the 3-D structure element, I∘

B=(I[−

]B)[+]B is a morphological opening operation,

6. The foreground identification method of claim 2, wherein MSCR operation is represented as

I_B^CR=(I{tilde over (

)}nB)=ε

_B^m+1(I

nB,I)=max(ε

_B^m(I

nB,I)[−

]nB,I)

where I is the original image, B is the 3-D structure element, I

B (I[+]B)[−

]B is a morphological closing operation,

7. The foreground identification method of claim 2, wherein each pixel in the original image subtracts corresponding pixel in the image obtained by performing MSOR operation to the original image by using the 3-D opening-by-reconstruction modest structure element, for generating the enhanced top-hat image.

8. The foreground identification method of claim 2, wherein each pixel in the image obtained by performing MSCR operation to the original image by using the 3-D closing-by-reconstruction modest structure element, subtracts corresponding pixel in the original image, for generating the enhanced bottom-hat image.

9. The foreground identification method of claim 2, wherein the foreground identification screen is the overlap region between the enhanced top-hat image and the enhanced bottom-hat image, represented as

10. The foreground identification method of claim 2, wherein the background of original image extracted by the foreground identification screen is represented as

I_G^α

=I_BG^α

•

I

where I is the original image, I_G^α

is the extracted background, I_G^α

(x,y)ε

{0, 1, . . . , 2ⁿ−

1}, I_BG^α

is a background identification screen derived from the foreground identification screen I_FG^α

by two-value transformation (i.e. when a pixel value of I_FG^α

is “

1”

, a corresponding pixel value of I_BG^α

is “

0”

;

when the pixel value of I_BG^α

is “

0”

, the corresponding pixel value of I_FG^α

is “

1”

), and operation symbol •

denotes element by element multiplications.

11. The foreground identification method of claim 10, wherein the interpolated background mesh is generated by utilizing Lagrangian interpolation algorithm to simulate the variation of pixel colors in the extracted background I_G^α

.

12. The foreground identification method of claim 2, wherein the refined foreground identification screen determined according to the refinement calculation is represented as

13. The foreground identification method of claim 2, wherein in the step (g), identifying and extracting foregrounds from the original image is represented as

I_FG=Ĩ

_FG^α

•

I 

 

(17)

where I is the original image, I_FG is the foregrounds extracted from the original image, Ĩ

_FG^α

is the refined foreground identification screen, and operation symbol •

denotes element by element multiplications.