Method and apparatus for segmenting images prior to coding

• US 6,301,385 B1
• Filed: 12/17/1998
• Issued: 10/09/2001
• Est. Priority Date: 02/10/1997
• Status: Expired due to Term

• Current Assignee: Hanger Solutions, LLC (IP Investments Group LLC)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Chen, Tsuhan, Swain, Cassandra Turner
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Werner, Brian P.
• Application Number:
  US09/213,417
• Time in Patent Office: 1,027 Days
• Field of Search: 382/103, 382/107, 382/156, 382/157, 382/158, 382/173, 382/174, 382/236, 382/255, 382/266, 382/274, 382/282, 382/291, 382/199, 382/176, 382/180, 382/239, 382/159, 345/358, 345/433-435, 706/15, 706/20, 706/22, 348/26, 348/14, 348/15, 348/17, 348/169, 348/170, 348/349, 348/411-418, 348/586, 348/700
• US Class Current: 382/173
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: H04N 19/20 using video object coding

Abstract

To segment moving foreground from background, where the moving foreground is of most interest to the viewer, this method uses three detection algorithms as the input to a neural network. The multiple cues used are focus, intensity, and motion. The neural network consists of a two-layered neural network. Focus and motion measurements are taken from high frequency data, edges; whereas, intensity measurements are taken from low frequency data, object interiors. Combined, these measurements are used to segment a complete object. Results indicate that moving foreground can be segmented from stationary foreground and moving or stationary background. The neural network segments the entire object, both interior and exterior, in this integrated approach. Results also demonstrate that combining cues allows flexibility in both type and complexity of scenes. Integration of cues improves accuracy in segmenting complex scenes containing both moving foreground and background. Good segmentation yields bit rate savings when coding the object of interest, also called the video object in MPEG4. This method combines simple measurements to increase segmentation robustness.

Claims

1. A method for segmenting an image comprising:

a) applying at least three segmentation techniques to an image producing independent segmentation maps, the segmentation techniques including focus, motion, and intensity measurements, the focus and motion measurements producing edge information on foreground portions of the image, and the intensity measurement producing interior information on the foreground portions of the image;

b) providing a neural network with the segmentations maps produced from the segmentation techniques, wherein each map is processed independently from and without combination with another map, and each map is provided as a separately weighted, dedicated, and independent input to the neural network; and

c) wherein the neural network integrates the dedicated, independent and separately weighted segmentation maps as a final segmentation map output without post-processing filling, the output including segmented, filled foreground portions of the image.

2. A method for segmenting a sequence of images prior to coding the image sequence comprising the steps of:

a) detecting motion within foreground and background portions of the image sequence;

b) detecting focus within the image sequence;

c) detecting intensity within the image sequence, the motion and focus detection producing measurements including edge information on foreground portions of the image sequence, and the intensity detection producing a measurement including interior information on foreground portions of the image sequence;

d) calculating segments using a neural network employing detected measurements from steps a) through c) wherein each measurement is processed independently from and without combination with another measurement, and each measurement is provided as a separately weighted, dedicated, and independent input to the neural network, the neural network integrating the dedicated, independent and separately weighted measurements as a final segmented output image without post-processing filling, the output image including segmented, filled foreground portions of the image.

3. The method according to claim 2, further comprising the steps of:

e) acquiring two consecutive images for use in the detecting steps.

4. The method according to claim 2, wherein the step of detecting motion comprises:

(i) detecting a difference between pixels in successive frames; and

(ii) determining that a pixel is in motion if the difference for that pixel exceeds a predetermined threshold.

5. The method according to claim 2, wherein the step of detecting focus comprises:

(i) calculating the magnitude of the Sobel edge detection over an nxn pixel square; and

(ii) dividing the magnitude of the Sobel edge detection by the edge width.

6. The method according to claim 2, wherein the step of detecting intensity comprises determining a gray level of the pixel.

7. A method for processing an image sequence to segment foreground from background, comprising the steps of:

a) acquiring successive images in the sequence;

b) simultaneously measuring motion, focus and intensity of pixels within successive images, wherein step of simultaneously measuring motion includes measuring motion within foreground and background portions of the images, the focus and motion measurements producing edge information on foreground portions of the image sequence, and the intensity measurement producing interior information on the foreground portions of the image sequence;

c) inputting the motion, focus and intensity measurements to a neural network wherein each measurement is processed independently from and without combination with another measurement, and each measurement is provided as a separately weighted, dedicated, and independent input to the neural network,

d) calculating foreground and background segmentation using the motion, focus, and intensity measurements with the neural network; and

e) drawing a segment map based on the calculated foreground and background segments without post-processing filling, the segment map including segmented, filled foreground portions of the image sequence.

8. The method according to claim 7, further comprising the steps of:

f) training the neural network using two initial frames and a hand-segmented result.

9. The method according to claim 8, further comprising the step of:

g) speeding up the training step f) using an adaptive learning rate.

10. The method according to claim 7, wherein the step of detecting motion comprises:

(i) detecting a difference between pixels in successive images; and

(ii) determining that a pixel is in motion if the difference for that pixel exceeds a predetermined threshold.

11. The method according to claim 7, wherein the step of detecting focus comprises:

(i) calculating the magnitude of the Sobel edge detection over an n×

n pixel square; and

(ii) dividing the magnitude of the Sobel edge detection by the edge width.

12. The method according to claim 7, wherein the step of detecting intensity comprises determining a gray level of the pixel.

13. An apparatus for segmenting foreground and background from a sequence of images;

a) a motion detection detecting motion of pixels within foreground and background portions of the image sequence and outputting a motion map;

b) a focus detector detecting pixels that are in focus and outputting a focus map;

c) an intensity detector detecting those pixels that have high intensity and those with low intensity and outputting an intensity map, the focus and motion detection producing measurements including edge information on foreground portions of the image sequence, and the intensity detection producing a measurement including interior information on the foreground portions of the image sequence;

d) a neural network being coupled to the motion detector, the focus detector and the intensity detector, weighing and integrating the outputs from these detectors and outputting a segmentation map without post-processing filling, the output segmentation map including segmented, filled foreground portions of the image sequence, and

e) wherein each output of the detectors is processed independently from and without combination with another output, and each output is provided as a separately weighted, dedicated, and independent input to the neural network.

14. The apparatus according to claim 13, wherein the neural network comprises a two layer neural network.

15. The apparatus according to claim 13, wherein the neural network further comprises a hidden layer with two neurons and an output layer with one neuron, and each of the neurons employ sigmoid functions.

16. The apparatus according to claim 15, wherein the two neurons in the hidden layer are coupled to the one neuron in the output layer using a ninth weight and a tenth weight, respectively.

17. The apparatus according to claim 16, wherein the one neuron in the output layer includes three inputs, one from each of the two neurons in the hidden layer and a bias input, having an eleventh weight.

18. A device for segmenting objects within a sequence of images prior to coding of the images for transmission or storage of the images comprising:

a) means for digitizing the image sequence to obtain a sequence of digitized images;

b) means for segmenting an image based on motion of an object in foreground and background portions within the image, said motion segmenting means being coupled to said means for digitizing and outputting a motion segmentation map;

c) means for segmenting an image using focus measurements, said focus segmenting means being coupled to said means for digitizing and outputting a focus segmentation map;

d) means for segmenting an image using brightness measurements, said brightness segmenting means being coupled to said means for digitizing and outputting a brightness segmentation map, the motion producing a measurement including edge information on foreground portions of the image, the focus measurements including edge information on the foreground portions of the image, and the brightness measurements including interior information on the foreground portions of the image;

e) a neural network for calculating a final segmentation map using segmentation maps output by the motion segmenting means, the brightness segmenting means and the focus segmenting means, and applying the outputted maps to the neural network, wherein each map is processed independently from and without combination with another map, and each map is provided as a separately weighted, dedicated, and independent input to the neural network, the neural network integrating the dedicated, independent and separately weighted maps as the final segmentation map without post-processing filling, and the final segmentation map including segmented, filled foreground portions of the image.