Learnable object segmentation

• US 7,203,360 B2
• Filed: 04/09/2003
• Issued: 04/10/2007
• Est. Priority Date: 04/09/2003
• Status: Active Grant

• Current Assignee: Leica Microsystems Gmbh (Danaher Corporation)
• Original Assignee: Shih-Jong J. Lee
• Inventors: Oh, Seho, Lee, Shih-Jong J.
• Primary Examiner(s): Do; Anh Hong
• Application Number:
  US10/410,063
• Publication Number:
  US 20040202368A1
• Time in Patent Office: 1,462 Days
• Field of Search: 382/103, 382/173, 382/171, 382/175, 382/180, 382/155, 382/128, 382/169, 382/181, 382/133, 382/224, 382/226, 382/240, 382/274, 348/150, 348/155, 348/144, 348/149, 348/159, 348/218.1, 348/169, 706/13, 706/10, 706/902, 345/629, 345/582, 345/583, 345/600, 345/605
• US Class Current: 382/173
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A segmentation method receives a learning image and an objects of interest specification. A segmentation learning method creates a segmentation recipe output. It performs a segmentation application using the second image and the segmentation recipe to create a segmentation result output. The segmentation learning method includes an object region of interest segmentation learning step and an object type specific segmentation learning step. The segmentation application method includes an object region of interest segmentation step and an object type specific segmentation step. The learnable object segmentation method further comprises an online learning and a feedback learning step that allows the update of the segmentation recipe automatically or under user direction.

Claims

1. A learnable object segmentation method comprising the steps of:

a) Inputting a learning image for a selected specific application;

b) Inputting an objects of interest specification for the specific application;

c) Performing purpose directed segmentation learning using the input image and the objects of interest specification to create a segmentation recipe output wherein the segmentation learning performs learning by example and wherein the segmentation learning method includes an object region of interest segmentation learning step and an object type specific segmentation learning step wherein the object region of interest segmentation learning is applied to the measurements associated with each pixel to arrive at a decision for the pixel label including all object types and a non-object type;

d) Inputting a second image for the specific application;

e) Performing segmentation application using the second image and the segmentation recipe to create a segmentation result output for the specific application.

2. A general purpose semi-automatic method for objects of interest specification comprising the steps of:

a) Inputting a learning image for a selected specific application;

b) Performing automatic initial object region segmentation using the learning image to create an initial object regions output for the specific application;

c) Performing first user confirmation and revision using the initial object regions presented to user to create a confirmed object regions output;

d) Performing automatic initial object segmentation using the confirmed object regions to create an initial object masks output for the specific application wherein the automatic initial object region segmentation uses a measurement-space-guided spatial clustering method including histogram mode seeking or a recursive histogram-directed spatial-clustering;

e) Performing second user confirmation and revision using the initial object masks presented to user to create a confirmed representative object masks output.

3. An object region of interest segmentation learning method comprising the steps of:

a) Inputting a learning image for a selected specific application;

b) Inputting object region of interest masks for the specific application;

c) Performing feature extraction using the learning image and the object region of interest masks to create a feature candidates output wherein a comprehensive set of features is extracted and features are associated with each pixel and wherein the feature extraction method extracts features associated with each pixel wherein the features are selected from the set consisting of

a) Intensity space features wherein the intensity space features are derived from the grayscale intensity of the learning image, and

b) Color space features wherein color transformation is applied to convert the color image into multi-bands of grayscale images;

d) Performing feature learning using the feature candidates to create at least one discriminate feature output wherein the feature learning includes feature selection and feature transformation steps;

e) Performing decision learning using the at least one discriminate feature to create an object region of interest segmentation recipe output for the specific application.

4. An object of interest segmentation learning method comprising the steps of:

a) Inputting a learning image for a selected specific application;

b) inputting an object of interest mask for the specific application;

c) Performing pixel classifier learning using the input image and the object of interest mask to create a pixel segmentation recipe output for the specific application wherein a comprehensive set of features is extracted and features are associated with each pixel for the pixel classifier learning;

d) Performing region constraint learning using the input image and the object of interest mask to create a region constraint recipe output for the specific application wherein

the pixel segmentation recipe and the region constraint recipe are grouped into an object of interest segmentation recipe output for the specific application.

5. The method of claim 4 wherein the pixel classifier learning method further comprises the steps of:

a) Performing pixel feature extraction using the input image and the object of interest mask to create a feature candidates output;

b) Performing pixel feature learning using the feature candidates output to create at least one discriminate feature output;

c) Performing pixel decision learning using the at least one discriminate feature to create a pixel segmentation recipe output.

6. The method of claim 5 wherein the pixel feature extraction method includes the extraction of a template matching score.

7. The method of claim 4 wherein the region constraint learning method further comprises the steps of:

a) Performing region property extraction using the input image and the object of interest mask to create a region property candidates output;

b) Performing region property learning using the region property candidates to create at least one discriminate region property output;

c) Performing region constraint learning using the at least one discriminate region property to create a region constraint recipe output.

8. The method of claim 7 wherein the region property extraction method includes the extraction of moments based region properties.

9. The method of claim 7 wherein the region constraint learning method uses sample quantile based statistics as region constraint.

10. A general purpose object region of interest region segmentation method based on learned recipes comprising the steps of:

a) Inputting an image;

b) Inputting an object region of interest segmentation recipe for a specific application;

c) Performing object region of interest pixel classification using the input image and the object region of interest segmentation recipe to create at least one object type specific initial region output for the specific application wherein the object region of interest pixel classification method further comprises a feature extraction step and a decision rule application step wherein the extracted features are stored in a multi-band feature image wherein each band stores the values of a specific feature for each pixel location;

d) Performing at least one object type specific region refinement for the specific application using the at least one object type specific initial region to create at least one object type specific region segmentation result output wherein the at least one object type specific region refinement method uses a structure-guided processing method.

11. A general purpose object of interest region segmentation method based on learned recipes comprising the steps of:

a) Inputting an image for a specific application;

b) Inputting an object of interest segmentation recipe for the specific application;

c) Performing object of interest pixel classification using the input image and the object of interest segmentation recipe to create an initial object, non-object, and unknown regions output for the specific application;

d) Performing region labeling and property extraction using the input image and the object of interest segmentation recipe to create an object, non-object region properties output for the specific application;

e) Performing iterative context dependent region merging using the initial object, non-object, and unknown regions and the object, non-object region properties to create an object, non-object region output wherein the iterative context dependent region merging step merges the unknown pixels into object and non-object regions;

f) Performing final region refinement using the object, non-object region properties to create an abject segmentation results output wherein the final region refinement performs homotopic thinning or thickening operations.

12. A general purpose iterative context dependent region merging process method comprising the steps of:

a) Inputting an image;

b) Inputting an images of object, non-object, and unknown regions for a specific application;

c) Inputting object, non-object region properties for the specific application;

d) Scanning the “

unknown”

pixels in the input image in a predetermined manner and perform region merging using the object, non-object region properties to create an intermediate region merging results output for the specific application;

e) Repeating the region merging process until no more “

unknown”

pixels exist or no more regions merge to create a final intermediate region merging results output;

f) Performing a final assignment using the final intermediate region merging results to create an interactive context dependent region merging result output for the specific application.