Image Segmentation Using Hierarchical Unsupervised Segmentation and Hierarchical Classifiers

• US 20140037198A1
• Filed: 08/06/2012
• Published: 02/06/2014
• Est. Priority Date: 08/06/2012
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Saunders, Craig, Csurka, Gabriela, Larlus-Larrondo, Diane, Hu, Rui
• Granted Patent: US 8,873,812 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods

Abstract

An image segmentation method includes generating a hierarchy of regions by unsupervised segmentation of an input image. Each region is described with a respective region feature vector representative of the region. Hierarchical structures are identified, each including a parent region and its respective child regions in the hierarchy. Each hierarchical structure is described with a respective hierarchical feature vector that is based on the region feature vectors of the respective parent and child regions. The hierarchical structures are classified according to a set of predefined classes with a hierarchical classifier component that is trained with hierarchical feature vectors of hierarchical structures of training images. The training images have semantic regions labeled according to the set of predefined classes. The input image is segmented into a plurality of semantic regions based on the classification of the hierarchical structures and optionally also on classification of the individual regions.

Claims

1. An image segmentation method comprising:

generating a hierarchy of regions by unsupervised segmentation of an input image;

describing each region of the hierarchy with a respective region feature vector that is representative of the hierarchy region;

identifying hierarchical structures in the hierarchy, each of the hierarchical structures comprising a parent hierarchy region and its respective child hierarchy regions;

describing each of the input image hierarchical structures with a respective hierarchical feature vector, the hierarchical feature vector being based on the region feature vectors of the respective parent hierarchy region and child hierarchy regions;

with a hierarchical classifier component, classifying each of the input image hierarchical structures according to a set of predefined classes, the hierarchical classifier component having been trained with hierarchical feature vectors of hierarchical structures of training images, the training images comprising semantic regions labeled according to the set of predefined classes; and

segmenting the input image into a plurality of semantic regions based on the classification of the hierarchical structures.

2. The method of claim 1, wherein at least one of the classifying and segmenting is performed with a processor.

3. The method of claim 1, wherein the generating of the hierarchy of regions comprises partitioning the input image to identify regions and organizing the identified regions into the hierarchy of regions.

4. The method of claim 1, further comprising:

classifying each of the hierarchy regions using a region classifier that has been trained with region feature vectors describing hierarchy regions of the training images, combinations of the training image hierarchy regions forming the hierarchical structures of the training images; and

wherein the segmenting of the input image into a plurality of semantic regions is also based on the classification of the hierarchy regions.

5. The method of claim 1, further comprising assigning pixel probabilities or scores for the predefined classes to pixels of the input image based on the classification of the input image hierarchical structures, and wherein the segmenting of the input image is based on the pixel probabilities or scores.

6. The method of claim 5, wherein each pixel score or probability for each semantic label is computed based on the classifications of all hierarchical structures that contain the pixel.

7. The method of claim 6, wherein each pixel score or probability for each semantic label is also based on a classification of all hierarchy regions that contains the pixel.

8. The method of claim 1, wherein the hierarchy of regions is in the form of a binary tree in which each parent hierarchy region has exactly two child hierarchy regions and no hierarchy region has more than one parent hierarchy region.

9. The method of claim 1, wherein each of the identified hierarchical structures in the hierarchy is a triplet of exactly three hierarchical regions.

10. The method of claim 1, wherein each hierarchical feature vector is generated by aggregating the feature vectors of the respective parent hierarchy region and child hierarchy regions.

11. The method of claim 10, wherein each hierarchical feature vector is generated by concatenating the feature vectors of the respective parent hierarchy region and child hierarchy regions.

12. The method of claim 1, wherein the region feature vectors are based on features selected from region appearance features, region shape features, and combinations thereof.

13. The method of claim 1, wherein each parent hierarchy region is the union of its child hierarchy regions.

14. The method of claim 1, wherein the image comprises a photograph.

15. A computer program product comprising a non-transitory recording medium storing instructions, which when executed by a computer processor, perform the method of claim 1.

16. A system comprising memory storing instructions for performing the method of claim 1 and a processor in communication with the memory, for executing the instructions.

17. An image segmentation system comprising:

memory which stores;

a region detector which identifies regions of an input image by unsupervised segmentation of the image;

a hierarchy generator which arranges the identified regions into a hierarchy of regions in the form of a tree structure comprising a plurality of hierarchical structures, each comprising a parent region and two child regions, each parent region being the union of its two respective child regions;

a region feature vector extractor which describes each of the identified regions with a respective region feature vector;

a hierarchical feature vector generator which describes each of the plurality of hierarchical structures with a respective hierarchical feature vector which is based on the region feature vectors of the respective parent region and two child regions;

optionally, a region classifier component which classifies each of the regions in the hierarchy according to a predefined set of classes, based on the respective region feature vector;

a hierarchical classifier component, which classifies each of the plurality of hierarchical structures according to the predefined set of classes based on the respective hierarchical feature vector; and

a labeling component for segmenting the input image into a plurality of semantic regions based on the classification of the hierarchical structures and optionally also the classification of the regions of the hierarchy; and

a processor in communication with the memory for implementing the region detector;

hierarchy generator, region feature vector extractor, hierarchical feature vector generator, hierarchical classifier component, and labeling component.

18. The system of claim 17, wherein the region classifier component is included in the system and wherein the labeling component segments the input image into the plurality of semantic regions based on the classification of the hierarchical structures and also the classification of the regions of the hierarchy.

19. A method for generating an image segmentation system, comprising:

receiving a set of training images in which semantic regions of each image have been identified and labeled with a semantic class selected from a predefined set of semantic classes;

for each of the training images;

segmenting the training image in an unsupervised manner into a hierarchy of regions, without regard to the labels;

extracting a region feature vector for each of the regions in the hierarchy of regions; and

generating a hierarchical feature vector for each of a plurality of hierarchical structures in the hierarchy, each hierarchical structure comprising a parent region which is the union of two respective child regions, each hierarchical feature vector being based on the region feature vectors of exactly three regions comprising the respective parent region and the two respective child regions;

training a hierarchical classifier model for at least one of the semantic classes based on the semantic region labels and the hierarchical feature vectors;

optionally, training a region classifier model for the at least one of the semantic classes based on the semantic region labels and the region feature vectors;

whereby, when a new image to be segmented according to semantic class is received, a labeling component is able to label a semantic region of the image according to a class in the set of classes based on hierarchical feature vectors for hierarchical structures comprising regions of a hierarchy that are generated by unsupervised segmentation of the input image.

20. The method of claim 19, wherein the method includes training of the region classifier model.

21. An image segmentation method comprising:

receiving an image to be segmented;

generating a hierarchy of regions of the input image by segmenting the image in an unsupervised manner, the hierarchy of regions being in the form of a binary tree comprising triplets, each triplet comprising a parent region and two child regions;

describing each region in the hierarchy with a region feature vector that describes at least one of the region appearance and the region shape;

describing each of the triplets with a feature vector that aggregates the feature vectors of only the three regions of the triplet;

optionally, providing a region classifier component trained with region feature vectors of training images comprising regions labeled according to semantic class;

optionally, classifying regions of the hierarchy, based on the respective region feature vectors, with the region classifier to generate a class score or probability for each of at least one of the semantic classes for each of the regions;

providing a hierarchical classifier component trained with triplet feature vectors of labeled training images;

classifying triplets of the hierarchy, based on the respective triplet feature vectors, with the hierarchical classifier to generate a class score or probability for each of at least one of the semantic classes for each of the triplets;

associating pixels of the input image with a score or a probability for each semantic class that is computed using the scores or probabilities of the triplets, and optionally of the hierarchy regions, that contain that pixel;

segmenting the input image into semantic regions based on the pixel scores or probabilities for the semantic classes.