System and method for object class localization and semantic class based image segmentation

• US 8,111,923 B2
• Filed: 08/14/2008
• Issued: 02/07/2012
• Est. Priority Date: 08/14/2008
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Csurka, Gabriela, Perronnin, Florent
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US12/191,579
• Publication Number:
  US 20100040285A1
• Time in Patent Office: 1,272 Days
• Field of Search: 382/159, 382/164, 382/170, 382/173, 382/190, 382/224, 382/225, 382/228, 382/254, 707/2, 707/3, 707/5, 707/723, 707/741, 707/E17.071, 707/E17.108, 358/1.15, 358/1.9, 704/245
• US Class Current: 382/190
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

An automated image processing system and method are provided for class-based segmentation of a digital image. The method includes extracting a plurality of patches of an input image. For each patch, at least one feature is extracted. The feature may be a high level feature which is derived from the application of a generative model to a representation of low level feature(s) of the patch. For each patch, and for at least one object class from a set of object classes, a relevance score for the patch, based on the at least one feature, is computed. For at least some or all of the pixels of the image, a relevance score for the at least one object class based on the patch scores is computed. An object class is assigned to each of the pixels based on the computed relevance score for the at least one object class, allowing the image to be segmented and the segments labeled, based on object class.

Claims

1. An automated image processing method comprising:

with a processor;

extracting a plurality of patches of an input image;

for each patch, extracting at least one high level feature based on its low level representation and a generative model built from low level features;

for each patch, and for at least one object class from a set of object classes, computing a relevance score for the patch based on the at least one high level feature and the output of at least one patch classifier;

for at least some of the pixels of the image, computing a relevance score for the at least one object class based on the patch scores; and

assigning an object class label to each of the pixels based on the computed relevance score for the at least one object class.

2. The method of claim 1, wherein the low level representation is based on low level features extracted from the patch.

3. The method of claim 1, further comprising:

segmenting the image based on the pixels assigned classes.

4. The method of claim 1, wherein the assigning of one of the object classes to each pixel includes assigning one of the set of object classes, if a threshold relevance score is met.

5. The method of claim 1, wherein the at least one object class comprises a plurality of object classes.

6. The method of claim 1, further comprising classifying the image with a global classifier and, based on the global classification, identifying a subset of the set of object classes for which a relevance score for the patch, based on the at least one high level feature, is computed.

7. The method of claim 1, further comprising:

partitioning the image into multiple regions, using low-level segmentation;

for each region and each class, combining the computed pixel relevance scores; and

wherein the assigned class is based on the combined pixel relevance scores.

8. The method of claim 1, wherein at least 20 patches are extracted.

9. The method of claim 1, wherein at least some of the pixels are present in more than one patch and the computed relevance score for the at least one object class based on the patch scores is an optionally weighted function of the patch scores for the patches in which the pixel is present.

10. The method of claim 1, wherein the relevance score for the at least one object class based on the patch scores is computed for all the pixels of the image.

11. The method of claim 1, wherein the patch classifier comprises a set of binary classifiers, each trained on positive and negative samples of the class.

12. The method of claim 11, wherein negative samples are extracted from images that each contain a positive sample.

13. The method of claim 11, wherein the positive samples are patches manually assigned to the respective class.

14. The method of claim 1, wherein the generative model is a Gaussian mixture model trained on low level features extracted from training images.

15. An automated image processing system comprising memory which stores instructions for performing the method of claim 1 and a processor, in communication with the memory, which executes the instructions.

16. An image segmented by the method of claim 1.

17. A computer program product comprising a non-transitory recording medium that stores instructions which, when executed by a computer, perform an image processing method comprising:

extracting a plurality of patches of an input image;

for each patch, extracting at least one high level feature based on its low level representation and a generative model built from low level features;

for each patch, and for at least one object class from a set of object classes, computing a relevance score for the patch based on the at least one high level feature and the output of at least one patch classifier;

for at least some of the pixels of the image, computing a relevance score for the at least one object class based on the patch scores; and

assigning an object class label to pixels of the image, based on the computed relevance score for the at least one object class.

18. An automated image processing system comprising:

a patch extractor which extracts patches of an input image;

a low level feature extractor which extracts, for each patch, a low level feature;

a high level feature extractor which extracts, for each patch, a high level feature based on the low level feature and a generative model built on low level features;

a classifier system, configured for classifying the patch, based on the high level feature, for each of a set of object classes;

a scoring component which for each patch, and for at least one object class from a set of object classes, computes a relevance score for the patch based on the classifier and, for at least some of the pixels of the image, computes a relevance score for the at least one object class based on the patch scores; and

a labeling component assigns an object class to each of the pixels based on the computed relevance score for the at least one object class.

19. A digital image processing method comprising:

with a processor;

for an input image, extracting patches in the image;

from each patch, extracting a low-level representation comprising a feature vector;

for each patch, using its low-level representation and a generative model to extract a high-level representation;

for each patch and each class, computing a relevance score based on the high-level representation and a patch classifier;

for each pixel and each class, computing a relevance score based on the patch scores; and

for each pixel, take a decision based on the class scores.

20. The method of claim 19, wherein the method further comprises:

partitioning the image into regions using low-level segmentation; and

or each region and each class, combining the pixel scores; and

for each region, take a decision based on the class scores, the decision for each pixel being based on the region decision.

21. The method of claim 19, wherein the image is assigned a global score with respect to each class and a class is considered only if the global score of the considered class exceeds a given threshold.