Region refocusing for data-driven object localization

• US 8,879,796 B2
• Filed: 08/23/2012
• Issued: 11/04/2014
• Est. Priority Date: 08/23/2012
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Rodriguez Serrano, Jos Antonio
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): YEH, EUENG NAN
• Application Number:
  US13/592,961
• Publication Number:
  US 20140056520A1
• Time in Patent Office: 803 Days
• Field of Search: 382/105, 382/159, 382/172
• US Class Current: 382/105
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/625 License plates

Abstract

A system and method are provided for segmenting an image. The method includes computing an image signature for an input image. One or more similar images are identified from a first set of images, based on the image signature of the input image and image signatures of images in the first set of images. The similar image or images are used to define a cropped region of the input image and a second image signature is computed, this time for the cropped region. One or more similar images are identified from a second set of images, based on the cropped image signature and the image signatures of images in the second set of images. The input image is segmented based on a segmentation map of at least one of the similar images identified in the second set of images.

Claims

1. A method for segmenting an image comprising:

computing a first image signature for an input image;

identifying at least one similar image from a first set of images, based on the first image signature and image signatures of images in the first set of images;

generating a cropped region of the input image based on the identified at least one similar image;

computing a second image signature for the cropped region;

identifying at least one similar image from a second set of images, based on the second image signature and image signatures of images in the second set of images; and

segmenting the input image based on a segmentation map of at least one of the at least one similar image identified in the second set of images.

2. The method of claim 1, wherein at least one of the computing of the first and second image signatures, identifying of similar images from the first and second sets, generation of the cropped region, and segmenting the input image is performed with a computer processor.

3. The method of claim 1, further comprising estimating a segmented region of the input image based on a segmentation map of at least one of the at least one similar image in the first set of images and wherein the generating of the cropped region of the input image is based on the segmented region.

4. The method of claim 3, wherein the generating of the cropped region comprises generating a bounding box surrounding the segmented region which is larger in size than the segmented region.

5. The method of claim 3, wherein the generating of the cropped region of the input image comprises shifting horizontal and vertical coordinates of the segmented region as a function of a width and a height of the segmented region, respectively, and a respective scaling factor.

6. The method of claim 1, wherein the images in the second set of images are each derived by cropping an image in the first set of images.

7. The method of claim 1, wherein the segmenting of the input image comprises:

generating a second cropped region of the input image based on the cropped region;

computing a third image signature for the second cropped region; and

identifying at least one similar image from a third set of images, based on the third image signature and image signatures of images in the third set of images.

8. The method of claim 1, wherein the computing of the image signature for the input image comprises generating a multidimensional vector based on features extracted from patches of the image.

9. The method of claim 1, wherein the identifying at least one similar image from at least one of the first and second sets of images comprises identifying the top K most similar images from the at least one of the first and second sets of images, where K is greater than 1.

10. The method of claim 1, further comprising at least one of:

projecting the image signature of the image to be segmented into a feature space with a first learned projection matrix and wherein the identifying of the at least one similar image from the first set of images comprises computing a similarity between the projected image signature and projected image signatures of the images in the first set projected with the learned first projection matrix; and

projecting the image signature of the cropped region into a feature space with a second learned projection matrix and wherein the identifying of the at least one similar image from the second set of images comprises computing a similarity between the projected image signature and projected image signatures of the images in the second set projected with the learned second projection matrix.

11. The method of claim 10, further comprising learning at least one of the first and second projection matrices using segmentation maps and image signatures of a set of training images.

12. The method of claim 11, wherein the learning of the at least one of the first and second projection matrices comprises:

for a plurality of iterations;

drawing a triplet of entries from the set of training images, each entry comprising a segmentation map and an image signature for an image in the training set;

computing a similarity between entries forming a first pair of the entries, based on their segmentation maps;

computing a similarity between entries forming a second pair of the entries, based on their segmentation maps;

generating a first ranking for the first and second pairs based on the computed segmentation map similarities;

computing a similarity between the entries forming the first pair of entries, based on their image signatures projected with a current projection matrix;

computing a similarity between the entries forming the second pair of entries, based on their image signatures projected with the current projection matrix;

generating a second ranking for the first and second pairs based on the computed projected image signature similarities; and

updating the current projection matrix if the first and second rankings do not agree; and

after the plurality of iterations, outputting the updated projection matrix as the projection matrix.

13. The method of claim 1, wherein at least one of:

the identifying of the at least one similar image from the first set of images is based on similarities between a projected image signature of the input image and projected image signatures of the images in the first set of images computed in a feature space in which similarities between projected image signatures of images in a first set of training images are more likely to correlate with similarities based on their segmentation maps than are computed similarities between the image signatures of the training images in the first set, prior to projection; and

the identifying of the at least one similar image from the second set of images is based on similarities between a projected image signature of the cropped image and projected image signatures of the images in the second set of images computed in a feature space in which similarities between projected image signatures of images in a second set of training images are more likely to correlate with similarities based on their segmentation maps than are computed similarities between the image signatures of the training images in the second set, prior to projection.

14. The method of claim 1, wherein the segmenting of the image comprises identifying a target region in one of the cropped region and the input image.

15. The method of claim 1, wherein the input image has an area which is at least twice an area of the cropped region.

16. The method of claim 1, wherein the generating of the image signature for the input image comprise generating a single global image signature for the input image and wherein the identifying of the at least one similar image from the set of images is based on the single global image signature.

17. The method of claim 1, wherein the first set of images comprises images of vehicles each including a license plate and the second set of images comprises images of license plates.

18. The method of claim 17, further comprising identifying a license plate number from a foreground region of the segmented input image.

19. The method of claim 1, wherein at least one of the first and second sets of images comprises synthesized images.

20. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer, causes the computer to perform the method of claim 1.

21. A system for image segmentation comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

22. A system for image segmentation comprising:

memory which stores;

a signature generator configured for generating;

a first image signature for an input image to be segmented, and

a second image signature for a cropped region of the input image;

a retrieval component configured for identifying at least one first similar image from an associated first set of images, based on the first image signature and image signatures of images in the first set of images;

a segmentation component configured for identifying a segmented region of the input image based on segmentation maps of the identified at least one similar image from the first set; and

a cropping component configured for cropping the input image to generate the cropped region, based on the identified segmented region;

the retrieval component being configured for identifying at least one second similar image from an associated second set of images, based on the second image signature and image signatures of images in the second set of images;

the segmentation component being configured for segmenting the input image based on segmentation maps of the identified at least one second similar image from the second set; and

a processor which implements the signature generator, retrieval component, cropping component, and segmentation component.

23. The system of claim 22, further comprising a learned projection matrix for projecting at least one of the first and second image signatures into a different feature space, the projection matrix having been learned on projected image signatures of training images drawn from a set of training images and their respective segmentation maps.

24. A method for forming a system for image segmentation comprising:

providing a signature generator for generating a first image signature for an input image to be segmented and a second image signature for a cropped region of the input image;

learning a first projection matrix, for projecting the first image signature into a first feature space, using image signatures of images drawn from a first training set and their respective segmentation maps;

learning a second projection matrix, for projecting the second image signature into a second feature space, using image signatures of images drawn from a second training set and their respective segmentation maps;

providing a retrieval component for;

identifying at least one similar first image from a first set of images, similarity being computed based on the projected image signature of the input image and the projected image signatures of the images in the first set of images, the projected image signature of the input image being generated by projecting the input image'"'"'s image signature with the first projection matrix, and

identifying at least one similar second image from a second set of images, similarity being computed based on the projected image signature of the cropped region and the projected image signatures of the images in the second set of images, the projected image signature of the cropped region being generated by projecting the cropped region'"'"'s image signature with the second projection matrix, the cropped region being generated based on the segmentation map of the identified at least one similar first image; and

providing a segmentation component for segmenting the input image based on a segmentation map of at least one of the at least one similar second image.

25. A method for image segmentation comprising:

with a computer processor, cropping an input image to generate a cropped region of the input image, based on a computed similarity between a representation of the input image and representations of images in a first set of images for which respective segmentation maps are defined;

identifying a target region within the cropped region, based on a computed similarity between a representation of the cropped region and representations of images in a second set of images for which respective segmentation maps are defined, the images in the second set being each derived from a respective image in the first set by cropping.