Data driven localization using task-dependent representations

• US 9,158,995 B2
• Filed: 03/14/2013
• Issued: 10/13/2015
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Rodriguez-Serrano, Jose Antonio, Larlus-Larrondo, Diane
• Primary Examiner(s): SHAH, UTPAL D
• Application Number:
  US13/803,334
• Publication Number:
  US 20140270350A1
• Time in Patent Office: 943 Days
• Field of Search: 382/105
• US Class Current: 1/1
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/625 License plates

Abstract

A computer implemented method for localization of an object, such as a license plate, in an input image includes generating a task-dependent representation of the input image based on relevance scores for the object to be localized. The relevance scores are output by a classifier for a plurality of locations in the input image, such as patches. The classifier is trained on patches extracted from training images and their respective relevance labels. One or more similar images are identified from a set of images, based on a comparison of the task-dependent representation of the input image and task-dependent representations of images in the set of images. A location of the object in the input image is identified based on object location annotations for the similar images.

Claims

1. A method for object localization in an image comprising:

for an input image, generating a task-dependent representation of the input image based on relevance scores for an object to be localized, comprising;

for each of a plurality of patches of the image, generating a patch-based representation based on low level features extracted from the patch;

with a classifier, outputting a relevance score for each patch, based on the respective patch-based representation; and

generating the task-dependent representation based on the patch relevance scores;

thereafter, identifying at least one similar image from a set of images, based on the task-dependent representation of the input image and task-dependent representations of images in the set of images; and

identifying a location of the object in the input image based on an object location annotation for at least one of the at least one similar images identified in the set of images,

wherein at least one of the generating of the task-dependent representation, identifying of the at least one similar image, and the identifying a location of the object in the input image is performed with a computer processor.

2. The method of claim 1, wherein the generating of the task-dependent representation further comprises:

generating a probability map for the input image based on the patch relevance scores.

3. The method of claim 2, wherein the task-dependent representation comprises a vectorial representation of the probability map.

4. The method of claim 1, wherein at least some of the plurality of patches are overlapping.

5. The method of claim 1, wherein the patch-based representations are each derived from low level features of the respective patch selected from color and gradient features.

6. The method of claim 1, wherein the patch-based representations are output by a generative model built from low level features.

7. The method of claim 6, wherein the patch-based representations comprise Fisher vectors.

8. The method of claim 2, wherein the generating of the task-dependent representation comprises generating a global representation of the image with a generative model based on representations of sub-regions of the image where the contribution of each the sub-regions to the global representation is weighted by the value of the probability map in the sub-region location.

9. The method of claim 1, wherein the identifying of the at least one similar image from the set of images comprises identifying a plurality of similar images and the identifying of the location of the object in the input image is based on object location annotations for the plurality of similar images.

10. The method of claim 1, wherein the identifying of the at least one similar image from the set of images comprises computing a linear kernel between the task-dependent representation of the input image and each of the task-dependent representations of a plurality of the images in the set of images.

11. The method of claim 1, wherein the identifying of the at least one similar image from the set of images comprises computing a similarity between the task-dependent representation of the input image and each of the task-dependent representations of a plurality of the images in the set of images wherein in computing the similarity, the task-dependent representations are embedded with a metric that has been learned on a training set of annotated images and their task-dependent representations.

12. The method of claim 1, wherein the set of images comprises a first set of images and a second set of images and the identifying a location of the object in the input image comprises:

identifying an approximate location of the object in the input image based on the object location annotation for at least one of the at least one similar images identified in the first set of images;

based on the approximate location, identifying a cropped region of the input image;

identifying at least one similar image from the second set of images, based on a task-dependent representation of the cropped region of the input image and task-dependent representations of the images in the second set of images; and

identifying a location of the object in the input image based on an object location annotation for at least one of the at least one similar images identified in the second set of images.

13. The method of claim 1, wherein the object to be located comprises a license plate.

14. The method of claim 1, further comprising extracting information from the image in the identified location.

15. The method of claim 1 further comprising training the classifier on patch-based representations of patches extracted from a set of training images, the patches of the training images being labeled with a label representing the overlap between the patch and a location of an object of in the training image which is in the object class.

16. The method of claim 1, wherein the object to be localized comprises a plurality of objects, each object being in a different object class, and the generating of the task-dependent representation of the input image comprises generating a first task-dependent representation of the input image based on relevance scores for the first class of object at locations in the input image output by a first classifier for the first class of object and generating a second task-dependent representation of the input image based on relevance scores for the second class of object at locations in the input image output by a second classifier for the second class of object.

17. The method of claim 1, further comprising outputting at least one of the location of the object in the input image and information extracted from the image in the object location.

18. A computer program product comprising a non-transitory recording medium storing instructions, which when executed by a computer, perform the method of claim 1.

19. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

20. A system for object localization in images comprising:

a patch representation generator which generates patch-based representations of a plurality of patches of an input image;

a classifier component which classifies each of the patches with a trained classifier model based on the respective patch-based representation;

a signature generator which generates a task-dependent representation of the input image based on the classifications of the patches;

a retrieval component configured for retrieving at least one similar image from a set of images, based on a comparison measure between the task-dependent representation of the input image and task-dependent representations of images in the set of images, the task dependent representations having each been generated by classifying patch-based representations of a plurality of patches of a respective one of the images in the set of images with the trained classifier model and generating the task-dependent representation of the image based on the classifications of the patches;

a segmentation component which segments the input image based on a location of an object in the at least one similar image and identifying a location of an object in the input image based on the segmentation; and

a processor which implements the patch representation generator, classifier component, signature generator, and segmentation component.

21. The system of claim 20, further comprising an information extraction component for extracting information from the identified location.

22. The system of claim 20, wherein the classifier model is trained on patches extracted from images of vehicles in which a license plate is localized and the object to be localized in the input image is a license plate.

23. The system of claim 20, wherein the classifier component classifies patches of images in the set of images with the trained classifier model based on respective patch-based representations of the patches of the images in the set of images and the signature generator generates a task-dependent representation of each of the images in the set of images based on the classifications of the patches of the respective image.

24. A method for object localization in an image comprising:

with a processor;

for each of a set of test images, generating a patch-based representation of a plurality of patches of the test image with a generative model;

classifying each of the test image patches with a trained classifier model based on the respective patch-based representation;

generating a task-dependent representation of each of the test images based on the classifications of the patches of the test image;

generating patch-based representations of a plurality of patches of an input image with the generative model;

classifying each of the patches of the input image with the trained classifier model based on the respective patch-based representation;

generating a task-dependent representation of the input image based on the classifications of the patches of the input image;

retrieving at least one similar test image from the set of test images, based on a comparison measure between the task-dependent representation of the input image and the task-dependent representations of the test images in the set of test images; and

segmenting the input image based on a location of an object in the at least one similar test image and identifying a location of an object in the input image based on the segmentation.