Framework for image thumbnailing based on visual similarity

• US 8,175,376 B2
• Filed: 03/09/2009
• Issued: 05/08/2012
• Est. Priority Date: 03/09/2009
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Marchesotti, Luca, Cifarelli, Claudio, Csurka, Gabriela
• Primary Examiner(s): PERUNGAVOOR, SATHYANARAYA V
• Application Number:
  US12/400,277
• Publication Number:
  US 20100226564A1
• Time in Patent Office: 1,156 Days
• Field of Search: 382/159
• US Class Current: 382/159
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects

Abstract

An apparatus and method for detecting a region of interest in an image are disclosed. Image representations for a set of images that have been manually annotated with regions of interest are stored, along with positive and negative representations of each image which are similarly derived to the image representations except that they are based on features extracted from patches within the region of interest and outside it, respectively. For an original image for which a region of interest is desired, the stored information for K similar images is automatically retrieved and used to train a classifier. The trained classifier provides, for each patch of the original image, a probability of being in a region of interest, based extracted features of the patch (represented, for example, as a Fisher vector), which can be used to determine a region of interest in the original image.

Claims

1. A method for detecting a region of interest in an image comprising:

for each image in a dataset of images for which a region of interest has been established respectively, storing a dataset image representation based on features extracted from the image;

for an original image for which a region of interest is to be detected;

generating an original image representation for the original image based on features extracted from the image;

identifying a subset of similar images from the images in the dataset, the identified subset being based on a measure of similarity between the original image representation and respective dataset image representations;

training a classifier with information extracted from the established regions of interest of the subset of similar images, the information for training the classifier comprising at least one of;

positive examples, comprising information extracted from patches of the identified similar images that are within the established region of interest, and

negative examples comprising information extracted from patches of the identified similar images that are not within the established region of interest; and

with the trained classifier, identifying a region of interest in the original image.

2. The method of claim 1, wherein each dataset image representation is based on features of patches extracted from the dataset image and wherein the original image representation is based on features of patches extracted from the original image.

3. The method of claim 2, wherein for each patch, a vector is generated, based on the extracted features.

4. The method of claim 3, wherein a plurality of types of features is extracted and wherein for each patch, one vector is generated for each of a type of feature extracted.

5. The method of claim 1, wherein the extracted features are selected from the group consisting of color features, texture features, and combinations thereof.

6. The method of claim 1, wherein the established regions of interest are generated from salient regions identified by a set of human observers.

7. The method of claim 1, wherein the identifying a subset of similar images in the dataset, based on a measure of similarity between the original image representation and each dataset image representation comprises computing a distance metric between the original image and images in the dataset.

8. The method of claim 1, wherein the identified region of interest in the original image is processed for more precisely identifying the region of interest.

9. The method of claim 1, wherein the identified region of interest in the original image is processed with a graph-cut technique.

10. The method of claim 1, further comprising generating a crop based on the region of interest which removes image data from the original image outside a crop area.

11. The method of claim 1, further comprising, outputting the identified region of interest in the original image.

12. The method of claim 11, wherein the outputting includes outputting a crop of the original image based on the identified region of interest.

13. The method of claim 12, further comprising, inputting the crop into a categorizer which has been trained on annotated image crops to identify a class for the original image.

21. A non-transitory computer program product encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1.

22. An apparatus for detecting a region of interest in an image comprising:

memory which stores;

at least one of a) the dataset image representations and b) feature vectors of patches of the dataset images from which the image representations are able to be generated, and

instructions for performing the method of claim 1; and

a processor with access to the instructions and dataset image representations which executes the instructions.

14. A method for detecting a region of interest in an image comprising:

for each image in a dataset of images for which a region of interest has been established respectively, storing a dataset image representation based on features extracted from the image, each dataset image representation beings based on features of patches extracted from the dataset image;

for an original image for which a region of interest is to be detected;

generating an original image representation for the original image based on features extracted from the image, the original image representation being based on features of patches extracted from the original image;

identifying a subset of similar images from the images in the dataset, the identified subset being based on a measure of similarity between the original image representation and respective dataset image representations;

training a classifier with information extracted from the established regions of interest of the subset of similar images;

with the trained classifier, identifying a region of interest in the original image;

wherein for each patch, a vector is generated, based on the extracted features, the vector comprising a Fisher vector.

15. The method of claim 14, wherein patches of the dataset images are identified as salient or non-salient, based on whether they are within the established region of interest or not.

16. The method of claim 15, wherein the dataset image representations are each derived from a +ve and a −

ve high level representation of the image, the +ve high level representation being based on features of salient patches from and the −

ve high level representation is based on features of non-salient patches.

17. The method of claim 16, wherein the classifier is trained with the +ve high level representations and the −

ve high level representations of the similar images.

18. The method of claim 14, wherein the information for training the classifier comprises at least one of:

positive examples, comprising information extracted from patches of the identified similar images that are within the established region of interest, and

negative examples comprising information extracted from patches of the identified similar images that are not within the established region of interest.

19. A method for detecting a region of interest in an image comprising:

for each image in a dataset of images for which a region of interest has been established respectively, storing a dataset image representation based on features extracted from the image,

wherein, for each of a plurality of the images in the dataset of images, a semantic label is associated with the established region of interest, the semantic label being selected from a set of semantic labels, each relating to a different context; and

for an original image for which a region of interest is to be detected;

generating an original image representation for the original image based on features extracted from the image;

identifying a subset of similar images from the images in the dataset, the identified subset being based on a measure of similarity between the original image representation and respective dataset image representations, and where a concept is specified, the identifying of the subset of similar images from the images in the dataset considers the semantic labels of the images in selecting the subset of similar images;

training a classifier with information extracted from the established regions of interest of the subset of similar images; and

with the trained classifier, identifying a region of interest in the original image.

20. A method for detecting a region of interest in an image comprising:

for each image in a dataset of images for which a region of interest has been established respectively, storing a dataset image representation based on features extracted from the image;

for an original image for which a region of interest is to be detected;

generating an original image representation for the original image based on features extracted from the image;

identifying a subset of similar images from the images in the dataset, the identified subset being based on a measure of similarity between the original image representation and respective dataset image representations;

training a classifier with information extracted from the established regions of interest of the subset of similar images; and

with the trained classifier, identifying a region of interest in the original image, comprising;

outputting a saliency map from the classifier in which patches of the image at multiple scales are each assigned a saliency value;

partitioning the image into a set of overlapping windows;

assigning each window a saliency score based on the saliency values of patches within the window;

assigning each pixel of the image a saliency value based on the saliency values of windows in which the pixel is located.

23. An apparatus for detecting a region of interest in an image comprising:

memory which, for a dataset of images for which a respective region of interest has been established, stores a set of dataset image representations, each dataset image representation being derived from features extracted from a respective one of the images in the dataset;

memory which stores instructions which, for an original image for which a region of interest is to be detected;

generate an original image representation for the original image based on features extracted from the original image;

identify a subset of similar images in the dataset, based on a measure of similarity between the original image representation and each dataset image representation;

train a classifier to identify a region of interest in the original image, the classifier being trained with positive and negative examples, each of the positive examples comprising a high level representation based on features extracted from the established region of interest of a respective one of the subset of similar images and each of the negative examples comprising a high level representation based on features extracted from outside the established region of interest of a respective one of the subset of similar images.

24. A method for detecting a region of interest in an image comprising:

storing a set of image representations, each image representation being based on features extracted from patches of a dataset image, where for each dataset image, the patch features are identified as salient or non-salient based on whether or not the patch is within a manually identified region of interest; and

for an original image for which a region of interest is to be detected;

generating an original image representation for the original image based on features extracted from patches of the image;

computing a distance measure between the original image representation and image representations in the set of image representations to identify a subset of similar image representations from the set of image representations;

training a classifier with positive and negative examples extracted from the images corresponding to subset of similar image representations, the positive examples each being based on the salient patch features of a respective image and the negative examples being based on non-salient patch features of the respective image; and

with the trained classifier, identifying a region of interest in the original image based on the patch features of the original image.

25. The method of claim 24, wherein the patch features are represented by Fisher vectors.