Modeling images as sets of weighted features

• US 20100189354A1
• Filed: 01/28/2009
• Published: 07/29/2010
• Est. Priority Date: 01/28/2009
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: de Campos, Teofilo E., Perronnin, Florent C.
• Granted Patent: US 8,774,498 B2
• Time in Patent Office: Days
• US Class Current: 382/190
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

An apparatus, method, and computer program product are provided for generating an image representation. The method includes receiving an input digital image, extracting features from the image which are representative of patches of the image, generating weighting factors for the features based on location relevance data for the image, and weighting the extracted features with the weighting factors to form a representation of the image.

Claims

1. A method for generating an image representation comprising:

receiving an input digital image;

extracting features from the image which are representative of patches of the image;

generating weighting factors for the features based on location relevance data for the image;

weighting the extracted features with the weighting factors to form a representation of the image.

2. The method of claim 1, wherein the patches are defined on a regular grid at one or more scales.

3. The method of claim 1 wherein the extraction of features comprises:

extracting low level features from the patches; and

generating high level features by inputting the low level features into a trained model.

4. The method of claim 3, wherein the trained model comprises a Gaussian mixture model.

5. The method of claim 4, wherein the high level features comprise Fisher Kernel gradient vectors.

6. The method of claim 1, wherein the weighting factors of the image representation are normalized to sum to a predetermined value.

7. The method of claim 1, wherein the location relevance data for the image comprises eye gaze data.

8. The method of claim 1, wherein the location relevance data for the image comprises automatically generated relevance data derived from the image.

9. The method of claim 1, wherein the generating of the weighting factors for the features comprises generating a saliency map in which pixels or groups of pixels of the image are assigned a relevance value based on the location relevance data.

10. The method of claim 9 wherein a weighting factor ψ

_t is computed for each patch by one of;

setting ψ

_t to the relevance value ψ

(m,n) where (m,n) corresponds to the coordinates of the patch center;

setting ψ

_t as the average of all the relevance values of ψ

(m,n) in the patch; and

setting ψ

_t as the maximum of the relevance values of ψ

(m,n) in the patch.

11. The method of claim 9, wherein the location relevance data for the image comprises eye gaze data and the generating of the saliency map comprises applying Gaussian functions to eye gaze points in the eye gaze data.

12. The method of claim 9, wherein the location relevance data for the image comprises eye gaze data and wherein in generating the saliency map, pixels or groups of pixels of the image are assigned a relevance value as a function of a proximity to gaze points of the eye gaze data.

13. The method of claim 1, further comprising outputting the image representation associated to the image.

14. A method of computing a similarity between images comprising generating representations of at least two original images by the method of claim 1, the method further comprising:

computing a measure of similarity between a first of the original images and at least a second of the original images based on their respective image representations.

15. A method of classifying images comprising:

generating image representations for a plurality of training images by the method of claim 1;

training a classifier based on the image representations and respective class labels of the training images; and

with the trained classifier, assigning a class to a new image based on its image representation.

16. The method of claim 1 further comprising:

clustering a set of images into at least two groups, based on image representations of the images in the set.

17. A computer program product which encodes instructions, which when executed by a computer, perform the method of claim 1.

18. A representation of an image generated by the method of claim 1 comprising a set of weighted features.

19. An apparatus for generating an image representation comprising:

memory which stores instructions for performing the method of claim 1; and

a processor in communication with the memory for executing the instructions.

20. A computer implemented apparatus for generating an image representation comprising:

memory for storing an image to be processed;

an image representation generator which outputs a representation of an original image as a set of weighted features, the weights for the features being derived from location relevance data for the image; and

a source of location relevance data in communication with the image representation generator.

21. The apparatus of claim 20, wherein the source of location relevance data comprises an eye gaze detector.

22. A method for generating an image representation comprising:

receiving an input digital image;

extracting features from the image which are representative of patches of the image without considering local relevance;

acquiring location relevance data comprising gaze data for at least one observer of the image;

generating a saliency map in which pixels of the image are assigned relevance values based on the location relevance data;

generating weighting factors for the features based on relevance values in regions the saliency map corresponding to the respective patches; and

weighting the extracted features with the weighting factors to form a representation of the image.