Generic visual classification with gradient components-based dimensionality enhancement

• US 20070258648A1
• Filed: 05/05/2006
• Published: 11/08/2007
• Est. Priority Date: 05/05/2006
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Perronnin, Florent
• Granted Patent: US 7,680,341 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

In an image classification system (70), a plurality of generative models (30) correspond to a plurality of image classes. Each generative model embodies a merger of a general visual vocabulary and an image class-specific visual vocabulary. A gradient-based class similarity modeler (40) includes (i) a model fitting data extractor (46) that generates model fitting data of an image (72) respective to each generative model and (ii) a dimensionality enhancer (50) that computes a gradient-based vector representation of the model fitting data with respect to each generative model in a vector space defined by the generative model. An image classifier (76) classifies the image respective to the plurality of image classes based on the gradient-based vector representations of class similarity.

Claims

1. A method for classifying an image, the method comprising:

extracting model fitting data for the image respective to a generative model that includes parameters relating to visual words of at least an image class-specific visual vocabulary;

computing a higher-dimensionality representation of the model fitting data that includes at least some components of a gradient of the model fitting data in a vector space defined by the parameters of the generative model;

repeating the extracting and computing for a plurality of generative models each having at least a different image class-specific vocabulary corresponding to a different class of images; and

classifying the image based on the higher-dimensionality representations.

2. The method as set forth in claim 1, wherein the generative model further includes parameters relating to visual words of a general visual vocabulary.

3. The method as set forth in claim 2, further comprising:

deriving the general visual vocabulary from a set of training images;

adapting the general visual vocabulary to a selected image class sub-set of the training images to generate each image class-specific visual vocabulary; and

forming each generative model at least in part by merging the general visual vocabulary and the corresponding image class-specific visual vocabulary.

4. The method as set forth in claim 1, wherein the extracting of the model fitting data comprises:

extracting a features vector from each of a plurality of key-patch regions of the image to define a set of feature vectors;

computing a likelihood of the set of feature vectors respective to the generative model; and

computing a logarithm of the likelihood.

5. The method as set forth in claim 4, wherein the computing of the higher-dimensionality representation comprises:

computing at least some components of the gradient of the logarithm of the likelihood scaled by a count of the feature vectors in the set of feature vectors.

6. The method as set forth in claim 1, wherein the computing of the higher-dimensionality representation comprises:

computing a sub-set of the components of the gradient of the model fitting data in the vector space defined by the parameters of the generative model.

7. The method as set forth in claim 1, wherein the generative model is a Gaussian mixture model including at least a weighting parameter, a features vector mean parameter, and a features vector covariance parameter corresponding to each visual word of the generative model.

8. The method as set forth in claim 7, wherein the computing of the higher-dimensionality representation comprises one of:

computing components of the gradient along only dimensions of the vector space defined by the mean parameters of the generative model;

computing components of the gradient along only dimensions of the vector space defined by diagonals of the covariance parameters of the generative model;

computing components of the gradient along only dimensions of the vector space defined by mean parameters of the generative model and by diagonals of the covariance parameters of the generative model; and

computing components of the gradient along dimensions of the vector space defined by weighting parameters of the generative model and defined by mean parameters of the generative model and by diagonals of the covariance parameters of the generative model.

9. The method as set forth in claim 1, wherein the parameters of the generative model include at least one multi-dimensional vector parameter with dimensions corresponding to dimensions of a features vector characterizing the image, and the computing of the higher-dimensionality representation comprises:

computing components of the gradient of the model fitting data along the dimensions of the at least one multi-dimensional vector parameter.

10. The method as set forth in claim 9, wherein the multi-dimensional vector parameter has dimensionality D corresponding to the D dimensions of the features vector characterizing the image, the generative model contains N visual words, and the computed higher-dimensionality representation has dimensionality greater than N×

D.

11. The method as set forth in claim 1, further comprising:

whitening the higher-dimensionality representations of the image prior to the classifying.

12. An image classification system comprising:

a plurality of generative models corresponding to a plurality of image classes, each generative model embodying at least an image class-specific visual vocabulary;

a gradient-based class similarity modeler including (i) a model fitting data extractor that generates model fitting data of an image respective to each generative model and (ii) a dimensionality enhancer that computes a gradient-based vector representation of the model fitting data with respect to each generative model in a vector space defined by the generative model; and

an image classifier that classifies the image respective to the plurality of image classes based on the gradient-based vector representations.

13. The image classification system as set forth in claim 12, wherein each generative model embodies a merger of a general visual vocabulary and an image class-specific visual vocabulary.

14. The image classification system as set forth in claim 12, wherein each generative model has dimensionality of at least N×

D where N is a count of vocabulary words and D is a number of features in features vectors characterizing the image, and the gradient-based vector representation computed by the dimensionality enhancer has dimensionality of at least N×

D.

15. The image classification system as set forth in claim 12, wherein (i) the model fitting data extractor generates a comparison of the image with each generative model that incorporates a conditional likelihood of a set of feature vectors characterizing the image respective to the generative model, and (ii) the dimensionality enhancer computes the gradient-based vector representations as gradients of the conditional likelihoods respective to at least some parameters of the generative models.

16. A method for classifying an image, the method comprising:

extracting model fitting data for the image respective to a generative model embodying a merger of a general visual vocabulary and an image class-specific visual vocabulary;

increasing a dimensionality of the model fitting data by computing derivatives of the model fitting data in a vector space defined by parameters of the generative model;

repeating the extracting and increasing for a plurality of generative models each embodying a merger of the general visual vocabulary and a different image class-specific vocabulary for a different image class; and

classifying the image based on the increased-dimensionality model fitting data.

17. The method as set forth in claim 16, further comprising:

deriving a general visual vocabulary from a set of training images;

adapting the general visual vocabulary to a first selected sub-set of the training images associated with a first class to generate a first image class-specific visual vocabulary;

merging the general visual vocabulary and the first image class-specific visual vocabulary, the generative model for the first class embodying the merger of the general visual vocabulary and the first image class-specific visual vocabulary; and

repeating the adapting and merging for each class to construct the generative models for the plurality of classes.

18. The method as set forth in claim 16, wherein the increasing comprises:

for one or more parameters of the generative model, computing a derivative for each feature of a features-based representation of the image.

19. The method as set forth in claim 16, wherein each generative model has parameters p_n,j(d) where index n denotes a visual word, the index j denotes a parameter type, and index d denotes a feature dimension of the features-based representation, and the increasing of dimensionality comprises:

computing n×

d derivatives of the model fitting data respective to at least one parameter type j.

20. The method as set forth in claim 19, wherein each generative model is a Gaussian mixing model, and the parameter types indexed by the index j include weighting, mean, and covariance parameter types for each visual word n.