Local feature representation for image recognition

• US 10,043,101 B2
• Filed: 11/07/2014
• Issued: 08/07/2018
• Est. Priority Date: 11/07/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Yang, Jianchao, Brandt, Jonathan
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US14/535,963
• Publication Number:
  US 20160132750A1
• Time in Patent Office: 1,369 Days
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 30/1906 based also on statistics of image patches, e.g. active appearance models of the pattern to be recognised

Abstract

Techniques are disclosed for image feature representation. The techniques exhibit discriminative power that can be used in any number of classification tasks, and are particularly effective with respect to fine-grained image classification tasks. In an embodiment, a given image to be classified is divided into image patches. A vector is generated for each image patch. Each image patch vector is compared to the Gaussian mixture components (each mixture component is also a vector) of a Gaussian Mixture Model (GMM). Each such comparison generates a similarity score for each image patch vector. For each Gaussian mixture component, the image patch vectors associated with a similarity score that is too low are eliminated. The selectively pooled vectors from all the Gaussian mixture components are then concatenated to form the final image feature vector, which can be provided to a classifier so the given input image can be properly categorized.

Claims

1. A computer-implemented method, comprising:

receiving a digital image including a plurality of image features;

dividing the digital image into image patches, such that each image patch represents only a portion of the received digital image;

generating an image patch vector for each image patch;

comparing each image patch vector to Gaussian mixture components of a Gaussian Mixture Model (GMM), each Gaussian mixture component being a vector, thereby generating a similarity score for each image patch vector;

for each Gaussian mixture component, eliminating one or more image patch vectors associated with a similarity score that is below a given threshold;

concatenating a plurality of remaining image patch vectors of all the Gaussian mixture components to generate a final image feature vector that represents the plurality of image features in the received digital image; and

categorizing the digital image using the final image feature vector.

2. The method of claim 1 further comprising providing the final image feature vector to a classifier so the digital image can be categorized.

3. The method of claim 1 wherein, for each Gaussian mixture component, multiple image patch vectors remain, so as to provide k-nearest neighbor pooling.

4. The method of claim 1 wherein, for each Gaussian mixture component, one image patch vector remains, so as to provide 1-nearest neighbor pooling.

5. The method of claim 1 wherein, for one or more Gaussian mixture components, no image patch vectors remain.

6. The method of claim 1 wherein the remaining image patch vectors are represented as matrices, and generating the final image feature vector includes combining those matrices via concatenation.

7. The method of claim 1 wherein an encoding matrix is used to combine select image patch vectors to produce the final image feature vector, and eliminating the one or more image patch vectors includes keeping only a largest value in each row of the encoding matrix and setting all others to be zero.

8. A non-transient computer program product having instructions encoded thereon that when executed by one or more processors causes a process to be carried out, the process comprising:

dividing a digital image into image patches, each of which represents only a portion of the digital image, wherein the digital image includes a plurality of features;

generating an image patch vector for each image patch;

comparing each image patch vector to Gaussian mixture components of a Gaussian Mixture Model (GMM), thereby generating a similarity score for each image patch vector;

for each Gaussian mixture component, eliminating one or more image patch vectors associated with a similarity score that is below a given threshold; and

concatenating a plurality of remaining image patch vectors of all the Gaussian mixture components to generate a final image feature vector that represents the plurality of features in the digital image; and

categorizing the digital image using the final image feature vector.

9. The computer program product of claim 8 wherein, for each Gaussian mixture component, multiple image patch vectors remain, so as to provide posterior thresholding.

10. The computer program product of claim 8 wherein, for each Gaussian mixture component, one image patch vector remains, so as to provide 1-nearest neighbor pooling.

11. The computer program product of claim 8 wherein, for one or more Gaussian mixture components, no image patch vectors remain.

12. The computer program product of claim 8 wherein the remaining image patch vectors are represented as matrices, and generating the final image feature vector includes combining those matrices via concatenation.

13. The computer program product of claim 8 wherein an encoding matrix is used to combine select image patch vectors to produce the final image feature vector, and eliminating the one or more image patch vectors includes keeping only a largest value in each row of the encoding matrix and setting all others to be zero.

14. A computing system, comprising:

an electronic memory for storing executable instructions; and

a processor configured to execute the instructions to;

divide a digital image into image patches, each of which represents only a portion of the digital image, wherein the digital image includes a plurality of features;

generate an image patch vector for each image patch;

compare each image patch vector to Gaussian mixture components of a Gaussian Mixture Model (GMM), each Gaussian mixture component being a vector, thereby generating a similarity score for each image patch vector;

for each Gaussian mixture component, eliminate one or more image patch vectors associated with a similarity score that is below a given threshold;

concatenate a plurality of remaining image patch vectors of all the Gaussian mixture components to generate a final image feature vector that represents the plurality of features in the digital image; and

categorize the digital image using the final image feature vector.

15. The system of claim 14 wherein, for each Gaussian mixture component, multiple image patch vectors remain, so as to provide radius pooling.

16. The system of claim 14 wherein, for each Gaussian mixture component, one image patch vector remains, so as to provide 1-nearest neighbor pooling.

17. The system of claim 14 wherein, for one or more Gaussian mixture components, no image patch vectors remain.

18. The system of claim 14 wherein

the remaining image patch vectors are represented as matrices, and

the processor is further configured to execute the instructions to generate the final image feature vector by combining those matrices via concatenation.

19. The system of claim 14 wherein an encoding matrix is used to combine select image patch vectors to produce the final image feature vector, and eliminating the one or more image patch vectors includes keeping only a largest value in each row of the encoding matrix and setting all others to be zero.