Optimizing multi-class image classification using patch features

• US 10,013,637 B2
• Filed: 01/22/2015
• Issued: 07/03/2018
• Est. Priority Date: 01/22/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Misra, Ishan, Li, Jin, Hua, Xian-Sheng
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US14/602,494
• Publication Number:
  US 20160217344A1
• Time in Patent Office: 1,258 Days
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

Optimizing multi-class image classification by leveraging patch-based features extracted from weakly supervised images to train classifiers is described. A corpus of images associated with a set of labels may be received. One or more patches may be extracted from individual images in the corpus. Patch-based features may be extracted from the one or more patches and patch representations may be extracted from individual patches of the one or more patches. The patches may be arranged into clusters based at least in part on the patch-based features. At least some of the individual patches may be removed from individual clusters based at least in part on determined similarity values that are representative of similarity between the individual patches. The system may train classifiers based in part on patch-based features extracted from patches in the refined clusters. The classifiers may be used to accurately and efficiently classify new images.

Claims

1. A computer storage media encoded with instructions that, when executed by a processor, configure a computer to perform acts comprising:

accessing a plurality of weakly supervised images;

extracting one or more patches from individual weakly supervised images of the plurality of weakly supervised images;

extracting patch-based features from the one or more patches;

arranging individual patches into a plurality of clusters based at least in part on the patch-based features;

removing at least some of the individual patches from at least one cluster of the plurality of clusters based at least in part on similarity values representative of a similarity between ones of the individual patches arranged in the at least one cluster; and

training a classifier for at least one label of a plurality of labels based at least in part on the plurality of clusters, comprising;

extracting new patch-based features from remaining individual patches of the at least one cluster; and

training the classifier based at least in part on the new patch-based features.

2. The computer storage media as claim 1 recites, wherein the acts further comprise, prior to extracting the one or more patches from the individual weakly supervised images, filtering the plurality of weakly supervised images, the filtering including:

determining nearest neighbors for each individual weakly supervised image of the plurality of weakly supervised images;

arranging one or more individual weakly supervised images within a predetermined distance into image clusters;

determining an entropy value for each individual weakly supervised image in an individual image cluster of the image clusters, wherein determining an entropy value for each individual weakly supervised image comprises determining a similarity value representing a similarity between each individual weakly supervised image in a particular image cluster and each individual weakly supervised image in one or more other image clusters; and

removing at least some of the individual weakly supervised images when the entropy value is above a predetermined threshold.

3. The computer storage media as claim 1 recites, wherein the acts further comprise, prior to extracting the one or more patches from the individual weakly supervised images, filtering the plurality of weakly supervised images, the filtering including:

collecting negative images;

comparing the individual weakly supervised images with the negative images; and

removing one or more of the individual weakly supervised images from the plurality of weakly supervised images based at least in part on the one or more of the individual weakly supervised images and the negative images having similarity values above a predetermined threshold.

4. The computer storage media as claim 1 recites, further comprising:

extracting patch representation from the one or more patches, wherein extracting patch representations from the individual patches comprises extracting features from the individual patches via latent Dirichlect allocation (LDA).

5. The computer storage media as claim 4 recites, wherein the acts further comprise determining the similarity values based at least in part on the patch representations.

6. The computer storage media as claim 5 recites, wherein determining the similarity values comprises:

standardizing patch representations extracted from a first individual patch of the individual patches and a second individual patch of the individual patches to a predetermined canonical size; and

computing a dot product based at least in part on the standardized patch representations of the first individual patch and the second individual patch.

7. The computer storage media as claim 6 recites, wherein the first individual patch and the second individual patch are part of a same cluster of the plurality of clusters, the same cluster associated with a same label of the plurality of labels.

8. The computer storage media as claim 1 recites, wherein removing at least some of the individual patches from the individual clusters comprises:

accessing a plurality of individual patches in a particular cluster of the plurality of clusters;

determining nearest neighbors of individual patches of the plurality of individual patches based at least in part on the similarity values;

determining entropy values for the individual patches based at least in part on determining the nearest neighbors of the individual patches; and

removing at least some individual patches with entropy values above a predetermined threshold.

9. The computer storage media as claim 8 recites, wherein the acts further comprise:

ordering the individual patches based at least in part on the entropy values associated with the individual patches;

processing the ordered individual patches via a plurality of iterations, the processing including determining nearest neighbor patches for each of the ordered individual patches;

removing nearest neighbor patches for each of the ordered individual patches in each iteration of the plurality of iterations;

determining that a particular individual patch of the individual patches had a number of nearest neighbors above a predetermined threshold with entropy values below a predetermined threshold; and

selecting the particular individual patch for training the classifier.

10. A system comprising:

one or more processing units; and

computer storage media storing instructions that when executable by the one or more processing units cause the system to perform operations comprising;

accessing a plurality of weakly supervised images;

extracting one or more patches from individual weakly supervised images of the plurality of weakly supervised images;

extracting patch-based features from the one or more patches;

arranging individual patches into a plurality of clusters based at least in part on the patch-based features;

removing at least some of the individual patches from at least one cluster of the plurality of clusters based at least in part on similarity values representative of similarity between ones of the individual patches arranged in the at least one cluster; and

training a classifier for at least one label of a plurality of labels based at least in part on the plurality of clusters, comprising;

extracting new patch-based features from remaining individual patches of the at least one cluster; and

training the classifier based at least in part on the new patch-based features.

11. The system as claim 10 recites, wherein the operations further comprise determining the similarity values based at least in part on:

extracting patch representations from the one or more patches;

standardizing patch representations extracted from a first individual patch of the individual patches and a second individual patch of the individual patches to a predetermined canonical size; and

computing a dot product based at least in part on the standardized patch representations of the first individual patch and the second individual patch.

12. The system as claim 11 recites, wherein the first individual patch and the second individual patch are part of a same cluster of the plurality of clusters, the same cluster associated with a same label of the plurality of labels.

13. The system as claim 10 recites, wherein removing at least some of the individual patches from the individual clusters comprises:

accessing a plurality of individual patches in a particular cluster of the plurality of clusters;

determining nearest neighbors of individual patches of the plurality of individual patches based at least in part on the similarity values;

determining entropy values for the individual patches based at least in part on determining the nearest neighbors of the individual patches; and

removing at least some individual patches with entropy values above a predetermined threshold.

14. The system as claim 13 recites, the operations further comprising:

ordering the individual patches based at least in part on the entropy values associated with the individual patches;

processing the ordered individual patches via a plurality of iterations, the processing including determining nearest neighbor patches for each of the ordered individual patches;

removing nearest neighbor patches for each of the ordered individual patches in each iteration of the plurality of iterations;

determining that a particular individual patch of the individual patches had a number of nearest neighbors above a predetermined threshold with entropy values below a predetermined threshold; and

selecting the particular individual patch for training the classifier.

15. A computer-implemented method comprising:

accessing a plurality of weakly supervised images;

extracting one or more patches from individual weakly supervised images of the plurality of weakly supervised images;

extracting patch-based features from the one or more patches;

arranging individual patches into a plurality of clusters based at least in part on the patch-based features;

removing at least some of the individual patches from at least one cluster of the plurality of clusters based at least in part on similarity values representative of similarity between ones of the individual patches arranged in the at least one cluster; and

training a classifier for at least one label of a plurality of labels based at least in part on the plurality of clusters, comprising;

extracting new patch-based features from remaining individual patches of the at least one cluster; and

training the classifier based at least in part on the new patch-based features.

16. The computer-implemented method as claim 15 recites, wherein the operations further comprise determining the similarity values based at least in part on:

extracting patch representations from the one or more patches;

standardizing patch representations extracted from a first individual patch of the individual patches and a second individual patch of the individual patches to a predetermined canonical size; and

computing a dot product based at least in part on the standardized patch representations of the first individual patch and the second individual patch.

17. The computer-implemented method as claim 15 recites, wherein removing at least some of the individual patches from the individual clusters comprises:

accessing a plurality of individual patches in a particular cluster of the plurality of clusters;

determining nearest neighbors of individual patches of the plurality of individual patches based at least in part on the similarity values;

determining entropy values for the individual patches based at least in part on determining the nearest neighbors of the individual patches; and

removing at least some individual patches with entropy values above a predetermined threshold.

18. The computer-implemented method as claim 17 recites, the operations further comprising:

ordering the individual patches based at least in part on the entropy values associated with the individual patches;

processing the ordered individual patches via a plurality of iterations, the processing including determining nearest neighbor patches for each of the ordered individual patches;

removing nearest neighbor patches for each of the ordered individual patches in each iteration of the plurality of iterations;

determining that a particular individual patch of the individual patches had a number of nearest neighbors above a predetermined threshold with entropy values below a predetermined threshold; and

selecting the particular individual patch for training the classifier.

19. The computer-implemented method as claim 16 recites, wherein the first individual patch and the second individual patch are part of a same cluster of the plurality of clusters, the same cluster associated with a same label of the plurality of labels.

20. The computer-implemented method as claim 16 recites, wherein the acts further comprise determining the similarity values based at least in part on the patch representations.