Multiple Cluster Instance Learning for Image Classification

• US 20140270495A1
• Filed: 03/14/2013
• Published: 09/18/2014
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chang, Eric I-Chao, Tu, Zhuowen, Xu, Yan, Zhu, Junyan
• Granted Patent: US 9,317,781 B2
• Time in Patent Office: Days
• US Class Current: 382/160
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/7753 Incorporation of unlabelled data, e.g. multiple instance learning [MIL]

Abstract

The techniques and systems described herein create and train a multiple clustered instance learning (MCIL) model based on image features and patterns extracted from training images. The techniques and systems separate each of the training images into a plurality of instances (or patches), and then learn multiple instance-level classifiers based on the extracted image features. The instance-level classifiers are then integrated into the MCIL model so that the MCIL model, when applied to unclassified images, can perform image-level classification, patch-level clustering, and pixel-level segmentation.

Claims

1. A method comprising:

receiving a plurality of training images, each training image including one or more objects associated with a high-level image classification;

separating each of the plurality of training images into a plurality of instances;

extracting image features from each of the plurality of instances in each of the plurality of training images;

training, via one or more processors configured with executable instructions, multiple instance-level classifiers based on the extracted image features, wherein each of the multiple instance-level classifiers are trained to associate an instance with one of the one or more objects; and

implementing the multiple instance-level classifiers in an image classification model.

2. The method as recited in claim 1, wherein the one or more objects are subclasses of the high-level image classification.

3. The method as recited in claim 1, wherein the one or more objects are components of the high-level image classification.

4. The method as recited in claim 1, wherein each of the instance-level classifiers are configured to perform patch-level clustering where multiple instances are associated with a single object.

5. The method as recited in claim 1, further comprising determining the plurality of instances based on saliency detection.

6. The method as recited in claim 1, wherein the image classification model, when applied to an unclassified image, performs image-level classification, patch-level clustering, and pixel-level segmentation in an integrated framework.

7. The method as recited in claim 1, wherein each of the multiple instance-level classifiers includes a learned weight based on a probability that the instance is associated with the one of the one or more objects.

8. The method as recited in claim 1, wherein the image classification model discovers different objects that cover multiple instances in a single image.

9. The method as recited in claim 1, wherein the high-level image classification is a human-provided label.

10. One or more computer storage media comprising computer-executable instructions that, when executed by one or more processors, perform operations comprising:

receiving input that provides a high-level image classification;

receiving a plurality of unclassified images;

applying an image classification model to the plurality of unclassified images, the image classification model including multiple cluster classifiers that associate image patches with a plurality of different objects associated with the input; and

identifying a portion of the plurality of unclassified images that include at least one of the plurality of different objects associated with the input.

11. The one or more computer storage media as recited in claim 10, wherein the operations further comprise outputting results that indicate the portion of the plurality of unclassified images.

12. The one or more computer storage media as recited in claim 11, wherein the results convey image-level classification, patch-level clustering and pixel-level segmentation.

13. The one or more computer storage media as recited in claim 10, wherein each of the plurality of different objects are a subclass of a class provided by the input.

14. The one or more computer storage media as recited in claim 10, wherein each of the plurality of different objects are a component of a larger object provided by the input.

15. The one or more computer storage media as recited in claim 10, wherein the multiple cluster classifiers associate the image patches with the plurality of different objects based on image features extracted from the image patches.

16. The one or more computer storage media as recited in claim 10, further comprising separating the portion of the plurality of unclassified images into a first subset where each unclassified image contains a first object and a second subset where each unclassified image contains a second object.

17. A system comprising:

one or more processors;

one or more memories, coupled to the one or more processors, storing;

an image access module, operable on the one or more processors, that receives a plurality of training images, each training image including one or more objects associated with a high-level image classification;

a classifier learning module, operable on the one or more processors, that learns multiple instance-level classifiers based on features extracted from each of the plurality of training images, each instance-level classifier configured to associate an image patch with one of the one or more objects; and

a model application model, operable on the one or more processors, that applies the multiple instance-level classifiers to a plurality of unclassified images to discover a subset of images that contain the one or more objects.

18. The system as recited in claim 17, wherein the one or more objects are subclasses of a class defined by the high-level image classification.

19. The system as recited in claim 17, wherein the one or more objects are components of a larger object defined by the image classification.

20. The system as recited in claim 17, wherein each of the instance-level classifiers are configured to perform patch-level clustering where multiple image patches are associated with a single object.