Visual language modeling for image classification

• US 8,126,274 B2
• Filed: 08/30/2007
• Issued: 02/28/2012
• Est. Priority Date: 08/30/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Ma, Wei-Ying, Li, Zhiwei, Wu, Lei
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US11/847,959
• Publication Number:
  US 20090060351A1
• Time in Patent Office: 1,643 Days
• Field of Search: 382/173, 382/181, 382/209, 382/224, 382/229, 382/230, 382/305, 704/251, 704/256.2
• US Class Current: 382/224
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06V 10/424 Syntactic representation, e.g. by using alphabets or grammars
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Systems and methods for visual language modeling for image classification are described. In one aspect the systems and methods model training images corresponding to multiple image categories as matrices of visual words. Visual language models are generated from the matrices. In view of a given image, for example, provided by a user or from the Web, the systems and methods determine an image category corresponding to the given image. This image categorization is accomplished by maximizing the posterior probability of visual words associated with the given image over the visual language models. The image category, or a result corresponding to the image category, is presented to the user.

Claims

1. A method at least partially implemented by a computing device, the method comprising:

modeling images representing multiple image categories as respective matrices of visual words;

generating visual language models from the respective matrices of visual words, the generating comprising;

correlating visual words in the matrices of visual words according to a visual word grammar indicating conditional distribution of the visual words; and

for each category of the multiple image categories, building respective visual language models based on the conditional distribution of the visual words;

estimating an image category for an image in view of the visual language models; and

presenting the image category or a result based on the image category to a user.

2. The method of claim 1, wherein modeling the images as respective matrices of visual words, the images are training images, and wherein the modeling further comprises:

for each training image;

dividing the training image into multiple image patches, each image patch being a group of pixels;

for each image patch of the image patches;

extracting features to describe one or more properties of the patch;

representing at least a subset of the features as one or more multidimensional vectors;

transforming, in view of a visual word grammar, the one or more multidimensional vectors into a respective hash code, the respective hash code being a visual word of the visual words, the visual word being in a visual document corresponding to the training image.

3. The method of claim 2, wherein the method further comprises selecting the features to emphasize aspects of a category corresponding to the training image.

4. The method of claim 1, wherein the visual grammar indicates that visual words are conditionally dependent on only previous visual words.

5. The method of claim 1, wherein a first model of the visual language models treats visual words corresponding to the training image as independent visual words, a second model of the visual language models is based on proximity between neighboring visual words, and a third model of the visual language models is based on visual word dependency on immediate vertical and horizontal neighboring visual words.

6. The method of claim 5, wherein the first model is a unigram model, the second model is a bigram model, and the third model is a trigram model.

7. The method of claim 1, wherein the images are training images, and wherein estimating the image category further comprises:

generating a visual document for the image, the visual document comprising a matrix of visual words;

maximizing, for each category of the multiple image categories, conditional distribution of n-grams of individual words of the visual words with respect to the visual language models; and

calculating the image category to be a category associated with a maximum posterior probability to indicate a likelihood that the image is generated by the category, the maximum posterior probability being a generalization probability represented by a product of all conditional probabilities of n-gram.

8. The method of claim 7, wherein “

n”

in the n-grams comprises 1, 2, and 3.

9. A computer storage device storing computer executable instructions, which, when executed by the computer, cause the computer to perform operations, the operations comprising:

building visual language models from matrices of visual words generated from a set of training images, the visual language models being based on a visual word grammar, the training images corresponding to one or more predetermined image classifications, the building comprising;

for each training image;

dividing the training image into multiple image patches, each image patch being a group of pixels;

for each image patch of the image patches;

extracting features to describe one or more properties of the patch;

representing at least a subset of the features as one or more multidimensional vectors;

transforming, in view of a visual word grammar, the one or more multidimensional vectors into a respective hash code, the respective hash code being a visual word of the visual words;

creating a visual document from an image for image categorization;

determining an image category for the image based on characteristics of the visual document in view of the visual language models, and the image category corresponding to a classification of the one or more predetermined image classifications; and

presenting the image category or a result based on the image category to a user.

10. The computer storage medium of claim 9, wherein the visual word grammar indicates that visual words are conditionally dependent on other visual words according to a predetermined order of word generation.

11. The computer storage medium of claim 9, wherein the operations further comprise operations for selecting the features to emphasize aspects of a classification corresponding to the training image.

12. The computer storage medium of claim 9, wherein the operations further comprise operations for:

correlating the visual words according to a conditional distribution of the visual words; and

for each classification of the one or more predetermined image classifications, building respective visual language models based on the conditional distribution of the visual words, a first model of the visual language models treating visual words corresponding to the training image as independent visual words, a second model of the visual language models being based on proximity between two neighboring visual words, a third model of the visual language models being based on visual word dependency on immediate vertical and horizontal neighboring visual words.

13. The computer storage medium of claim 12, wherein the first model is a unigram model, the second model is a bigram model, and the third model is a trigram model.

14. The computer storage medium of claim 9, wherein determining the image category further comprises:

generating a visual document for the image, the visual document comprising a matrix of visual words;

maximizing, for each classification of the one or more predetermined image classifications, conditional distribution of n-grams of individual words of the visual words with respect to the visual language models; and

calculating the image category to be a category associated with a maximum posterior probability to indicate a likelihood that the image is generated by the category, the maximum posterior probability being a generalization probability represented by a product of all conditional probabilities of n-gram.

15. A computing device comprising:

a processor; and

a memory couple to the processor, the memory including computer-program instructions encoded thereon, the computer-program instructions, when executed by the processor, for performing operations comprising;

loading a set of training images associated with corresponding image categories;

for each training image of the training images;

(a) dividing the training image into a respective set of image patches;

(b) generating a visual word for each image patch to form a respective visual document for the training image;

for each category of the one or more image categories, generating visual language model(s);

estimating, using the visual language model(s), an image category for a given image comprising;

generating a visual document comprising respective visual words from the given image to determine a conditional distribution of the visual words over respective ones of these visual language model(s), a visual language model associated with a largest conditional distribution of the visual words indicating the image category; and

presenting the image category or a result corresponding to the image category to a user.

16. The computing device of claim 15, wherein the visual language model(s) comprise a unigram visual life which model, a bigram visual language model, and a trigram visual language model.

17. The computing device of claim 15:

wherein generating the visual word for each image patch further comprises extracting one or more features from the image patch to describe properties of the image patch, the one or more features being selected according to a image category corresponding to the training image; and

wherein the features are used to generate a model of the visual language model(s).