Image classification based on visual words

• US 9,342,758 B2
• Filed: 12/11/2012
• Issued: 05/17/2016
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Xue, Hui
• Primary Examiner(s): Rush, Eric
• Application Number:
  US13/711,500
• Publication Number:
  US 20130148881A1
• Time in Patent Office: 1,253 Days
• Field of Search: 382/100, 382/155, 382/159, 382/168, 382/170, 382/181, 382/190, 382/195, 382/224, 382/225, 382/228
• US Class Current: 1/1
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions

Abstract

The present disclosure introduces a method and an apparatus for classifying images. Classification image features of an image for classification are extracted. Based on a similarity relationship between each classification image feature and one or more visual words in a pre-generated visual dictionary, each classification image feature is quantified by multiple visual words in the visual dictionary and a similarity coefficient between each classification image feature and each of the visual words is determined. Based on the similarity coefficient of each visual word that corresponds to different classification image features, a weight of each visual word is determined to establish a classification visual word histogram of the image for classification. The classification visual word histogram is input into an image classifier that is trained by sample visual word histograms arising from multiple sample images. An output result is used to determine a classification of the image for classification.

Claims

1. A method performed by one or more processors configured with computer-executable instructions, the method comprising:

extracting one or more classification image features from an image for classification;

quantifying, based on a similarity relationship between each classification image feature and each visual word in a pre-generated visual dictionary, each classification image feature by multiple visual words in the visual dictionary and determining a similarity coefficient between each classification image feature and each visual word after the quantifying;

determining, based on one or more similarity coefficients of each visual word corresponding to different classification image features, a weight of each visual word to establish a classification visual word histogram;

inputting the classification visual word histogram into an image classifier; and

using an output of the inputting to determine a classification of the image for classification,

wherein the quantifying each classification image feature by multiple visual words in the visual dictionary and determining the similarity coefficient between each classification image feature and each visual word after the quantifying includes;

calculating, based on the similarity relationship between each classification image feature and the visual words in the pre-generated visual dictionary, a Euclidean distance between each classification image feature and each visual word,

determining a smallest Euclidean distance among calculated Euclidean distances,

determining, with respect to each classification image feature, one or more visual words of which Euclidean distances are within a preset times range of the smallest Euclidean distance as the visual words for quantification of the respective classification image feature, and

calculating, based on the Euclidean distance between the respective classification image feature and each of the visual words for quantification, the one or more similarity coefficients between the respective classification image feature and the visual words, the one or more similarity coefficients being calculated respectively as a percentage relationship of each of the one or more visual words for quantification,

wherein the percentage relationship of a particular visual word for quantification is calculated by dividing the respective Euclidean distance of the particular visual word for quantification by a sum total of the Euclidean distances of the one or more visual words for quantification.

2. The method as recited in claim 1, wherein the image classifier is generated through training by sample visual word histograms from multiple sample images.

3. The method as recited in claim 2, further comprising

comparing, after inputting the classification visual word histogram into an image classifier, the inputted classification visual word histogram to a pre-generated classification visual word histogram in the image classifier to determine a classification of the image for classification.

4. The method as recited in claim 1, wherein the determining the weight of each visual word to establish the classification visual word histogram comprises:

adding up the one or more coefficients of a respective visual word that correspond to different classification image features to calculate the weight of the respective visual word; and

establishing the classification visual word histogram.

5. The method as recited in claim 1, wherein the determining the weight of each visual word to establish the classification visual word histogram comprises:

dividing the image for classification into multiple layer images based on a pyramid image algorithm; and

dividing each layer image to form multiple child images.

6. The method as recited in claim 1, wherein the visual dictionary is generated through clustering of multiple sample image features extracted from multiple sample images.

7. The method as recited in claim 1, wherein the classification visual word histogram increases space information of the image for classification.

8. A system comprising:

one or more processors; and

memory containing computer readable media storing one or more modules including instructions, which when executed by the processors, cause the modules to perform;

extracting one or more classification image features from an image for classification,

quantifying, based on a similarity relationship between each classification image feature and each visual word in a pre-generated visual dictionary, each classification image feature by multiple visual words in the visual dictionary and determining a similarity coefficient between each classification image feature and each visual word after the quantifying,

determining, based on one or more similarity coefficients of each visual word corresponding to different classification image features, a weight of each visual word to establish a classification visual word histogram, and

inputting the classification visual word histogram into an image classifier and using an output to determine a classification of the image for classification,

wherein the quantifying includes;

calculating, based on the similarity relationship between each classification image feature and the visual words in the pre-generated visual dictionary, a Euclidean distance between each classification image feature and each visual word,

determining a smallest Euclidean distance among calculated Euclidean distances and, with respect to each classification image feature, determining one or more visual words of which Euclidean distances are within a preset times range of the smallest Euclidean distance as the visual words for quantification of the respective classification image feature, and

calculating, based on the Euclidean distance between the respective classification image feature and each of the visual words for quantification, the one or more similarity coefficients between the respective classification image feature and the visual words, the one or more similarity coefficients being calculated respectively as a percentage relationship of each of the one or more visual words for quantification, and

wherein the percentage relationship of a particular visual word for quantification is calculated by dividing the respective Euclidean distance of the particular visual word for quantification by a sum total of the Euclidean distances of the one or more visual words for quantification.

9. The apparatus as recited in claim 8, wherein the image classifier is generated through training by sample visual word histograms from multiple sample images.

10. The apparatus as recited in claim 9, wherein the image classifier compares the inputted classification visual word histogram with a pre-generated classification visual word histogram to determine a classification of the image for classification.

11. The apparatus as recited in claim 8, wherein the determining a weight of each visual word includes:

dividing the image for classification into multiple child images based on a pyramid image algorithm,

determining classification image features of each child image,

adding the coefficients of a respective visual word that correspond to each classification image features in a respective child image,

calculating the weight of the respective visual word corresponding to the respective child image to establish a child classification visual word histogram of the respective child image, and

combining each child classification visual word histogram of each child image to establish the classification visual word histogram.

12. The apparatus as recited in claim 8, wherein the classification visual word histogram increases space information of the image for classification.

13. One or more computer storage media including processor-executable instructions that, when executed by one or more processors, direct the one or more processors to perform a method comprising:

extracting one or more classification image features from an image for classification;

quantifying, based on a similarity relationship between each classification image feature and each visual word in a pre-generated visual dictionary, each classification image feature by multiple visual words in the visual dictionary and determining a similarity coefficient between each classification image feature and each visual word after the quantifying;

determining, based on one or more similarity coefficients of each visual word corresponding to different classification image features, a weight of each visual word to establish a classification visual word histogram;

inputting the classification visual word histogram into an image classifier that is generated through training by sample visual word histograms from multiple sample images; and

using an output of the inputting to determine a classification of the image for classification,

wherein the quantifying each classification image feature by multiple visual words in the visual dictionary and determining the similarity coefficient between each classification image feature and each visual word after the quantifying includes;

calculating, based on the similarity relationship between each classification image feature and the visual words in the pre-generated visual dictionary, a Euclidean distance between each classification image feature and each visual word,

determining a smallest Euclidean distance among calculated Euclidean distances,

determining, with respect to each classification image feature, one or more visual words of which Euclidean distances are within a preset times range of the smallest Euclidean distance as the visual words for quantification of the respective classification image feature, and

calculating, based on the Euclidean distance between the respective classification image feature and each of the visual words for quantification, the one or more similarity coefficients between the respective classification image feature and the visual words, the one or more similarity coefficients being calculated respectively as a percentage relationship of each of the one or more visual words for quantification,

wherein the percentage relationship of a particular visual word for quantification is calculated by dividing the respective Euclidean distance of the particular visual word for quantification by a sum total of the Euclidean distances of the one or more visual words for quantification.

14. The one or more computer storage media as recited in claim 13, wherein the method further comprises

comparing, after inputting the classification visual word histogram into an image classifier, the inputted classification visual word histogram to a pre-generated classification visual word histogram in the image classifier to determine a classification of the image for classification.

15. The one or more computer storage media as recited in claim 13, wherein the classification visual word histogram increases space information of the image for classification.