IMAGE CLASSIFICATION

• US 20090252413A1
• Filed: 04/04/2008
• Published: 10/08/2009
• Est. Priority Date: 04/04/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Viola, Paul, Hua, Gang
• Granted Patent: US 8,131,066 B2
• Time in Patent Office: Days
• US Class Current: 382/170
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/245 relating to the decision surface
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Images are classified as photos (e.g., natural photographs) or graphics (e.g., cartoons, synthetically generated images), such that when searched (online) with a filter, an image database returns images corresponding to the filter criteria (e.g., either photos or graphics will be returned). A set of image statistics pertaining to various visual cues (e.g., color, texture, shape) are identified in classifying the images. These image statistics, combined with pre-tagged image metadata defining an image as either a graphic or a photo, may be used to train a boosting decision tree. The trained boosting decision tree may be used to classify additional images as graphics or photos based on image statistics determined for the additional images.

Claims

1. A method for classifying photo and graphic images using features generated from image statistics, the method comprising:

determining one or more classifying features of an image, comprising at least one of;

using a gradient magnitude-orientation histogram to determine one or more classifying features of the image, comprising;

generating a gradient magnitude-orientation histogram for the image; and

measuring a difference between an energy value of a lower frequency color band and an energy value of a higher frequency color band using the gradient magnitude-orientation histogram;

using multidimensional independent histograms for respective red, green, and blue color bands to determine one or more classifying features of the image, comprising;

generating respective multidimensional independent histograms corresponding to the red, green, and blue color bands for the image;

using the multidimensional independent histograms for the image to determine one or more of;

a statistical mean value;

a statistical variance value;

a statistical skewness value; and

a statistical kurtosis value; and

using the classifying features in a boosting decision tree for image classification, comprising at least one of;

training a boosting decision tree for image classification using the classifying features; and

classifying the image using the classifying features in a trained boosting decision tree.

2. The method of claim 1, using a gradient magnitude-orientation histogram to determine one or more classifying features of the image comprising determining an entropy value of the gradient magnitude-orientation histogram.

3. The method of claim 1, generating a gradient magnitude-orientation histogram, comprising a 40×

8=320 gradient magnitude-orientation histogram.

4. The method of claim 1, the multidimensional independent histograms comprising independent histograms of one hundred dimensions for the respective red, green, and blue color bands.

5. The method of claim 1, using multidimensional independent histograms for respective red, green, and blue color bands to determine one or more classifying features of the image comprising:

using the multidimensional independent histograms for the image to determine a statistical discreteness value.

6. The method of claim 1, determining one or more classifying features of an image comprising:

using a joint multidimensional red-green-blue histogram of the image to determine one or more classifying features of the image comprising;

generating a joint multidimensional red-green-blue histogram of the image; and

determining an entropy value for the red-green-blue histogram of the image.

7. The method of claim 1, determining one or more classifying features of an image comprising:

using a multidimensional local binary pattern texture histogram of the image to determine one or more classifying features of the image comprising;

generating a multidimensional local binary pattern texture histogram of the image; and

determining an entropy value for the texture histogram of the image.

8. The method of claim 7, the multidimensional texture histogram comprising a fifty-nine dimensional texture histogram.

9. The method of claim 8, the fifty-nine dimensional texture histogram comprising:

respective bins for fifty-eight uniform local binary patterns; and

one bin for non-uniform local binary patterns.

10. The method of claim 1, the determining one or more classifying features of an image comprising:

using an edge detector on the image to determine one or more classifying features of the image.

11. The method of claim 10, the edge detector comprising a canny edge detector.

12. The method of claim 10, using an edge detector on an image to determine one or more classifying features of the image comprising:

determining a total number of edges in the image.

13. The method of claim 10, using an edge detector on an image to determine one or more classifying features of the image comprising:

determining an average length of edges in the image.

14. The method of claim 1, the determining one or more classifying features of an image comprising:

using a spatial correlogram of gray level pixels of the image to determine one or more classifying features of the image.

15. The method of claim 14, using a spatial correlogram of gray level pixels of an image to determine one or more classifying features of the image comprising determining an average skewness of respective slices of the spatial correlogram.

16. The method of claim 14, using a spatial correlogram of gray level pixels of an image to determine one or more classifying features of the image comprising:

determining an average variance ratio, the variance ratio comprising a ratio of variance of respective slices to radius of a symmetric range for the spatial correlogram.

17. A system configured to classify photo and graphic images using features generated from image statistics, the system comprising:

An image classifying feature determiner comprising;

a gradient magnitude orientation histogram generator configured to generate a gradient magnitude orientation histogram of an image;

a gradient magnitude orientation histogram energy value determiner, configured to measure a difference between an energy value of a lower frequency color band and an energy value of a higher frequency color band of the gradient magnitude orientation histogram;

a color histogram generator configured to generate independent histograms for respective red, green, and blue color bands of the image; and

a color histogram statistical values determiner configured to determine statistical values of a color histogram, the values comprising at least one of;

a statistical discreteness value;

a statistical mean value;

a statistical variance value;

a statistical skewness value; and

a statistical kurtosis value; and

a boosting decision tree trainer configured to train an image classification boosting decision tree for graphic and photo image classification.

18. The system of claim 17, the image classifying feature determiner comprising:

a gradient magnitude orientation histogram entropy value determiner, configured to determine an entropy value of the gradient magnitude orientation histogram of the image;

a joint red-green-blue histogram generator configured to generate a multidimensional joint red-green-blue histogram of the image;

a joint red-green-blue histogram entropy value determiner configured to determine an entropy value for the joint red-green-blue histogram of the image

a local binary pattern texture histogram generator configured to generate a local binary pattern texture histogram of the image;

a local binary pattern texture histogram entropy value determiner configured to determine an entropy value for the local binary pattern texture histogram of the image;

a canny edge detector configured to;

determine a total number of edges in the image; and

determine an average length of edges in the image;

a gray level pixel spatial correlogram generator configured to generate a special correlogram of gray level pixel for the image;

a gray level pixel spatial correlogram skewness determiner configured to determine an average skewness of respective slices of the spatial correlogram; and

a gray level pixel spatial correlogram variance determiner configured to determine an average variance ratio, the variance ratio comprising a ratio of variance of respective slices to radius of a symmetric range for the spatial correlogram.

19. The system of claim 17 comprising an image classifier configured to classify the image as either a photo or a graphic and attach corresponding image metadata, the image classifier comprising:

a trained image classification decision tree configured to classify the image as either a photo or a graphic; and

an image metadata attacher configured to attached metadata to the image corresponding to the image'"'"'s classification as either a photo or a graphic.

20. A method for classifying photo and graphic images using features generated from image statistics, the method comprising:

determining one or more classifying features of an image, comprising at least one of;

using a gradient magnitude-orientation histogram to determine one or more classifying features of the image, comprising;

generating a gradient magnitude-orientation histogram for the image;

measuring a difference between an energy value of a lower frequency color band and an energy value of a higher frequency color band using the gradient magnitude-orientation histogram; and

determining an entropy value of the gradient magnitude-orientation histogram;

using a joint multidimensional red-green-blue histogram of the image to determine one or more classifying features of the image comprising;

generating a joint multidimensional red-green-blue histogram of the image; and

determining an entropy value for the red-green-blue histogram of the image;

using multidimensional independent histograms for respective red, green, and blue color bands to determine one or more classifying features of the image, comprising;

generating respective multidimensional independent histograms corresponding to the red, green, and blue color bands for the image;

using the multidimensional independent histograms for the image to determine one or more of;

a statistical discreteness value

a statistical mean value;

a statistical variance value;

a statistical skewness value; and

a statistical kurtosis value;

using a multidimensional local binary pattern texture histogram of the image to determine one or more classifying features of the image comprising;

generating a multidimensional local binary pattern texture histogram of the image; and

determining an entropy value for the texture histogram of the image;

using a canny edge detector on the image to determine one or more classifying features of the image, comprising;

determining a total number of edges in the image; and

determining an average length of edges in the image;

using a spatial correlogram of gray level pixels of the image to determine one or more classifying features of the image, comprising;

determining an average skewness of respective slices of the spatial correlogram; and

determining an average variance ratio, the variance ratio comprising a ratio of variance of respective slices to radius of a symmetric range for the spatial correlogram; and

using the classifying features in a boosting decision tree for image classification, comprising at least one of;

training a boosting decision tree for image classification using the classifying features; and

classifying the image using the classifying features in a trained boosting decision tree.