AUTOMATIC CLASSIFICATION OF OBJECTS WITHIN IMAGES

• US 20080037877A1
• Filed: 08/14/2006
• Published: 02/14/2008
• Est. Priority Date: 08/14/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Hua, Ma, Wei-Ying, Chen, Zheng, Xie, Xing, Jia, Menglei
• Granted Patent: US 7,813,561 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

A system for automatically classifying an object of a target image is provided. A classification system provides a collection of classified images along with a classification of the dominant object of the image. The classification system attempts to classify the object of a target image based on similarity of the target image to the classified images. To classify a target image, the classification system identifies the classified images of the collection that are most similar to the target image based on similarity between salient points of the target image and the classified images. The classification system selects a classification associated with the classified images that are most similar to the target image as a classification for the object of the target image.

Claims

1. A method in a computing device for classifying an object of a target image, the method comprising:

providing classified images and an associated classification for each classified image, the associated classification indicating a classification of an object represented by the classified image;

identifying classified images that are most similar to the target image; and

selecting a classification for the object of the target image based on the classification of the identified classified image.

2. The method of claim 1 including:

identifying salient points of the classified images; and

generating a mapping from similar salient points to the classified images that contain those salient points

wherein the identifying of the classified images uses the generated mapping to identify classified images that contain salient points that are similar to salient points of the target image.

3. The method of claim 2 wherein each salient point is identified using a difference of Gaussian technique and represented by a feature vector.

4. The method of claim 2 wherein the mapping is generated using a clustering technique.

5. The method of claim 4 wherein the clustering technique is based on a growing cell structures algorithm.

6. The method of claim 1 wherein the identifying of classified images that are most similar to the target image includes:

identifying classified images as matching images based on similar salient points; and

calculating a similarity between the target image and the matching images based on the similar salient points of the target image and the matching images.

7. The method of claim 6 wherein the similarity is calculated according to the following equation:

$\mathrm{Sim}<br><br>\left(I,J\right)=\underset{i,j}{\sum <br><br>}<br><br>\mathrm{corr}<br><br>\left(X_i,Y_j\right)$

8. The method of claim 6 including filtering out a matching image when an arrangement between its matching salient points and an arrangement of the corresponding similar salient points of the target image are inconsistent.

9. The method of claim 6 wherein the filtering out includes applying a RANSAC-based algorithm.

10. The method of claim 1 including:

identifying salient points of the classified images; and

generating a mapping from similar salient points to the classified images that contain those salient points

wherein the identifying of the classified images uses the generated mapping to identify classified images that contain salient points that are similar to salient points of the target image, and

wherein the identifying of the classified images identifies the classified images as matching images based on similar salient points and calculates similarities between the target image and the matching images based on similar salient points of the target image and the matching images.

11. A computer system for classifying an object of a target image, comprising:

a classified images store containing classified images of objects and classifications of the objects;

a component that generates an index that maps feature vectors of salient points of the classified images to the classified images;

a component that generates feature vectors for salient points of the target image;

a component that identifies, from the generated index, classified images that match the target image based on similarity between feature vectors of salient points of the target image and the classified images; and

a component that classifies the object of the target image based on the classifications of the matching classified images.

12. The computer system of claim 11 wherein the component that generates an index includes a component to identify salient points of the classified images and to generate feature vectors for the salient points and a component to generate clusters of similar feature vectors.

13. The computer system of claim 12 wherein the salient points are identified using a difference of Gaussian algorithm.

14. The computer system of claim 12 wherein the clusters are identified using a growing cell structures algorithm.

15. The computer system of claim 11 wherein the component that identifies classified images that match includes a component that discards matching images whose similarity is not within a threshold similarity.

16. The computer system of claim 15 wherein the component that identifies classified images that match includes a component that filters out a matching image when an arrangement between its matching salient points and an arrangement of the corresponding salient points of the target image are inconsistent.

17. The computer system of claim 11 wherein the component that classifies the object of the target image selects the classification of a matching classified image based on number of similar salient points.

18. A computer-readable medium containing instructions for controlling a computing device to classify an object of a target image, by a method comprising:

providing an index that maps feature vectors for salient points of classified images to the classified images and their classifications;

generating feature vectors for salient points of the target image;

identifying, from the provided index, classified images that match the target image based on similarity between feature vectors of salient points of the target image and the classified images; and

selecting a classification for the object of the target image based on the classifications of the matching classified images.

19. The computer-readable medium of claim 18 wherein the identifying of classified images that match includes calculating a similarity between the target image and the matching classified images based on similar salient points of the target image and the matching classified images.

20. The computer-readable medium of claim 19 wherein the selecting of a classification includes selecting a dominant classification of the matching classified images.