Statistical Approach to Large-scale Image Annotation

• US 20090297050A1
• Filed: 05/30/2008
• Published: 12/03/2009
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Xiaoguang, Li, Mingjing
• Granted Patent: US 8,150,170 B2
• Time in Patent Office: Days
• US Class Current: 382/225
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

Statistical approaches to large-scale image annotation are described. Generally, the annotation technique includes compiling visual features and textual information from a number of images, hashing the images visual features, and clustering the images based on their hash values. An example system builds statistical language models from the clustered images and annotates the image by applying one of the statistical language models.

Claims

1. A method of annotating an image comprising:

compiling visual features and textual information from a plurality of images;

hashing the plurality of visual features, and clustering the plurality of images based on the hash value;

building one or more statistical language models based on the clustered images; and

annotating the image using one or more of the statistical language models.

2. A method of annotating an image as recited in claim 1, wherein the plurality of images are gathered by crawling one or more large-scale image databases.

3. A method of annotating an image as recited in claim 1, wherein hashing the plurality of visual features comprises a vector quantization process in which the visual features are transformed into a binary string.

4. A method of annotating an image as recited in claim 1, wherein the images with the same hash code are grouped into clusters.

5. A method of annotating an image as recited in claim 1, wherein the one or more statistical language models is a unigram model.

6. A method of annotating an image as recited in claim 1, wherein the one or more statistical language models is a bigram model.

7. A method of annotating an image as recited in claim 1, wherein the image is a personal image, and the image is annotated by selecting words with a maximum joint probability between the image and the clustered images.

8. A method of annotating an image as recited in claim 1, wherein the image is a web image, and the image is annotated by a two step probabilistic modeling technique.

9. A method of annotating an image as recited in claim 1, further comprising extracting visual information from the plurality of images by using a gray block methodology.

10. A method of annotating an image as recited in claim 9, wherein the gray block methodology comprises:

partitioning the image into equal size blocks,

measuring an average luminescence for each block, and

representing the image as a vector.

11. A method of annotating an image as recited in claim 9, further comprising reducing the visual information of the plurality of images by employing a projection matrix.

12. A computer readable storage medium comprising computer executable instructions that when executed by a processor perform a method comprising:

crawling a large-scale image database to gather a plurality of images;

compiling visual features and textual information from the plurality of images;

extracting visual information from the plurality of images by using a gray block methodology;

reducing the visual information by employing a projection matrix;

hashing the reduced visual information, and clustering the plurality of images based on a hash value;

building one or more statistical language models based on the clustered images; and

annotating a query image using one or more of the statistical language models.

13. A computer readable storage medium as recited in claim 12, wherein hashing the reduced visual information comprises a vector quantization process in which the visual features are transformed into a binary string.

14. A computer readable storage medium as recited in claim 12, wherein the images with the same hash code are grouped into clusters.

15. A computer readable storage medium as recited in claim 12, wherein the one or more statistical language models is a unigram model.

16. A computer readable storage medium as recited in claim 12, wherein the one or more statistical language models is a bigram model.

17. A computer readable storage medium as recited in claim 12, wherein the query image is a personal image which is annotated by selecting words with the maximum joint probability with the query image.

18. A computer readable storage medium as recited in claim 12, wherein the query image is a web image which is annotated by two step probabilistic modeling technique.

19. A data structure embodied on a computer readable media to represent an item in an item catalog, the data structure comprising:

a digital image; and

a textual annotation corresponding to the digital image, the textual annotation being associated with the digital image by;

compiling visual features and textual information from a plurality of images;

hashing the plurality of visual features, and clustering the plurality of images based on the hash value;

building one or more statistical language models based on the clustered images; and

annotating the image using one or more of the statistical language models.

20. A data structure embodied on a computer readable media to represent an item in an item catalog as recited in claim 19, wherein the plurality of images are gathered by crawling one or more large-scale image databases.