Method and apparatus for automatically annotating images

• US 20080021928A1
• Filed: 07/24/2006
• Published: 01/24/2008
• Est. Priority Date: 07/24/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Yagnik, Jay N.
• Granted Patent: US 8,065,313 B2
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

One embodiment of the present invention provides a system that automatically annotates an image. During operation, the system receives the image. Next, the system extracts image features from the image. The system then identifies other images which have similar image features. The system next obtains text associated with the other images, and identifies intersecting keywords in the obtained text. Finally, the system annotates the image with the intersecting keywords.

Claims

1. A method for automatically annotating an image, comprising:

receiving the image;

extracting image features from the image;

identifying other images which have similar image features;

obtaining text associated with the other images;

identifying intersecting keywords in the obtained text associated with the other images; and

annotating the image with the intersecting keywords.

2. The method of claim 1, wherein extracting the image features from the image involves:

partitioning the image into tiles; and

extracting the image features from the tiles.

3. The method of claim 2, wherein for each given tile size in a set of differing tile sizes, partitioning the image into tiles involves partitioning the image into tiles of the given tile size.

4. The method of claim 1, wherein prior to identifying the other images, the method further comprises matching the extracted image features against known image features in an image feature library.

5. The method of claim 4, wherein the method further comprises combining the matched image features to form one or more image-feature-combinations for the image.

6. The method of claim 5, wherein identifying the other images which have similar image features involves identifying similar image-feature-combinations in the other images.

7. The method of claim 1, wherein extracting the image features from the image can involve:

generating color histograms;

generating orientation histograms;

using a direct cosine transform (DCT) technique;

using a principal component analysis (PCA) technique;

or using a Gabor wavelet technique.

8. The method of claim 1, wherein the image features can be defined in terms of:

shapes;

colors; and

textures.

9. The method of claim 1, wherein identifying the other images involves searching through images on the Internet.

10. The method of claim 1, wherein identifying the other images having the similar image features involves using probability models.

11. The method of claim 1, wherein prior to identifying the intersecting keywords, the method further comprises expanding keywords in the obtained text by adding synonyms for the keywords.

12. The method of claim 1, wherein receiving the image involves receiving the image from a video.

13. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for automatically annotating an image, the method comprising:

receiving the image;

extracting image features from the image;

identifying other images which have similar image features;

obtaining text associated with the other images;

identifying intersecting keywords in the obtained text associated with the other images; and

annotating the image with the intersecting keywords.

14. The computer-readable storage medium of claim 13, wherein extracting the image features from the image involves:

partitioning the image into tiles; and

extracting the image features from the tiles.

15. The computer-readable storage medium of claim 14, wherein for each given tile size in a set of differing tile sizes, partitioning the image into tiles involves partitioning the image into tiles of the given tile size.

16. The computer-readable storage medium of claim 13, wherein prior to identifying the other images, the method further comprises matching the extracted image features against known image features in an image feature library.

17. The computer-readable storage medium of claim 16, wherein the method further comprises combining the matched image features to form one or more image-feature-combinations for the image.

18. The computer-readable storage medium of claim 17, wherein identifying the other images which have similar image features involves identifying similar image-feature-combinations in the other images.

19. The computer-readable storage medium of claim 13, wherein extracting the image features from the image can involve:

generating color histograms;

generating orientation histograms;

using a direct cosine transform (DCT) technique;

using a principal component analysis (PCA) technique;

or using a Gabor wavelet technique.

20. The computer-readable storage medium of claim 13, wherein the image features can be defined in terms of:

shapes;

colors; and

textures.

21. The computer-readable storage medium of claim 13, wherein identifying the other images involves searching through images on the Internet.

22. The computer-readable storage medium of claim 13, wherein identifying the other images having the similar image features involves using probability models.

23. The computer-readable storage medium of claim 13, wherein prior to identifying the intersecting keywords, the method further comprises expanding keywords in the obtained text by adding synonyms for the keywords.

24. The computer-readable storage medium of claim 13, wherein receiving the image involves receiving the image from a video.

25. A computer system that automatically annotates an image, comprising:

a processor;

a memory;

a receiving mechanism configured to receive the image;

an extraction mechanism configured to extract image features from the image;

an identification mechanism configured to identify other images which have similar image features;

an obtaining mechanism configured to obtain text associated with the other images;

wherein the identification mechanism is further configured to identify intersecting keywords in the obtained text associated with the other images; and

an annotation mechanism configured to annotate the image with the intersecting keywords.

26. The computer system of claim 25, wherein the extraction mechanism is configured to:

partition the image into tiles; and

to

extract the image features from the tiles.

27. The computer system of claim 25, further comprising a matching mechanism configured to match the extracted image features against known image features in an image feature library.

28. The computer system of claim 27, wherein the matching mechanism is configured to combine the matched image features to form one or more image-feature-combinations for the image.

29. The computer system of claim 28, wherein the identification mechanism is configured to identify similar image-feature-combinations in the other images.

30. The computer system of claim 25, wherein the identification mechanism is further configured to:

search through images on the Internet; and

to

expand keywords in the obtained text by adding synonyms for the keywords.

31. The computer system of claim 25, wherein the receiving mechanism is configured to receive the image from a video.