ANNOTATING IMAGES

• US 20090304272A1
• Filed: 04/17/2009
• Published: 12/10/2009
• Est. Priority Date: 06/06/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kumar, Sanjiv, Makadia, Ameesh
• Granted Patent: US 8,422,832 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Methods, systems, and apparatus, including computer program products, for generating data for annotating images automatically. In one aspect, a method includes receiving an input image, identifying one or more nearest neighbor images of the input image from among a collection of images, in which each of the one or more nearest neighbor images is associated with a respective one or more image labels, assigning a plurality of image labels to the input image, in which the plurality of image labels are selected from the image labels associated with the one or more nearest neighbor images, and storing in a data repository the input image having the assigned plurality of image labels. In another aspect, a method includes assigning a single image label to the input image, in which the single image label is selected from labels associated with multiple ranked nearest neighbor images.

Claims

1. A system comprising:

means for determining in one or more computers that one or more reference images are nearest neighbor images of an input image, the one or more reference images collectively being associated with multiple image labels; and

means for assigning in the one or more computers two or more image labels to the input image, the two or more image labels being selected from among the multiple image labels associated with one or more reference images.

2. The system of claim 1, wherein each of the one or more reference images is associated with a plurality of image labels.

3. A method of image annotation performed by a data processing apparatus, the method comprising:

receiving an input image in the data processing apparatus;

identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the data processing apparatus, wherein each of the one or more nearest neighbor images is associated with a respective one or more image labels;

assigning a plurality of image labels to the input image, wherein the plurality of image labels are selected by the data processing apparatus from the image labels associated with the one or more nearest neighbor images; and

storing in a digital data repository the input image having the assigned plurality of image labels.

4. The method of claim 3, wherein the input image is stored in the digital data repository in an image file including the plurality of image labels as metadata.

5. The method of claim 3, wherein the plurality of image labels are selected from a single nearest neighbor image.

6. The method of claim 3, wherein assigning a plurality of image labels comprises:

sorting the multiple nearest neighbor images according to a respective whole-image distance associated with each nearest neighbor image, each whole-image distance representing a degree of difference between the input image and a corresponding nearest neighbor image;

ranking one or more first image labels according to a respective frequency of occurrence in the collection of digital images, wherein each of the one or more first image labels is associated with a first nearest neighbor; and

assigning at least one of the first image labels to the input image based on the ranking of the one or more first image labels.

7. The method of claim 6, wherein assigning the plurality of image labels to the input image further comprises:

ranking one or more second image labels, wherein each of the second image labels is associated with one or more remaining nearest neighbor image; and

assigning at least one of the second image labels to the input image based on the ranking of the one or more second image labels.

8. The method of claim 7, wherein ranking of the one or more second image labels comprises sorting the one or more second image labels according to a co-occurrence of each of the second image labels with each first image label in the collection of digital images.

9. The method of claim 7, wherein ranking of the one or more second image labels comprises sorting the one or more second image labels according to a local frequency of each of the second image labels in the one or more remaining nearest neighbor images.

10. The method of claim 6, wherein the whole-image distance comprises a combination of feature distances, each feature distance representing a degree of difference between an image feature associated with the input image and a respective image feature associated with the reference image.

11. The method of claim 10, wherein the image feature associated with the input image and the respective image feature associated with the reference image each comprise a global image feature.

12. The method of claim 10, wherein the image feature associated with the input image and the respective image feature associated with the reference image each comprise a local image feature.

13. The method of claim 10, further comprising:

deriving the whole-image distance as an average of the feature distances.

14. The method of claim 13, wherein the average is based on an equal contribution from each of the feature distances.

15. The method of claim 13, wherein the average is based on a weighted contribution from each of the feature distances.

16. The method of claim 15, further comprising:

calculating a weighting for each of the feature distances based on the collection of digital images, wherein the collection of digital images is-a group of training images comprising pairs of similar and dissimilar images.

17. The method of claim 10, further comprising:

calculating at least one of the feature distances as a difference between a texture feature of the input image and a corresponding texture feature of the reference image.

18. The method of claim 10, further comprising:

calculating at least one of the feature distances as a difference between a color feature of the input image and a corresponding color feature of the reference image.

19. A system comprising:

a server implemented on one or more computers and operable to perform operations comprising;

receiving an input image in the server;

identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the server, wherein each of the one or more nearest neighbor images is associated with a respective one or more image labels;

assigning a plurality of image labels to the input image, wherein the plurality of image labels are selected by the server from the image labels associated with the one or more nearest neighbor images; and

storing in a digital data repository the input image having the assigned plurality of image labels.

20. The system of claim 19, wherein the plurality of image labels are selected from a single nearest neighbor image.

21. The system of claim 19, wherein assigning a plurality of image labels comprises:

sorting the multiple nearest neighbor images according to a respective whole-image distance associated with each nearest neighbor image, each whole-image distance representing a degree of difference between the input image and a corresponding nearest neighbor image;

ranking one or more first image labels according to a respective frequency of occurrence in the collection of digital images, wherein each of the one or more first image labels is associated with a first nearest neighbor; and

assigning at least one of the first image labels to the input image based on the ranking of the one or more first image labels.

22. The system of claim 21, wherein assigning the plurality of image labels to the input image further comprises:

ranking one or more second image labels, wherein each of the second image labels is associated with one or more remaining nearest neighbor image; and

assigning at least one of the second image labels to the input image based on the ranking of the one or more second image labels.

23. The system of claim 22, wherein ranking of the one or more second image labels comprises sorting the one or more second image labels according to a co-occurrence of each of the second image labels with each first image label in the collection of digital images.

24. The system of claim 22, wherein ranking of the one or more second image labels comprises sorting the one or more second image labels according to a local frequency of each of the second image labels in the one or more remaining nearest neighbor images.

25. The system of claim 21, wherein the whole-image distance comprises a combination of feature distances, each feature distance representing a degree of difference between an image feature associated with the input image and a respective image feature associated with the reference image.

26. The system of claim 25, wherein the image feature associated with the input image and the respective image feature associated with the reference image each comprise a global image feature.

27. The system of claim 25, wherein the image feature associated with the input image and the respective image feature associated with the reference image each comprise a local image feature.

28. The system of claim 25, wherein the server is operable to perform operations further comprising:

deriving the whole-image distance as an average of the feature distances.

29. The system of claim 28, wherein the average is based on an equal contribution from each of the feature distances.

30. The system of claim 28, wherein the average is based on a weighted contribution from each of the feature distances.

31. The system of claim 30, wherein the server is operable to perform operations further comprising:

calculating a weighting for each of the feature distances based on the collection of digital images, wherein the collection of digital images is a group of training images comprising pairs of similar and dissimilar images.

32. The system of claim 25, wherein the server is operable to perform operations further comprising:

calculating at least one of the feature distances as a difference between a texture feature of the input image and a corresponding texture feature of the reference image.

33. The system of claim 25, wherein the server is operable to perform operations further comprising:

calculating at least one of the feature distances as a difference between a color feature of the input image and a corresponding color feature of the reference image.

34. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving an input image in the data processing apparatus;

identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the data processing apparatus, wherein each of the one or more nearest neighbor images is associated with a respective one or more image labels;

assigning a plurality of image labels to the input image, wherein the plurality of image labels are selected by the data processing apparatus from the image labels associated with the one or more nearest neighbor images; and

storing in a digital data repository the input image having the assigned plurality of image labels.

35. The computer storage medium of claim 34, wherein the plurality of image labels are selected from a single nearest neighbor image.

36. The computer storage medium of claim 34, wherein assigning a plurality of image labels comprises:

sorting the multiple nearest neighbor images according to a respective whole-image distance associated with each nearest neighbor image, each whole-image distance representing a degree of difference between the input image and a corresponding nearest neighbor image;

ranking one or more first image labels according to a respective frequency of occurrence in the collection of digital images, wherein each of the one or more first image labels is associated with a first nearest neighbor; and

assigning at least one of the first image labels to the input image based on the ranking of the one or more first image labels.

37. The computer storage medium of claim 36, wherein assigning the plurality of image labels to the input image further comprises:

ranking one or more second image labels, wherein each of the second image labels is associated with one or more remaining nearest neighbor image; and

assigning at least one of the second image labels to the input image based on the ranking of the one or more second image labels.

38. The computer storage medium of claim 37, wherein ranking of the one or more second image labels comprises sorting the one or more second image labels according to a co-occurrence of each of the second image labels with each first image label in the collection of digital images.

39. The computer storage medium of claim 37, wherein ranking of the one or more second image labels comprises sorting the one or more second image labels according to a local frequency of each of the second image labels in the one or more remaining nearest neighbor images.

40. The computer storage medium of claim 36, wherein the whole-image distance comprises a combination of feature distances, each feature distance representing a degree of difference between an image feature associated with the input image and a respective image feature associated with the reference image.

41. The computer storage medium of claim 40, wherein the image feature associated with tile input image and the respective image feature associated with the reference image each comprise a global image feature.

42. The computer storage medium of claim 40, wherein the image feature associated with the input image and the respective image feature associated with the reference image each comprise a local image feature.

43. The computer storage medium of claim 40, operable to cause data processing apparatus to perform operations further comprising:

deriving the whole-image distance as an average of the feature distances.

44. The computer storage medium of claim 43, wherein the average is based on an equal contribution from each of the feature distances.

45. The computer storage medium of claim 43, wherein the average is based on a weighted contribution from each of the feature distances.

46. The computer storage medium of claim 45, operable to cause data processing apparatus to perform operations further comprising:

calculating a weighting for each of the feature distances based on the collection of digital images, wherein the collection of digital images is a group of training images comprising pairs of similar and dissimilar images.

47. The computer storage medium of claim 40, operable to cause data processing apparatus to perform operations further comprising:

calculating at least one of the feature distances as a difference between a texture feature of the input image and a corresponding texture feature of the reference image.

48. The computer storage medium of claim 40, operable to cause data processing apparatus to perform operations further comprising:

calculating at least one of the feature distances as a difference between a color feature of the input image and a corresponding color feature of the reference image.