ANNOTATING IMAGES

• US 20130198601A1
• Filed: 03/13/2013
• Published: 08/01/2013
• Est. Priority Date: 06/06/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Makadia, Ameesh, Kumar, Sanjiv
• Granted Patent: US 9,298,682 B2
• Time in Patent Office: Days
• US Class Current: 715/231
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Methods, systems, and apparatus, including computer program products, for generating data for annotating images automatically. In one aspect, a method includes receiving an input image, identifying one or more nearest neighbor images of the input image from among a collection of images, in which each of the one or more nearest neighbor images is associated with a respective one or more image labels, assigning a plurality of image labels to the input image, in which the plurality of image labels are selected from the image labels associated with the one or more nearest neighbor images, and storing in a data repository the input image having the assigned plurality of image labels. In another aspect, a method includes assigning a single image label to the input image, in which the single image label is selected from labels associated with multiple ranked nearest neighbor images.

Claims

1. A method of image annotation performed by a data processing apparatus, the method comprising:

receiving an input image in the data processing apparatus;

identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the data processing apparatus, each of the one or more nearest neighbor images being associated with a respective one or more image labels, wherein the collection of digital images comprises a plurality of reference images, and wherein identifying the one or more nearest neighbor images comprises;

calculating, for each reference image, a whole-image distance between the input image and the reference image as

2. The method of claim 1, wherein at least one image feature extracted from both the input image and the reference image is a global image feature.

3. The method of claim 1, wherein at least one image feature extracted from both the input image and the reference image is a local image feature.

4. The method of claim 1, wherein each feature distance is weighted equally.

5. The method of claim 1, wherein two or more feature distances are weighted differently from one another.

6. The method of claim 1, further comprising estimating a weighting vector w that minimizes the expression

7. The method of claim 6, wherein y_l is +1 when the l^th training image pair is a pair of similar training images and y_l is −

1 when the l^th training image pair is a pair of dissimilar training images.

8. A system comprising:

a server implemented on one or more computers and operable to perform operations comprising;

receiving an input image in the server;

identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the server, each of the one or more nearest neighbor images being associated with a respective one or more image labels, wherein the collection of digital images comprises a plurality of reference images, and wherein identifying the one or more nearest neighbor images comprises;

calculating, for each reference image, a whole-image distance between the input image and the reference image as

9. The system of claim 8, wherein at least one image feature extracted from both the input image and the reference image is a global image feature.

10. The system of claim 8, wherein at least one image feature extracted from both the input image and the reference image is a local image feature.

11. The system of claim 8, wherein each feature distance is weighted equally.

12. The system of claim 8, wherein two or more feature distances are weighted differently from one another.

13. The system of claim 8, wherein the server is operable to perform operations further comprising estimating a weighting vector ω

that minimizes the expression

14. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving an input image in the data processing apparatus;

identifying one or more nearest neighbor images of the input image from among a collection of digital images stored on computer-readable media by operation of the data processing apparatus, each of the one or more nearest neighbor images being associated with a respective one or more image labels, wherein the collection of digital images comprises a plurality of reference images, and wherein identifying the one or more nearest neighbor images comprises;

calculating, for each reference image, a whole-image distance between the input image and the reference image as

15. The computer storage medium of claim 14, wherein at least one image feature extracted from both the input image and the reference image is a global image feature.

16. The computer storage medium of claim 14, wherein at least one image feature extracted from both the input image and the reference image is a local image feature.

17. The computer storage medium of claim 14, wherein each feature distance is weighted equally.

18. The computer storage medium of claim 14, wherein two or more feature distances are weighted differently from one another.

19. The computer storage medium of claim 14, operable to cause the data processing apparatus to perform operations further comprising estimating a weighting vector ω

that minimizes the expression

20. The computer storage medium of claim 19, wherein y_l is +1 when the l^th training image pair is a pair of similar training images and y_l is −

1 when the l^th training image pair is a pair of dissimilar training images.