System and method for labeling a collection of images

• US 8,724,908 B2
• Filed: 11/30/2010
• Issued: 05/13/2014
• Est. Priority Date: 09/03/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Dale, Kevin T., Bourdev, Lubomir D., Avidan, Shmuel, Parenteau, Alexandre S.
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): LE, TOTAM HA
• Application Number:
  US12/956,467
• Publication Number:
  US 20130129231A1
• Time in Patent Office: 1,260 Days
• Field of Search: 382/224
• US Class Current: 382/224
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06V 10/7788 the supervisor being a human, e.g. interactive learning with a human teacher

Abstract

Various embodiments of a system and methods for labeling images are described. An image labeling system may receive multiple images. A subset of the images may be labeled to identify image content or elements, such as faces. The system may display some of the labeled image elements in different portions of a display area. Unlabeled image elements may be displayed in the same display area. The display size and position of each unlabeled image element may be dependent on similarities between the unlabeled image element and the displayed, labeled image elements. The system may receive input which indicates that one or more of the displayed, unlabeled image elements should receive the same label as one of the displayed, labeled image elements. Similarities between image elements may be re-analyzed, dependent on the assigned labels, and the display of unlabeled images may be updated.

Claims

1. A method comprising:

displaying, in a display, two or more labeled image elements corresponding to one or more images, each of the two or more labeled image elements comprising a different label and each of the two or more labeled image elements being displayed in a different region of the display;

displaying, in the display, one or more unlabeled image elements at a distance relative to at least one of the displayed labeled image elements, the distance being dependent on a determined similarity between a respective unlabeled image element and a respective labeled image element;

receiving a selection of at least one displayed unlabeled image element;

responsive to receiving the selection of the at least one displayed unlabeled image element, automatically selecting a labeled image element that is most similar to the selected unlabeled image element; and

responsive to receiving user input, assigning a label of the automatically selected labeled image element to the selected unlabeled image element.

2. The method of claim 1,

wherein the determined similarity is determined by calculating a similarity metric between the one or more unlabeled image elements and at least one of the two or more labeled image elements.

3. The method of claim 2, further comprising determining a display size for each displayed unlabeled image element based on the similarity metric for the respective unlabeled image element relative to the respective labeled image element.

4. The method of claim 1, further comprising:

responsive to assigning the label, automatically updating the display to include an additional unlabeled image element.

5. The method of claim 4, wherein automatically updating the display comprises:

removing, from the display, the selected unlabeled image element to which the label is assigned; and

displaying, in the display, the additional unlabeled image element at a distance relative to at least one of the displayed labeled image elements.

6. The method of claim 1, wherein the two or more labeled image elements are faces detected in the corresponding one or more images.

12. The method of claim 1, the assigning comprising:

associating the selected unlabeled image element with the automatically selected labeled image element; and

causing the label associated with the automatically selected labeled image element to be assigned to the selected unlabeled image element.

13. The method of claim 1, wherein the label assigned to the selected unlabeled image element is independent from a label of an image that includes the selected unlabeled image element.

14. The method of claim 1, wherein the determined similarity between the respective unlabeled image element and the respective labeled image element indicates a probability that a subject of the respective unlabeled image element is similar to a subject of the respective labeled image element.

7. A computer-readable storage device storing program instructions that, responsive to execution, direct a computer to perform operations comprising:

locating a plurality of image elements in the collection of images;

displaying, in a display area, two or more labeled image elements from the plurality of image elements, each of the two or more labeled image elements comprising a different label and each of the two or more labeled image elements being displayed in a different region of the display area;

displaying, in the display area, one or more unlabeled image elements from the plurality of image elements at a distance relative to at least one of the displayed labeled image elements, the distance being dependent on similarities between a respective unlabeled image element and a respective labeled image element;

receiving a selection of at least one displayed unlabeled image element;

responsive to receiving the selection of the at least one displayed unlabeled image element, automatically selecting a labeled image element that is most similar to the selected unlabeled image element; and

responsive to receiving user input, assigning a label of one of the automatically selected labeled image elements to the selected unlabeled image element.

15. The computer-readable storage device of claim 7, further comprising including the label assigned to the unlabeled image element as metadata for an image corresponding to the unlabeled image element.

16. The computer-readable storage device of claim 7, wherein two or more unlabeled image elements are included in a single image.

17. The computer-readable storage device of claim 7, wherein the label assigned to the selected unlabeled image element includes coordinate information specifying a location of the selected unlabeled image element in an image associated with the selected unlabeled image element.

18. The computer-readable storage device of claim 7, the operations further comprising displaying, in the display area, a source image that includes the selected unlabeled image element.

19. The computer-readable storage device of claim 7, further comprising receiving an entry to assign a new label via a text entry box associated with the selected unlabeled image element.

20. The computer-readable storage device of claim 7, wherein the selection of the at least one displayed unlabeled image elements comprises selecting at least two displayed unlabeled image elements.

8. A system, comprising:

a memory; and

one or more processors coupled to the memory, the memory storing program instructions executable by the one or more processors to implement an image labeling module configured to;

receive a plurality of image elements extracted from a collection of images, the plurality of image elements including an unlabeled image element and two or more labeled image elements;

automatically select a subset of the two or more labeled image elements for display;

display, in a different region of a display area, each of the selected labeled image elements, each displayed labeled image element including a different label;

determine a similarity between the unlabeled image element and each of the displayed labeled image elements;

display, in the display area, the unlabeled image element at a distance relative to one of the displayed labeled image elements based, at least in part, on the determined similarity; and

receive a request to assign a label of one of the displayed labeled image elements to the unlabeled image element.

9. The system of claim 8, wherein determining the similarity includes calculating a similarity metric between the unlabeled image element and at least one of the displayed two or more labeled image elements.

10. The system of claim 9, wherein the image labeling module is further configured to determine a display size for the unlabeled image element, the display size being dependent on the determined similarity between the unlabeled image element and at least one of the displayed two or more labeled image elements.

11. The system of claim 8, wherein the image labeling module is further configured to:

assign, dependent on the received request, the label to the unlabeled image element;

recalculate, dependent on the assigned label, similarity between an additional unlabeled image element and the displayed two or more labeled image elements; and

display, in the display area, the additional unlabeled image element based on the recalculated similarity.