Facilitating computer-assisted tagging of object instances in digital images

• US 7,889,946 B1
• Filed: 08/04/2009
• Issued: 02/15/2011
• Est. Priority Date: 02/28/2005
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Bourdev, Lubomir B.
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US12/535,426
• Time in Patent Office: 560 Days
• Field of Search: 382/107, 382/118, 382/132, 382/181, 382/275, 382/305, 382/312, 707/3, 707/6, 707/100, 396/429, 348/231.1, 235/487
• US Class Current: 382/291
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06V 10/987 with the intervention of an operator
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

One embodiment of the present invention provides a system that facilitates computer-assisted tagging of objects in a digital image. During operation, the system receives locations for one or more objects-of-interest in the digital image. Next, the system determines likelihoods of specific tags being assigned to the objects-of-interest. The system then automatically assigns tentative tags to the objects-of-interest based on the determined likelihoods. Next, the system displays the assignments of tentative tags to a user, and receives corrections to the assignments, if any, from the user.

Claims

1. A method, comprising:

receiving a collection of one or more digital images;

automatically extracting one or more faces of people from within the one or more digital images;

detecting that one or more of the extracted one or more faces is untagged;

automatically assigning one or more tentative tags for one or more of the one or more untagged faces based on similarities between one or more regions containing the one or more untagged faces and one or more regions containing one or more faces that have already been tagged;

displaying the assignment of the one or more tentative tags to a user; and

receiving correction or confirmation for the displayed tentative tag assignments from the user.

2. The method of claim 1, further comprising:

grouping the one or more tentatively-tagged faces into bins that correspond to the one or more automatically-assigned tentative tags; and

displaying one or more tentatively-tagged faces grouped in each of the bins;

wherein said receiving correction or confirmation comprises receiving user input to select and drag one or more of the one or more tentatively-tagged faces from one bin to another.

3. The method of claim 2, further comprising ordering a plurality of tentatively-tagged faces in one of the bins based on system confidence that each one of the plurality of tentatively-tagged faces grouped into that bin belongs in that bin.

4. The method of claim 2, further comprising normalizing to the same size the one or more tentatively-tagged faces grouped into the bins.

5. The method of claim 2, further comprising, concurrently with said displaying one or more tentatively-tagged faces grouped in each of the bins, separately displaying one of the one or more digital images in its entirety that includes a particular one of the displayed one or more tentatively-tagged faces.

6. The method of claim 2, further comprising ceasing to display the displayed one or more tentatively-tagged faces whose tentative tag assignments have been corrected or confirmed by the user.

7. The method of claim 1, further comprising determining a likelihood that that a specific combination of people will occur together in a same digital image, wherein said automatically assigning the one or more tentative tags is further based on the determined likelihood.

8. The method of claim 1, wherein the similarities comprise a pose or a scale of one or more of the one or more tentatively-tagged faces, or attire or physical features of a person.

9. The method of claim 1, wherein said automatically assigning the one or more tentative tags is further based on a previous tag assignment by the user.

10. The method of claim 1, wherein said displaying the assignment of the one or more tentative tags comprises displaying the one or more tentative tags in proximity to the corresponding one or more tentatively-tagged faces.

11. The method of claim 1, wherein said displaying the assignment of the one or more tentative tags comprises displaying a group comprising each face corresponding to one of the one more tentative tags.

12. The method of claim 1, wherein said displaying the assignment of the one or more tentative tags comprises displaying a set of most likely tentative tags for one or more of the one or more tentatively-tagged faces.

13. A non-transitory computer-readable storage device storing instructions that when executed by a computer cause the computer to perform:

receiving a collection of one or more digital images;

automatically extracting one or more faces of people from within the one or more digital images;

detecting that one or more of the extracted one or more faces is untagged;

automatically assigning one or more tentative tags for one or more of the one or more untagged faces based on similarities between one or more regions containing the one or more untagged faces and one or more regions containing one or more faces that have already been tagged;

displaying the assignment of the one or more tentative tags to a user; and

receiving correction or confirmation for the displayed tentative tag assignments from the user.

14. The non-transitory computer-readable storage device of claim 13, wherein, when executed by the computer, the stored instructions cause the computer to further perform:

grouping the one or more tentatively-tagged faces into bins that correspond to the one or more automatically-assigned tentative tags; and

displaying one or more tentatively-tagged faces grouped in each of the bins;

wherein said receiving correction or confirmation comprises receiving user input to select and drag one or more of the one or more tentatively-tagged faces from one bin to another.

15. The non-transitory computer-readable storage device of claim 14, wherein, when executed by the computer, the stored instructions cause the computer to further perform ordering a plurality of tentatively-tagged faces in one of the bins based on system confidence that each one of the plurality of tentatively-tagged faces grouped into that bin belongs in that bin.

16. The non-transitory computer-readable storage device of claim 14, wherein, when executed by the computer, the stored instructions cause the computer to further perform normalizing to the same size the one or more tentatively-tagged faces grouped into the bins.

17. The non-transitory computer-readable storage device of claim 14, wherein, when executed by the computer, the stored instructions cause the computer to further perform, concurrently with said displaying one or more tentatively-tagged faces grouped in each of the bins, separately displaying one of the one or more digital images in its entirety that includes a particular one of the displayed one or more tentatively-tagged faces.

18. The non-transitory computer-readable storage device of claim 14, wherein, when executed by the computer, the stored instructions cause the computer to further perform ceasing to display the displayed one or more tentatively-tagged faces whose tentative tag assignments have been corrected or confirmed by the user.

19. The non-transitory computer-readable storage device of claim 13, wherein, when executed by the computer, the stored instructions cause the computer to further perform determining a likelihood that that a specific combination of people will occur together in a same digital image, wherein said automatically assigning the one or more tentative tags is further based on the determined likelihood.

20. The non-transitory computer-readable storage device of claim 13, wherein the similarities comprise a pose or a scale of one or more of the one or more tentatively-tagged faces, or attire or physical features of a person.

21. The non-transitory computer-readable storage device of claim 13, wherein said automatically assigning the one or more tentative tags is further based on a previous tag assignment by the user.

22. The non-transitory computer-readable storage device of claim 13, wherein said displaying the assignment of the one or more tentative tags comprises displaying the one or more tentative tags in proximity to the corresponding one or more tentatively-tagged faces.

23. A system, comprising:

one or more processors; and

a computer-readable storage medium storing program instructions executable by the one or more processors to perform;

receiving a collection of one or more digital images;

automatically extracting one or more faces of people from within the one or more digital images;

detecting that one or more of the extracted one or more faces is untagged;

automatically assigning one or more tentative tags for one or more of the one or more untagged faces based on similarities between one or more regions containing the one or more untagged faces and one or more regions containing one or more faces that have already been tagged;

displaying the assignment of the one or more tentative tags to a user; and

receiving correction or confirmation for the displayed tentative tag assignments from the user.

24. The system of claim 23, wherein the stored program instructions are executable by the one or more processors to further perform:

grouping the one or more tentatively-tagged faces into bins that correspond to the one or more automatically-assigned tentative tags; and

displaying one or more tentatively-tagged faces grouped in each of the bins;

wherein said receiving correction or confirmation comprises receiving user input to select and drag one or more of the one or more tentatively-tagged faces from one bin to another.

25. The system of claim 24, wherein the stored program instructions are executable by the one or more processors to further perform ordering a plurality of tentatively-tagged faces in one of the bins based on system confidence that each one of the plurality of tentatively-tagged faces grouped into that bin belongs in that bin.

26. The system of claim 24, wherein the stored program instructions are executable by the one or more processors to further perform normalizing to the same size the one or more tentatively-tagged faces grouped into the bins.

27. The system of claim 24, wherein the stored program instructions are executable by the one or more processors to further perform, concurrently with said displaying one or more tentatively-tagged faces grouped in each of the bins, separately displaying one of the one or more digital images in its entirety that includes a particular one of the displayed one or more tentatively-tagged faces.

28. The system of claim 24, wherein the stored program instructions are executable by the one or more processors to further perform ceasing to display the displayed one or more tentatively-tagged faces whose tentative tag assignments have been corrected or confirmed by the user.

29. The system of claim 23, wherein the similarities comprise a pose or a scale of one or more of the one or more tentatively-tagged faces, or attire or physical features of a person.

30. The system of claim 23, wherein said automatically assigning the one or more tentative tags is further based on a previous tag assignment by the user.

31. The system of claim 23, wherein said displaying the assignment of the one or more tentative tags comprises displaying a group comprising each face corresponding to one of the one more tentative tags.