Interactive photo annotation based on face clustering

• US 8,189,880 B2
• Filed: 05/29/2007
• Issued: 05/29/2012
• Est. Priority Date: 05/29/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wen, Fang, Xiao, Rong, Tang, Xiaoou, Yang, Qiong
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US11/754,933
• Publication Number:
  US 20080298766A1
• Time in Patent Office: 1,827 Days
• Field of Search: 382/115, 382/118, 382/209, 382/218, 382224-225, 382/305
• US Class Current: 382/118
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5854 using shape and object relationship

Abstract

An interactive photo annotation method uses clustering based on facial similarities to improve annotation experience. The method uses a face recognition algorithm to extract facial features of a photo album and cluster the photos into multiple face groups based on facial similarity. The method annotates a face group collectively using annotations, such as name identifiers, in one operation. The method further allows merging and splitting of face groups. Special graphical user interfaces, such as displays in a group view area and a thumbnail area and drag-and-drop features, are used to further improve the annotation experience.

Claims

1. A computer-implemented method comprising:

extracting facial features of a collection of digital photos;

clustering the collection of digital photos into a plurality of face groups, at least first and second face groups of the plurality of face groups each including multiple digital photos, each photo having a facial feature satisfying a programmed facial similarity requirement;

associating an annotation with the at least first face group;

adding the second face group to the first face group; and

responsive to the adding, automatically associating the second face group with an annotation of the first face group.

2. The method as recited in claim 1, wherein the multiple digital photos each has a torso feature satisfying a programmed torso similarity requirement.

3. The method as recited in claim 1, wherein the multiple digital photos each has a photo time feature satisfying a programmed photo time similarity requirement.

4. The method as recited in claim 1, wherein the facial similarity requirement comprises a facial similarity level at or beyond which the facial features of the multiple digital photos are considered that of a person of same identity, and wherein the annotation comprises an identifier of the person.

5. The method as recited in claim 1, wherein clustering the collection of digital photos comprising:

automatically clustering the collection of digital photos by applying a face clustering algorithm.

6. The method as recited in claim 1, further comprising:

receiving from a user at least one ad hoc annotation for a plurality of digital photos in the collection of digital photos; and

clustering the collection of digital photos based on the at least one ad hoc annotation received from the user.

7. The method as recited in claim 1, further comprising:

modifying a face group of the plurality of face groups by removing a photo from the face group; and

disassociating the removed photo with the annotation of the face group.

8. The method as recited in claim 1, further comprising:

automatically merging face groups of the plurality of face groups if the face groups are associated with an identical annotation.

9. The method as recited in claim 1, further comprising:

displaying in a first view area an image representation of the at least one face group, the image representation being selectable by a user to select the at least one face group; and

displaying in a second view area thumbnails of the multiple digital photos of the at least one face group when selected by the user.

10. The method as recited in claim 1, further comprising:

allowing a user to select a target face group from the plurality of face groups;

ranking the plurality of face groups according to face similarity of each face group to the target face group; and

displaying the plurality of face groups according to a result of the ranking.

11. The method as recited in claim 1, further comprising:

allowing a user to select a target face group from the plurality of face groups;

allowing the user to select a target photo from the target face group;

ranking photos of the target face group according to face similarity of each photo to the target photo; and

displaying the photos of target face group according to a result of the ranking.

12. The method as recited in claim 1, further comprising

clustering the collection of digital photos into a plurality of event/location group;

and

associating an event/location annotation with the at least one event/location group.

13. A graphical user interface for adding annotations to a photo album, the graphical user interface comprising:

a group view area in which a plurality of face groups of the photo album are displayed, each displayed face group being selectable by a user, wherein the displayed face groups are enabled for a drag-and-drop operation to merge two or more face groups; and

an annotation input interface via which a user can enter an annotation for a selected face group.

14. The graphical user interface as recited in claim 13, further comprising a thumbnail area in which thumbnails of a plurality of photos of a selected face group are displayed.

15. The graphical user interface as recited in claim 14, wherein the displayed thumbnails are enabled for a drag-and-drop operation to remove a corresponding photo from the selected face group or to move a corresponding photo to another face group.

16. The graphical user interface as recited in claim 13, wherein the annotation input interface comprises:

an annotation field listing existing annotations enabled for selection by the user to be the annotation of the selected face group.

17. The graphical user interface as recited in claim 13, wherein the annotation input interface comprises:

an annotation field via which the user can manually enter the annotation for the selected face group.

18. One or more computer storage devices having stored thereupon a plurality of instructions that, when executed by a processor, perform operations including:

extracting facial features of a collection of digital photos;

clustering the collection of digital photos into a plurality of face groups, at least first and second face groups of the plurality of face groups each including multiple digital photos, each photo having a facial feature satisfying a facial similarity requirement;

associating an annotation with the at least first face group;

adding the second face group to the first face group; and

responsive to the adding, automatically associating the second face group with an annotation of the first face group.