Method and apparatus to incorporate automatic face recognition in digital image collections

• US 8,358,811 B2
• Filed: 04/01/2009
• Issued: 01/22/2013
• Est. Priority Date: 04/02/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Steffens, Johannes, Kiyohara, Keith, Neven, Hartmut, Westphal, Brian, Magnusson, Tobias, Doughtie, Gavin, Benjamin, Henry, Horowitz, Michael, Ong, Hong-Kien Kenneth, Adams, Hartwig
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US12/416,632
• Publication Number:
  US 20090252383A1
• Time in Patent Office: 1,392 Days
• Field of Search: 382115-118, 382/220, 382/224, 382/305, 382/312, 382/173, 382/225, 713/182, 713/186, 707 5- 6, 707/9, 707/705, 707/737
• US Class Current: 382/118
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method and apparatus for creating and updating a facial image database from a collection of digital images is disclosed. A set of detected faces from a digital image collection is stored in a facial image database, along with data pertaining to them. At least one facial recognition template for each face in the first set is computed, and the images in the set are grouped according to the facial recognition template into similarity groups. Another embodiment is a naming tool for assigning names to a plurality of faces detected in a digital image collection. A facial image database stores data pertaining to facial images detected in images of a digital image collection. In addition, the naming tool may include a graphical user interface, a face detection module that detects faces in images of the digital image collection and stores data pertaining to the detected faces in the facial image database, a face recognition module that computes at least one facial recognition template for each facial image in the facial image database, and a similarity grouping module that groups facial images in the facial image database according to the respective templates such that similar facial images belong to one similarity group.

Claims

1. A method, comprising:

detecting a first set of facial images in images from a collection of digital images;

grouping the first set of facial images into similarity groups, wherein in each of the similarity groups facial recognition templates of facial images are within a predetermined range;

displaying facial images from a plurality of the similarity groups in a graphical user interface, wherein each of the similarity groups is substantially separately displayed;

receiving user input to confirm or reject individual facial images in one or more of the displayed similarity groups;

updating one or more of the displayed similarity groups based upon the user input; and

storing the changed similarity groups in a facial images database.

2. The method of claim 1, further comprising:

receiving a subject name for one or more of the displayed similarity groups; and

updating one or more of the displayed similarity groups with the subject name.

3. The method of claim 2, wherein the receiving a subject name includes:

enabling the user to select a name for at least one of the displayed similarity groups from a list of entries, each entry of the list of entries having at least one of a name and a facial image, wherein said each entry corresponds to one or more facial images associated with a name, and wherein the list of entries is ordered according to similarity to the at least one of the displayed similarity groups.

4. The method of claim 1, wherein displaying one or more of the similarity groups, comprises:

displaying representative facial images from each of the plurality of similarity groups in a selectable filmstrip layout;

selecting a representative facial image from the displayed representative facial images based on user input; and

displaying facial images of at least one similarity group that correspond to the selected representative facial image.

5. A method of updating a facial image database from a collection of digital images, comprising:

receiving a subject name from a user;

retrieving a first set of facial images from at least one named similarity group of facial images stored in the facial images database, wherein a name associated with said at least one similarity group substantially matches the subject name;

determining a first facial recognition template for the first set;

retrieving a second set of facial images from one or more unnamed similarity groups in the facial images database, wherein the unnamed similarity groups have no associated name, and wherein facial recognition templates of the unnamed similarity groups substantially match the first facial recognition template;

displaying the first set in a graphical user interface;

displaying the second set in the graphical user interface;

receiving user input associated with the second set; and

updating the second set according to the user input.

6. The method of claim 5, wherein displaying the first set comprises:

displaying the first set arranged in a cluster in the graphical user interface.

7. The method of claim 6, wherein displaying the second set comprises:

displaying the second set in the graphical user interface with each similarity group of the second set arranged in a separate cluster.

8. The method of claim 5, wherein receiving user input further includes one or more of:

rejecting one or more facial images based on the user input, wherein the rejected one or more facial images are not assigned the subject name;

rejecting one or more similarity groups of the second set based on the user input, wherein the rejected one or more similarity groups are not assigned said subject name; and

confirming one or more similarity groups of the second set based on the user input.

9. The method of claim 5, wherein the updating the second set further includes:

assigning the subject name to facial images of similarity groups of the second set based on the user input.

10. The method of claim 5, wherein displaying the first set includes:

displaying the first set within a first area of the graphical user interface.

11. The method of claim 10, wherein displaying the second set includes:

displaying the second set in a second area of the graphical user interface, the second set ordered according to similarity to the first set.

12. A system, comprising:

a facial image database configured to store data pertaining to facial images detected in images from the digital image collection;

at least one graphical user interface;

at least one user input device that enables manipulation of one or both of the data and the images, through the at least one graphical user interface;

a face detection module configured to detect faces in images of the digital image collection and to store data pertaining to the detected faces in the facial image database;

a face recognition module configured to compute at least one facial recognition template for each facial image in the facial image database;

a similarity ranking module configured to group facial images in the facial image database according to respective facial recognition templates such that substantially similar facial images belong to one similarity group; and

a display layout generation module configured to lay out the facial images to display a plurality of similarity groups,

wherein the at least one user input device, the at least one graphical user interface, the digital image collection, and the facial image database are communicatively coupled to the face detection module, the face recognition module, the similarity ranking module, and the display layout generation module.

13. A method of accessing digital images from a collection of digital images, comprising:

receiving a subject name;

selecting a first set of facial images in a facial image database, wherein facial images in the first set are assigned the subject name; and

finding digital images from the collection of digital images, wherein one or more similarity groups including the digital images were previously associated with a similarity group including the first set of facial images, wherein the one or more similarity groups do not have an associated subject name, and wherein facial recognition templates of the one or more similarity groups substantially match a facial recognition template of the first set.

14. The method in claim 13, further comprising:

displaying the digital images; and

displaying special effects on at least one face in one or more displayed digital images, using data stored in the facial image database.

15. The method of claim 13, further comprising:

authenticating a user to access facial images assigned the subject name; and

allowing the user to access images in the digital image collection that were previously associated with the first set.

16. A method, comprising:

receiving a subject name from a user;

retrieving a first set of facial images from at least one named similarity group of facial images stored in the facial images database, wherein a name associated with said at least one similarity group substantially matches the subject name;

determining a first facial recognition template for the first set; and

retrieving a second set of facial images from one or more unnamed similarity groups in the facial images database, wherein the unnamed similarity groups have no associated name, and wherein facial recognition templates of the unnamed similarity groups substantially match the first facial recognition template.

17. The method of claim 16 comprising:

searching one or more digital image collections wherein the searching includes automatic face recognition using at least one of said first or said second set of facial images.

18. The method in claim 17, further comprising:

generating an alert when a matching image is found, wherein the matching image includes a facial image that substantially matches at least one facial image in at least one of the first or the second set of facial images.

19. The method of claim 18, further comprising:

associating a user with the subject name.

20. The method of claim 19, wherein the generating the alert includes:

finding a matching image, wherein the matching image includes a facial image that substantially matches at least one facial image in at least one of the first set or the second set of facial images;

detecting at least one of the predetermined types of display events in a posted image, wherein the posted image includes a matching facial image that substantially matches the at least one facial image;

notifying the user about the at least one of the predetermined types of display events; and

receiving an input choice from the user.

21. The method of claim 20, further comprising:

automatically deleting the posted image from one or more locations based on the input choice.

22. The method of claim 20, further comprising:

modifying one or more tags assigned to the posted image based on the input choice.

23. The method of claim 20, further comprising:

notifying one or more sources having the at least one image based on the input choice.

24. The method of claim 20, wherein finding a matching image includes:

detecting a facial image in the at least one image;

determining the facial image to represent the subject name; and

assigning a tag associating one or more of the subject name and the user to the at least one image.