Method and apparatus to incorporate automatic face recognition in digital image collections

• US 8,897,508 B2
• Filed: 12/17/2012
• Issued: 11/25/2014
• Est. Priority Date: 04/02/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Adam, Hartwig, Steffens, Johannes, Kiyohara, Keith, Neven, Harmut, Westphal, Brian, Magnusson, Tobias, Doughtie, Gavin, Benjamin, Henry, Horowitz, Michael, Ong, Hong-Kien Kenneth
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US13/717,426
• Publication Number:
  US 20130251217A1
• Time in Patent Office: 708 Days
• Field of Search: 382115-118, 382/173, 382/220, 382/224, 382/225, 382/305, 382/312, 713/182, 713/186, 707 5- 6, 707/9, 707/705, 707736-737
• US Class Current: 382/118
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method and apparatus for creating and updating a facial image database from a collection of digital images is disclosed. A set of detected faces from a digital image collection is stored in a facial image database, along with data pertaining to them. At least one facial recognition template for each face in the first set is computed, and the images in the set are grouped according to the facial recognition template into similarity groups. Another embodiment is a naming tool for assigning names to a plurality of faces detected in a digital image collection. A facial image database stores data pertaining to facial images detected in images of a digital image collection. In addition, the naming tool may include a graphical user interface, a face detection module that detects faces in images of the digital image collection and stores data pertaining to the detected faces in the facial image database, a face recognition module that computes at least one facial recognition template for each facial image in the facial image database, and a similarity grouping module that groups facial images in the facial image database according to the respective templates such that similar facial images belong to one similarity group.

Claims

1. A method for creating and updating a facial image database from a collection of digital images, comprising:

detecting a set of facial images in images from the collection of digital images;

detecting a gender of an individual in each image in the set of facial images;

grouping the set of facial images into similarity groups based at least in part on the detected genders, wherein facial recognition templates of facial images in each of the similarity groups are within a predetermined range;

displaying one or more of the similarity groups in a graphical user interface, wherein each of the similarity groups is substantially separately displayed;

receiving user input to confirm or reject individual facial images in one or more of the displayed similarity groups; and

updating the one or more of the displayed similarity groups with the user input in the facial image database.

2. A system comprising a memory and at least one processor, the at least one processor configured to:

detect a set of facial images in images from a collection of digital images;

detecting a gender of an individual in each image in the set of facial images;

group the set of facial images into similarity groups based at least in part on the detected genders, wherein facial recognition templates of facial images in each of the similarity groups are within a predetermined range;

display facial images from a plurality of the similarity groups in a graphical user interface, wherein each of the similarity groups is substantially separately displayed;

receive user input to confirm or reject individual facial images in one or more of the displayed similarity groups;

update the one or more of the displayed similarity groups based upon the user input; and

store the changed similarity groups in a facial images database.

3. A tangible computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:

detecting a set of facial images in images from a collection of digital images;

detecting a gender of an individual in each image in the set of facial images;

grouping the set of facial images into similarity groups based at least in part on the detected genders, wherein facial recognition templates of facial images in each of the similarity groups are within a predetermined range;

displaying facial images from a plurality of the similarity groups in a graphical user interface, wherein each of the similarity groups is substantially separately displayed;

receiving user input to confirm or reject individual facial images in one or more of the displayed similarity groups;

updating the one or more of the displayed similarity groups based upon the user input; and

storing the changed similarity groups in a facial images database.

4. A system comprising a memory and at least one processor, the at least one processor configured to:

detect a first set of facial images in images from a collection of digital images;

group the first set of facial images into similarity groups, wherein facial recognition templates including an aggregate value for a set of landmark facial recognition values associated with facial images in each of the similarity groups are determined to be within a predetermined range based on measuring a Euclidean distance between the facial recognition templates of facial images;

display facial images from a plurality of the similarity groups in a graphical user interface,

wherein each of the similarity groups is substantially separately displayed;

receive user input to confirm or reject individual facial images in one or more of the displayed similarity groups;

update one or more of the displayed similarity groups based upon the user input; and

store the changed similarity groups in a facial images database.

5. The system of claim 4, wherein the at least one processor is further configured to:

receive a subject name for one or more of the displayed similarity groups; and

update one or more of the displayed similarity groups with the subject name.

6. The system of claim 5, wherein to receive a subject name, the at least one processor is configured to:

enable the user to select a name for at least one of the displayed similarity groups from a list of entries, each entry of the list of entries having at least one of a name and a facial image, wherein said each entry corresponds to one or more facial images associated with a name, and wherein the list of entries is ordered according to similarity to the at least one of the displayed similarity groups.

7. The system of claim 4, wherein to display one or more of the similarity groups, the at least one processor is configured to:

display representative facial images from each of the plurality of similarity groups in a selectable filmstrip layout;

select a representative facial image from the displayed representative facial images based on user input; and

display facial images of at least one similarity group that correspond to the selected representative facial image.

8. A system comprising a memory and at least one processor, the at least one processor configured to:

receive a subject name from a user;

retrieve a first set of facial images from at least one named similarity group of facial images stored in the facial images database, wherein a name associated with said at least one similarity group substantially matches the subject name;

determine a first facial recognition template for the first set;

retrieve a second set of facial images from one or more unnamed similarity groups in the facial images database, wherein the unnamed similarity groups have no associated name, and wherein facial recognition templates of the unnamed similarity groups substantially match the first facial recognition template;

display the first set in a graphical user interface;

display the second set in the graphical user interface;

receive user input associated with the second set; and

update the second set according to the user input.

9. The system of claim 8, wherein to display the first set, the at least one processor is configured to:

display the first set arranged in a cluster in the graphical user interface.

10. The system of claim 9, wherein to display the second set, the at least one processor is configured to:

display the second set in the graphical user interface with each similarity group of the second set arranged in a separate cluster.

11. The system of claim 8, wherein to receive user input further, the at least one processor is configured to perform one of:

reject one or more facial images based on the user input, wherein the rejected one or more facial images are not assigned the subject name;

reject one or more similarity groups of the second set based on the user input, wherein the rejected one or more similarity groups are not assigned said subject name; and

confirm one or more similarity groups of the second set based on the user input.

12. The system of claim 8, wherein to update the second set, the at least one processor is configured to:

assign the subject name to facial images of similarity groups of the second set based on the user input.

13. The system of claim 8, wherein to display the first set, the at least one processor is configured to:

display the first set within a first area of the graphical user interface.

14. The system of claim 8, wherein to display the second set, the at least one processor is configured to:

display the second set in a second area of the graphical user interface, the second set ordered according to similarity to the first set.

15. A system comprising a memory and at least one processor, the at least one processor configured to:

receive a subject name;

select a first set of facial images in a facial image database, wherein facial images in the first set are assigned the subject name; and

find digital images from the collection of digital images, wherein one or more similarity groups including the digital images were previously associated with a similarity group including the first set of facial images, wherein the one or more similarity groups do not have an associated subject name, and wherein facial recognition templates of the one or more similarity groups substantially match a facial recognition template of the first set.

16. The system of claim 15, wherein the at least one processor is further configured to:

display the digital images; and

display special effects on at least one face in one or more displayed digital images, using data stored in the facial image database.

17. The system of claim 15, wherein the at least one processor is further configured to:

authenticate a user to access facial images assigned the subject name; and

allow the user to access images in the digital image collection that were previously associated with the first set.

18. A system comprising a memory and at least one processor, the at least one processor configured to:

receive a subject name from a user;

retrieve a first set of facial images from at least one named similarity group of facial images stored in the facial images database, wherein a name associated with said at least one similarity group substantially matches the subject name;

determine a first facial recognition template for the first set; and

retrieve a second set of facial images from one or more unnamed similarity groups in the facial images database, wherein the unnamed similarity groups have no associated name, and wherein facial recognition templates of the unnamed similarity groups substantially match the first facial recognition template.

19. The system of claim 18, wherein the at least one processor is further configured to:

search one or more digital image collections wherein the searching includes automatic face recognition using at least one of said first or said second set of facial images.

20. The system claim 18, wherein the at least one processor is further configured to:

generate an alert when a matching image is found, wherein the matching image includes a facial image that substantially matches at least one facial image in at least one of the first or the second set of facial images.

21. The system of claim 18, wherein the at least one processor is further configured to:

associate a user with the subject name.

22. The system of claim 20, wherein to generate the alert, the at least one processor is configured to:

find a matching image, wherein the matching image includes a facial image that substantially matches at least one facial image in at least one of the first set or the second set of facial images;

detect at least one of the predetermined types of display events in a posted image, wherein the posted image includes a matching facial image that substantially matches the at least one facial image;

notify the user about the at least one of the predetermined types of display events; and

receive an input choice from the user.

23. The system of claim 22, wherein the at least one processor is further configured to:

automatically delete the posted image from one or more locations based on the input choice.

24. The system of claim 22, wherein the at least one processor is further configured to:

modify one or more tags assigned to the posted image based on the input choice.

25. The system of claim 22, wherein the at least one processor is further configured to:

notify one or more sources having the at least one image based on the input choice.

26. The system of claim 22, wherein to find a matching image, the at least one processor is configured to:

detect a facial image in the at least one image;

determine the facial image to represent the subject name; and

assign a tag associating one or more of the subject name and the user to the at least one image.

27. A computer-readable storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:

detecting a first set of facial images in images from a collection of digital images;

grouping the first set of facial images into similarity groups, wherein facial recognition templates including an aggregate value for a set of landmark facial recognition values associated with facial images in each of the similarity groups are determined to be within a predetermined range based on measuring a Euclidean distance between the facial recognition templates of facial images;

displaying facial images from a plurality of the similarity groups in a graphical user interface, wherein each of the similarity groups is substantially separately displayed;

receiving user input to confirm or reject individual facial images in one or more of the displayed similarity groups;

updating one or more of the displayed similarity groups based upon the user input; and

storing the changed similarity groups in a facial images database.

28. A computer-readable storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:

receiving a subject name from a user;

retrieving a first set of facial images from at least one named similarity group of facial images stored in the facial images database, wherein a name associated with said at least one similarity group substantially matches the subject name;

determining a first facial recognition template for the first set;

retrieving a second set of facial images from one or more unnamed similarity groups in the facial images database, wherein the unnamed similarity groups have no associated name, and wherein facial recognition templates of the unnamed similarity groups substantially match the first facial recognition template;

displaying the first set in a graphical user interface;

displaying the second set in the graphical user interface;

receiving user input associated with the second set; and

updating the second set according to the user input.

29. A computer-readable storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:

receiving a subject name;

selecting a first set of facial images in a facial image database, wherein facial images in the first set are assigned the subject name; and

finding digital images from the collection of digital images, wherein one or more similarity groups including the digital images were previously associated with a similarity group including the first set of facial images, wherein the one or more similarity groups do not have an associated subject name, and wherein facial recognition templates of the one or more similarity groups substantially match a facial recognition template of the first set.

30. A computer-readable storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:

receiving a subject name from a user;

retrieving a first set of facial images from at least one named similarity group of facial images stored in the facial images database, wherein a name associated with said at least one similarity group substantially matches the subject name;

determining a first facial recognition template for the first set; and

retrieving a second set of facial images from one or more unnamed similarity groups in the facial images database, wherein the unnamed similarity groups have no associated name, and wherein facial recognition templates of the unnamed similarity groups substantially match the first facial recognition template.