Method and Apparatus to Incorporate Automatic Face Recognition in Digital Image Collections

• US 20090252383A1
• Filed: 04/01/2009
• Published: 10/08/2009
• Est. Priority Date: 04/02/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Steffens, Johannes, Horowitz, Michael, Westphal, Brian, Doughtie, Gavin, Magnusson, Tobias, Kiyohara, Keith, Ong, Hong-Kien Kenneth, Benjamin, Henry, Neven, Hartmut, ADAM, Hartwig
• Granted Patent: US 8,358,811 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method and apparatus for creating and updating a facial image database from a collection of digital images is disclosed. A set of detected faces from a digital image collection is stored in a facial image database, along with data pertaining to them. At least one facial recognition template for each face in the first set is computed, and the images in the set are grouped according to the facial recognition template into similarity groups. Another embodiment is a naming tool for assigning names to a plurality of faces detected in a digital image collection. A facial image database stores data pertaining to facial images detected in images of a digital image collection. In addition, the naming tool may include a graphical user interface, a face detection module that detects faces in images of the digital image collection and stores data pertaining to the detected faces in the facial image database, a face recognition module that computes at least one facial recognition template for each facial image in the facial image database, and a similarity grouping module that groups facial images in the facial image database according to the respective templates such that similar facial images belong to one similarity group.

Claims

1. A method for creating and updating a facial image database from a collection of digital images, comprising:

(a) detecting a first set of facial images in images from the collection of digital images;

(b) grouping the first set of facial images into similarity groups, wherein in each of the similarity groups facial recognition templates of facial images are within a predetermined range;

(c) displaying one or more of the similarity groups in a graphical user interface, wherein each of the similarity groups is substantially separately displayed;

(d) receiving user input to confirm or reject individual facial images in one or more of the displayed similarity groups; and

(e) updating one or more of the displayed similarity groups with the user input.

2. The method of claim 1, further comprising:

(f) storing the updated similarity groups in the facial images database.

3. The method of claim 2, further comprising:

(g) receiving a subject name for one or more of the displayed similarity groups; and

(h) updating one or more of the displayed similarity groups with the subject name.

4. The method of claim 3, wherein said stage (g) includes:

enabling the user to select a name for at least one of the displayed similarity groups from a list of entries, each entry of the list of entries having at least one of a name and a facial image, wherein said each entry corresponds to one or more facial images associated with a name, and wherein the list of entries is ordered according to similarity to the at least one of the displayed similarity groups.

5. The method of claim 2, wherein said stage (c) comprises:

(i) displaying representative facial images from each of the one or more similarity groups in a selectable filmstrip layout;

(ii) selecting a representative facial image based on user input; and

(iii) displaying facial images of at least one similarity group that correspond to the selected representative facial image.

6. A method of updating a facial image database from a collection of digital images, comprising:

(a) receiving a subject name from a user;

(b) retrieving a first set of facial images from at least one named similarity group of facial images stored in the facial images database, wherein a name associated with said at least one similarity group substantially matches the subject name;

(c) determining a first facial recognition template for the first set;

(d) retrieving a second set of facial images from one or more unnamed similarity groups in the facial images database, wherein the unnamed similarity groups have no associated name, and wherein facial recognition templates of the unnamed similarity groups substantially match the first facial recognition template;

(e) displaying the first set in a graphical user interface;

(f) displaying the second set in the graphical user interface;

(g) receiving user input associated with the second set; and

(h) updating the second set according to the user input.

7. The method of claim 6, wherein step (e) comprises:

(i) displaying the first set arranged in a cluster in the graphical user interface.

8. The method of claim 7, wherein step (f) comprises:

(i) displaying the second set in the graphical user interface with each similarity group of the second set arranged in a separate cluster.

9. The method of claim 6, wherein receiving user input further includes one or more of:

(i) rejecting one or more facial images based on the user input, wherein the rejected one or more facial images are not assigned the subject name;

(ii) rejecting one or more similarity groups of the second set based on the user input, wherein the rejected one or more similarity groups are not assigned said subject name; and

(iii) confirming one or more similarity groups of the second set based on the user input.

10. The method of claim 6, wherein the stage (h) further includes:

(i) assigning the subject name to facial images of similarity groups of the second set based on the user input.

11. The method of claim 6, wherein step (e) includes:

(i) displaying the first set within a first area of the graphical user interface.

12. The method of claim 11, wherein step (f) includes:

(i) displaying the second set in a second area of the graphical user interface, the second set ordered according to similarity to the first set;

13. A naming tool for assigning names to a plurality of faces detected in a digital image collection comprising:

(a) a facial image database configured to store data pertaining to facial images detected in images from the digital image collection;

(b) at least one graphical user interface;

(c) at least one user input device that enables manipulation of one or both of the data and the images, through the at least one graphical user interface;

(d) a face detection module configured to detect faces in images of the digital image collection and to store data pertaining to the detected faces in the facial image database;

(e) a face recognition module configured to compute at least one facial recognition template for each facial image in the facial image database;

(f) a similarity ranking module configured to group facial images in the facial image database according to respective facial recognition templates such that substantially similar facial images belong to one similarity group; and

(g) a display layout generation module configured to lay out the facial images according to similarity groups,

wherein the at least one user input device, the at least one graphical user interface, the digital image collection, and the facial image database are communicatively coupled to the face detection module, the face recognition module, the similarity ranking module, and the display layout generation module.

14. A method of accessing digital images from a collection of digital images, comprising:

(a) receiving a subject name;

(b) selecting a first set of facial images in a facial image database, wherein facial images in the first set are assigned the subject name; and

(c) finding digital images from the collection of digital images, wherein the digital images were previously associated with the first set of facial images.

15. The method in claim 14, further comprising:

(d) displaying the digital images; and

(e) displaying special effects on at least one face in one or more displayed digital images, using data stored in the facial image database.

16. The method of claim 14, further comprising:

(d) authenticating a user to access facial images assigned the subject name; and

(e) allowing the user to access images in the digital image collection that were previously associated with the first set.

17. A method of searching for images of a person utilizing multiple source images, comprising:

(a) receiving a subject name;

(b) selecting a first set of facial images in a facial image database that are assigned the subject name; and

(c) searching one or more digital image collections wherein search includes automatic face recognition using said first set of facial images.

18. The method in claim 17, further comprising:

(d) generating an alert when a matching image is found, wherein the matching image includes a facial image that substantially matches at least one facial image in the first set of facial images.

19. The method of claim 18, further comprising:

(e) associating a user with the subject name.

20. The method of claim 19, wherein the generating the alert includes:

(i) finding a matching image, wherein the matching image includes a facial image that substantially matches at least one facial image in the first set of facial images;

(ii) detecting at least one of the predetermined types of display events in a posted image, wherein the posted image includes a matching facial image that substantially matches the at least one facial image;

(iii) notifying the user about the at least one of the predetermined types of display events; and

(iv) receiving an input choice from the user.

21. The method of claim 20, further comprising:

(f) automatically deleting the posted image from one or more locations based on the input choice.

22. The method of claim 20, further comprising:

(f) modifying one or more tags assigned to the posted image based on the input choice.

23. The method of claim 20, further comprising:

(f) notifying one or more sources having the at least one image based on the input choice.

24. The method of claim 20, wherein finding a matching image includes:

(i) detecting a facial image in the at least one image;

(ii) determining the facial image to represent the subject name; and

(iii) assigning a tag associating one or more of the subject name and the user to the at least one image.