Identifying images using face recognition

• US 9,053,357 B2
• Filed: 12/13/2011
• Issued: 06/09/2015
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Yagnik, Jay
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US13/324,058
• Publication Number:
  US 20120093375A1
• Time in Patent Office: 1,274 Days
• Field of Search: 382/118, 382/181
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method includes identifying a named entity, retrieving images associated with the named entity, and using a face detection algorithm to perform face detection on the retrieved images to detect faces in the retrieved images. At least one representative face image from the retrieved images is identified, and the representative face image is used to identify one or more additional images representing the at least one named entity.

Claims

1. A method, comprising:

identifying at least one named entity;

retrieving a set of three or more images associated with the at least one named entity, each of the images in the set including a textual description of the image derived from metadata specific to only that image, wherein the textual description of the image includes at least a portion of a name of the named entity;

performing face detection on the retrieved images to detect faces in the retrieved images, wherein the face detection is performed using a face detection algorithm;

identifying a representative face image from the retrieved images by comparing each detected face to each of the other detected faces to identify faces that match the representative face image associated with the at least one named entity;

determining a number of faces that match the representative face image;

comparing the determined number of faces that match the representative face image to a threshold value;

determining that the determined number of faces that match the representative face image satisfies the threshold value;

in response to determining that the determined number of faces that match the representative face image satisfies the threshold value, building a face model using the representative face image including identifying a training set that includes the representative face image and all other images that match the representative face image from the set of three or more images; and

using the face model to identify one or more additional images representing the at least one named entity from images not included in the set.

2. The method of claim 1 wherein identifying the representative face image includes identifying the representative face image as the image that contains the detected face that matches a highest number of the other of the detected faces.

3. The method of claim 1, further comprising receiving a search query from a user including at least another portion of the name of the at least one named entity.

4. The method of claim 3, further comprising generating search results based on the search query, wherein the search results include the one or more additional images.

5. The method of claim 1, wherein the at least one named entity is a person.

6. The method of claim 5, wherein identifying the at least one named entity comprises identifying the name of the at least one named entity from a query log or a web document.

7. The method of claim 1, wherein retrieving images associated with the at least one named entity comprises executing an image search to identify all images associated with the at least one named entity.

8. The method of claim 2, further comprising determining the detected face in the retrieved images that matches a highest number of other detected faces in the retrieved images.

9. The method of claim 1, further comprising labeling the face model with at least a portion of the name of the at least one named entity.

10. The method of claim 1, wherein using the face model to identify the one or more additional images comprises comparing the face model to a collection of images to identify the one or more additional images.

11. The method of claim 10, further comprising associating the one or more additional images with the at least one named entity.

12. The method of claim 1 wherein building a face model further includes constructing a composite built from a sum of the faces in the training set.

13. The method of claim 1 where the face model is a complex model.

14. The method of claim 13 where the complex model is built using oriented PCA or Kernal OPCA.

15. A method comprising

identifying a named entity that is eligible for generation of a face model, the face model being used to identify un-tagged images of the named entity;

collecting a set of images of the named entity including collecting one or more images from the Web, each of the images including a textual description of the image derived from metadata specific to only that image, wherein the textual description of the image includes at least a portion of a name of the named entity;

performing face detection on each image in the set of images to detect faces in the images, wherein the face detection is performed using a face detection algorithm;

identifying a representative face image from the set of images by comparing each detected face to each of the other detected faces to identify faces that match the representative face image associated with the named entity;

determining a number of faces that match the representative face image;

comparing the determined number of faces that match the representative face image to a threshold value;

determining that the determined number of faces that match the representative face image satisfies the threshold value;

in response to determining that the determined number of faces that match the representative face image satisfies the threshold value, building a face model using the representative face image including identifying a training set that includes the representative face image and all other images that match the representative face image from the set of images; and

using the face model to identify one or more additional images representing the named entity from images not included in the set of images.

16. The method of claim 15 wherein identifying the representative face image includes identifying the representative face image as the image that contains the detected face that matches a highest number of the other of the detected faces.

17. The method of claim 15, further comprising receiving a search query from a user including the name of the named entity.

18. The method of claim 17, further comprising generating search results based on the search query, wherein the search results include the one or more additional images.

19. The method of claim 15, wherein the named entity is a person.

20. The method of claim 19, wherein identifying the named entity comprises identifying the named entity from a query log or a web document.

21. The method of claim 15, wherein retrieving images associated with the named entity comprises executing an image search to identify all images associated with the named entity.

22. The method of claim 16, further comprising determining the detected face in the set of images that matches a highest number of other detected faces in the set of images.

23. The method of claim 15, further comprising labeling the face model with at least a portion of the name of the named entity.

24. The method of claim 15, wherein using the face model to identify the one or more additional images comprises comparing the face model to a collection of images to identify the one or more additional images.

25. The method of claim 24, further comprising associating the one or more additional images with the named entity.

26. The method of claim 15, wherein building a face model further includes constructing a composite built from a sum of the faces in the training set.

27. The method of claim 15, where the face model is a complex model.

28. The method of claim 27 where the complex model is built using oriented PCA or Kernal OPCA.

29. The method of claim 15, where eligibility is measured by one or more criteria and where the criteria are selected from a group comprising popularity, ranking, search frequency or minimum number of occurrences in web documents.

30. A computer program product comprising instructions that when executed by one or more processors cause the processors to:

identify at least one named entity;

retrieve a set of three or more images associated with the at least one named entity, each of the images in the set including a textual description of the image derived from metadata specific to only that image, wherein the textual description of the image includes at least a portion of a name of the named entity;

perform face detection on the retrieved images to detect faces in the retrieved images, wherein the face detection is performed using a face detection algorithm;

identify a representative face image from the retrieved images by comparing each detected face to each of the other detected faces to identify faces that match the representative face image associated with the at least one named entity;

determine a number of faces that match the representative face image;

compare the determined number of faces that match the representative face image to a threshold value;

determine that the determined number of faces that match the representative face image satisfies the threshold value;

in response to determining that the determined number of faces that match the representative face image satisfies the threshold value, build a face model using the representative face image including identifying a training set that includes the representative face image and all other images that match the representative face image from the set of three or more images; and

use the face model to identify one or more additional images representing the at least one named entity from images not included in the set.

32. The computer program product of claim 30, wherein instructions cause the processors to receive a search query from a user including at least another portion of the name of the at least one named entity.

33. The computer program product of claim 32, wherein instructions cause the processors to generate search results based on the search query, wherein the search results include the one or more additional images.

34. The computer program product of claim 30, wherein the at least one named entity is a person.

35. The computer program product of claim 34, wherein the instructions to identify the at least one named entity comprise instructions to identify the name of the at least one named entity from a query log or a web document.

36. The computer program product of claim 30, wherein the instruction to retrieve images associated with the at least one named entity comprise instructions to execute an image search to identify all images associated with the at least one named entity.

37. The computer program product of claim 30, wherein the instructions to build a face model comprise instructions to construct a composite built from a sum of the faces in the training set.

31. A computer program product comprising instructions that when executed by one or more processors cause the processors to:

identify a named entity that is eligible for generation of a face model, the face model being used to identify un-tagged images of the named entity;

collect a set of images of the named entity including collecting one or more images from the Web, each of the images including a textual description of the image derived from metadata specific to only that image, wherein the textual description of the image includes at least a portion of a name of the named entity;

perform face detection on each image in the set of images to detect faces in the images, wherein the face detection is performed using a face detection algorithm;

identify a representative face image from the set of images by comparing each detected face to each of the other detected faces to identify faces that match the representative face image associated with the named entity;

determine a number of faces that match the representative face image;

compare the determined number of faces that match the representative face image to a threshold value;

determine that the determined number of faces that match the representative face image satisfies the threshold value;

in response to determining that the determined number of faces that match the representative face image satisfies the threshold value, build a face model using the representative face image including identifying a training set that includes the representative face image and all other images that match the representative face image from the set of images; and

use the face model to identify one or more additional images representing the named entity from images not included in the set of images.