Computer-Implemented Method for Performing Similarity Searches

• US 20080080745A1
• Filed: 07/12/2007
• Published: 04/03/2008
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Tengli, Ashwin, Anguelov, Dragomir, Vanhoucke, Vincent, Lee, Kuang-chih, Gokturk, Salih
• Granted Patent: US 7,760,917 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

A similarity search may be performed on the image of a person, using visual characteristics and information that is known about the person. The search identifies images of other persons that are similar in appearance to the person in the image.

Claims

1. A computer-implemented method for determining face similarity, the method comprising:

identifying a person from an image;

identifying at least one of text or metadata associated with the image;

performing a similarity search using the person of the image, in order to identify images of other persons that are similar in appearance to the person in the image;

wherein performing the similarity search includes using at least some of the text and metadata to determine one or more classifications for the person in the image.

2. The method of claim 1, wherein identifying at least one of text or metadata associated with the image includes determining biographical information about the person in the image using the text and metadata.

3. The method of claim 1, wherein performing a similarity search using the person of the image includes determining one or more of the classifications corresponding to an ethnicity, a gender or an age classification of the person in the image, using the text and metadata.

4. The method of claim 3, wherein determining one or more of the classifications further comprises using image data from the image of the person to determine one or more of the classifications corresponding to the ethnicity, the gender or the age classification.

5. The method of claim 1, wherein identifying a person from an image includes detecting a face of the person from the image, including one or more features of the face.

6. The method of claim 1, wherein performing the similarity search includes formulating a search query to include image data that represents one or more visual characteristics of a face of the person.

8. The method of claim 1, wherein identifying at least one of text or metadata associated with the image includes identifying a text contained in a file that includes the image of the person.

9. The method of claim 1, wherein identifying at least one of text or metadata associated with the image includes identifying a label or a tag provided with the image.

10. The method of claim 1, wherein identifying at least of text or metadata associated with the image includes identifying a text contained in the image.

7. The method of 1, wherein performing a similarity search using the person of the image includes (i) determining one or more of the classifications corresponding to an ethnicity, a gender or an age classification of the person in the image, using the text and metadata;

and (ii) determining a result by selecting one or more images in a collection that has the same determined classifications.

11. A system for determining face similarity, the system comprising:

an analysis module that is configured to identify (i) a face from an image, and (ii) at least one of text or metadata associated with the face of the image;

a characteristic determination module that is configured to identify one or more biographical classifications of the face based at least in part on the at least one of the text or metadata associated with the face of the face;

a comparison module that compares a query image input of the face with a collection of images of other faces in order to identify one or more faces in the collection that are determined as being similar to the face of the query image input, wherein the query image input comprises image data representing the face as well as the one or more biographical classifications determined about the face using at least the text and metadata.

12. The system of claim 11, wherein the one or more biographical determinations include one or more of a gender classification, an ethnicity classification, and an age classification.

13. The system of claim 12, wherein the characteristic determination module is configured to determine any of the one or more biographical classifications using a combination of the text and metadata and any image data representing the face of the image.

14. The system of claim 11, wherein the comparison module is configured to generate a result of images that includes faces of persons that have all of the one or more biographical classifications.

15. The system of claim 11, further comprising a system that includes an optimization module to optimize one or more operations for determining one or more faces of the images in the collection that are determined to be similar to the face of the query image input.

16. The system of claim 15, wherein the optimization module is configured to determine whether a given image in a record is suitable for the query image input.

17. The system of claim 15, wherein the optimization module is configured to use an identity of either a person of the query image input or of a person in one of the images of the collection in order to determine the one or more faces of the images in the collection that are determined to be similar to the face of the query image input.

18. The system of claim 11, wherein the characteristic determination module identifies at least one of text or metadata associated with the image by identifying a text contained in a file that includes the image of the person.

19. The system of claim 11, wherein the characteristic determination module identifies at least one of text or metadata associated with the image by identifying a label or a tag provided with the image.

20. A computer-readable medium for determining face similarity, the computer-readable medium comprising instructions, that when executed by one or more processors, cause the one or more processors to perform steps comprising:

identifying a person from an image;

identifying at least one of text or metadata associated with the image;

performing a similarity search using the person of the image, in order to identify images of other persons that are similar in appearance to the person in the image;

wherein performing the similarity search includes using at least some of the text and metadata to determine one or more classifications for the person in the image.