Searching images and identifying images with similar facial features

• US 9,235,782 B1
• Filed: 12/24/2012
• Issued: 01/12/2016
• Est. Priority Date: 12/24/2012
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zomet, Asaf, Chechik, Gal
• Primary Examiner(s): Ahmed, Samir
• Application Number:
  US13/726,390
• Time in Patent Office: 1,114 Days
• Field of Search: 382/159, 382/181, 382/190, 382/201
• US Class Current: 1/1
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A system and method for searching images and identifying images with similar facial features is disclosed. In one implementation, the system includes a pre-processing module, a feature extraction module, a model creation module, a similarity identifier module and a results display module. The pre-processing module receives a facial image, determines key-points associated with a facial feature and identifies a facial area including the facial feature. The feature extraction module extracts the key-points. The model creation module creates a similarity model for determining similar facial features at least in part by comparing the facial feature from a plurality of images. The similarity identifier module applies the similarity model to the facial feature an image in relation to the facial feature in other images and determines which other image has a most similar facial feature. The results display module presents a result based at least in part on the determination.

Claims

1. A method for searching images and identifying images with similar facial features, the method comprising:

receiving a facial image, the facial image including a query image;

applying a transformation to the facial image based on a set of key points associated with the facial image, the transformation of the facial image conforms to a first canonical depiction associated with an orientation of the facial area;

identifying at least one facial area associated with the transformed facial image based on the set of key-points, the facial area including at least one facial feature;

generating a similarity model for the at least one facial feature associated with the facial area of the facial image, the similarity model including determining similarity between the at least one facial feature associated with the facial area of the facial image and at least one corresponding facial feature associated with a plurality of training images using online machine learning, the plurality of training images including a second canonical depiction independent of the first canonical depiction associated with the facial image;

applying the similarity model of the at least one facial feature associated with the query image to a plurality of other images;

determining at least one similar facial feature between the query image and at least one other image of the plurality of other images using the similarity model, the at least one similar facial feature in the at least one other image of the plurality of other images satisfying a similarity measure threshold; and

responsive to the at least one similar facial feature satisfying the similarity measure threshold, presenting for display the at least one other image including the similar facial feature.

2. The method of claim 1, wherein the similarity model is created at least in part by comparing a plurality of triplets comprising three images of the facial feature from three training images.

3. The method of claim 1, wherein the similarity model is created at least in part by comparing a plurality of mini-batches of triplets, wherein the mini-batch comprises a plurality of triplets comprising three images of the facial feature from three training images.

4. The method of claim 1, wherein the facial image is received from one or more sources including a user, a social network and an image search engine.

5. The method of claim 1, wherein the at least one facial feature is at least one of an eye, an ear, a nose, a mouth, a chin, a jaw line, a cheekbone, lips, a complexion, a skin tone, a hair color, an eyebrow, a forehead and a hairline.

6. The method of claim 1, wherein the similarity model is created at least in part by a learning algorithm with a large margin criterion.

7. The method of claim 1, wherein applying the similarity-model of the at least one facial feature associated with the query image to the plurality of other images comprises applying the similarity model to a feature set from the query image in relation to a feature set from each of the plurality of other images, the feature set based at least in part on at least one key-point associated with the facial feature.

8. The method of claim 1, wherein the at least one similar facial feature includes an area parameter.

9. The method of claim 1, wherein a plurality of results is presented, and the plurality of results are ranked and presented in ranked order.

19. The method of claim 1, wherein the similarity model is generated based on a plurality of inequality constraints.

20. The method of claim 1, wherein the transformation includes one or more of translation, rotation and standardization scaling.

10. A system for searching images and identifying images with similar facial features, the system comprising:

one or more processors;

a pre-processing model executable by the one or more processors to;

receive a facial image, the facial image including a query image;

apply a transformation to the facial image based on a set of key points associated with the facial image, the transformation of the facial image conforms to a first canonical depiction associated with an orientation of the facial area;

identify at least one facial area associated with the transformed facial image based on the set of key-points, the facial area including at least one facial feature;

a model creation module executable by the one or more processors to;

generate a similarity model for the at least one facial feature associated with the facial area of the facial image, the similarity model including determining similarity between the at least one facial feature associated with the facial area of the facial image and at least one corresponding facial feature associated with a plurality of training images using online machine learning, the plurality of training images including a second canonical depiction independent of the first canonical depiction associated with the facial image;

a similarity identifier module executable by the one or more processors to;

apply the similarity model of the at least one facial feature associated with the query image to a plurality of other images;

determine at least one similar facial feature between the query image and at least one other image of the plurality of other images using the similarity model, the at least one similar facial feature in the at least one other image of the plurality of other images satisfying a similarity measure threshold; and

a results display module executable by the one or more processors to;

responsive to the at least one similar facial feature satisfying the similarity measure threshold, present for display the at least one other image including the similar facial feature.

11. The system of claim 10, wherein the model creation module creates the similarity model at least in part by comparing a plurality of triplets comprising three images of the facial feature from three training images.

12. The system of claim 10, wherein the model creation module creates the similarity model at least in part by comparing a plurality of mini-batches of triplets, wherein the mini-batch comprises a plurality of triplets comprising three images of the facial feature from three training images.

13. The system of claim 10, wherein a plurality of results are presented by the results display module, and the system further comprises:

a ranking module operable to rank the plurality of results; and

wherein the results display module presents the plurality of results in ranked order.

14. The system of claim 10, wherein the pre-processing module receives the facial image from one or more sources including a user, a social network and an image search engine.

15. The system of claim 10, wherein the at least one facial feature is at least one of an eye, an ear, a nose, a mouth, a chin, a jaw line, a cheekbone, lips, a complexion, a skin tone, a hair color, an eyebrow, a forehead and a hairline.

16. The system of claim 10, wherein the model creation module creates the similarity model at least in part by using a learning algorithm with a large margin criterion.

17. The system of claim 10, wherein the similarity identifier module is operable to apply the similarity model of the at least one facial feature associated with the query image to the plurality of other images, the feature set based at least in part on at least one key-point associated with the facial feature.

18. The system of claim 10, wherein the at least one similar facial feature associated with the at least one other image includes an area parameter for the one similar facial feature.