ADJUSTMENT OF FACIAL IMAGE SEARCH RESULTS

• US 20150261787A1
• Filed: 03/12/2014
• Published: 09/17/2015
• Est. Priority Date: 03/12/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ding, Zhongli, Adam, Hartwig, Keller, Randall G., Hu, Wenze
• Granted Patent: US 9,268,793 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for adjusting facial image search results. In one aspect, a method includes detecting, in images that are responsive to a query, faces in the images and building facial templates. The images are then clustered according to similarity measures of the facial templates. The cluster with the highest quantity is selected, and each image in the set of returned images is compared to each image in the selected cluster. The similarity of each face with the faces in the largest cluster, based on the facial templates, is determined as an inlier score for the image in which the face is depicted. The system then increases the rank of images with high inlier scores and decreases the rank of images with low inlier scores.

Claims

1. A method performed by data processing apparatus, the method comprising:

receiving data identifying images determined to be responsive to a query, each image having an associated search score that is a measure of responsiveness to the query, and the images ranked according to a first order based on the search scores, and in response to receiving the data identifying the images;

for each image in a first set of the images, detecting a face depicted in the image and generating a face template from the depicted face;

clustering the images in the first set of images into a plurality of clusters based on similarity of the facial templates to each other, each cluster including at least one of the images, and each image belonging to only one cluster;

determining a quantity of images belonging to each cluster;

selecting the cluster with the highest quantity relative to the quantity of other clusters as an inlier cluster;

for each image in the first set of images, determining an inlier score for the image that is a measure of similarity of the facial template of the image to the facial templates of the images that belong to the inlier cluster; and

re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images.

2. The method of claim 1, where the first set of images is a proper subset of the images determined to be responsive to a query.

3. The method of claim 1, wherein re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images comprises increasing the rank of images with high inlier scores relative to images with low inlier scores.

4. The method of claim 1, wherein re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images comprises:

adjusting the search score of each image in the set of images by its respective inlier score; and

re-ranking the images determined to be responsive to the query based, in part, on the adjusted search scores of the images determined to be responsive to the query.

5. The method of claim 1, wherein:

clustering the images comprises;

determining pairwise similarity scores based on the facial templates of image pairs;

determining that image pairs having pairwise similarity scores that meet a duplicate threshold are duplicate images and grouping duplicate images into respective sets of duplicate images; and

determining that each image for which no pairwise similarity score meets the duplicate threshold is a unique image.

6. The method of claim 5, wherein determining the inlier score comprises determining the inlier scores based on the unique images and only one image from each set of duplicate images.

7. The method of claim 1, wherein determining the inlier score comprises determining, for each image that belongs to a cluster with a quantity less than or equal to a minimum quantity threshold, a minimum inlier score for the image.

8. The method of claim 7, wherein the minimum quantity threshold is 1.

9. The method of claim 1, wherein determining an inlier score for the image image comprises determining a central tendency similarity measure of the facial template of the image to the facial templates of the images that belong to the inlier cluster.

10. The method of claim 1, wherein receiving data identifying images determined to be responsive to a query comprises receiving only data identifying images determined to be responsive to a name query that specifies a person name and that meets a popularity threshold.

11. A computer storage medium encoded instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving data identifying images determined to be responsive to a query, each image having an associated search score that is a measure of responsiveness to the query, and the images ranked according to a first order based on the search scores, and in response to receiving the data identifying the images;

for each image in a first set of the images, detecting a face depicted in the image and generating a face template from the depicted face;

clustering the images in the first set of images into a plurality of clusters based on similarity of the facial templates to each other, each cluster including at least one of the images, and each image belonging to only one cluster;

determining a quantity of images belonging to each cluster;

selecting the cluster with the highest quantity relative to the quantity of other clusters as an inlier cluster;

for each image in the first set of images, determining an inlier score for the image that is a measure of similarity of the facial template of the image to the facial templates of the images that belong to the inlier cluster; and

re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images.

12. A system comprising:

a data processing apparatus; and

computer storage medium encoded instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving data identifying images determined to be responsive to a query, each image having an associated search score that is a measure of responsiveness to the query, and the images ranked according to a first order based on the search scores, and in response to receiving the data identifying the images;

for each image in a first set of the images, detecting a face depicted in the image and generating a face template from the depicted face;

clustering the images in the first set of images into a plurality of clusters based on similarity of the facial templates to each other, each cluster including at least one of the images, and each image belonging to only one cluster;

determining a quantity of images belonging to each cluster;

selecting the cluster with the highest quantity relative to the quantity of other clusters as an inlier cluster;

for each image in the first set of images, determining an inlier score for the image that is a measure of similarity of the facial template of the image to the facial templates of the images that belong to the inlier cluster; and

re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images.

13. The system of claim 12, where the first set of images is a proper subset of the images determined to be responsive to a query.

14. The system of claim 12, wherein re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images comprises increasing the rank of images with high inlier scores relative to images with low inlier scores.

15. The system of claim 12, wherein re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images comprises:

adjusting the search score of each image in the set of images by its respective inlier score; and

re-ranking the images determined to be responsive to the query based, in part, on the adjusted search scores of the images determined to be responsive to the query.

16. The system of claim 12, wherein:

clustering the images comprises;

determining pairwise similarity scores based on the facial templates of image pairs;

determining that image pairs having pairwise similarity scores that meet a duplicate threshold are duplicate images and grouping duplicate images into respective sets of duplicate images; and

determining that each image for which no pairwise similarity score meets the duplicate threshold is a unique image.

17. The system of claim 16, wherein determining the inlier score comprises determining the inlier scores based on the unique images and only one image from each set of duplicate images.

18. The system of claim 12, wherein determining the inlier score comprises determining, for each image that belongs to a cluster with a quantity less than or equal to a minimum quantity threshold, a minimum inlier score for the image.

19. The system of claim 12, wherein determining an inlier score for the image comprises determining a central tendency similarity measure of the facial template of the image to the facial templates of the images that belong to the inlier cluster.

20. The system of claim 12, wherein receiving data identifying images determined to be responsive to a query comprises receiving only data identifying images determined to be responsive to a name query that specifies a person name and that meets a popularity threshold.