Using Relevance Feedback In Face Recognition

• US 20090034805A1
• Filed: 05/10/2006
• Published: 02/05/2009
• Est. Priority Date: 05/10/2006
• Status: Active Grant

• Current Assignee: California Institute of Technology, Isis Innovation Limited, Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: AOL LLC (Apollo Global Management, Inc.)
• Inventors: Perona, Pietro, Perlmutter, Sharon M., Everingham, Mark, Holub, Alex, Alspector, Joshua, Zisserman, Andrew, Perlmutter, Keren O.
• Granted Patent: US 7,783,085 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/2163 Partitioning the feature space
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Images are searched to locate faces that are the same as a query face. Images that include a face that is the same as the query face may be presented to a user as search result images. Images also may be sorted by the faces included in the images and presented to the user as sorted search result images. The user may provide explicit or implicit feedback regarding the search result images. Additional feedback may be inferred regarding the search result images based on the user-provided feedback, and the results may be updated based on the user-provided and inferred feedback.

Claims

1. A computer-implemented method for recognizing faces within images, the method comprising:

accessing at least one query face;

determining a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face;

providing an indication of the search result images to a user;

receiving feedback related to the accuracy of the search result images;

inferring additional feedback based on the received feedback; and

updating the set of search result images based on the received feedback and the inferred additional feedback.

2. The method of claim 1 wherein determining a set of search result images includes:

creating a query feature vector for the query face;

forming a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces;

determining distances between the query feature vector and the search feature vectors;

selecting one or more search feature vectors that are within a selected distance of the query feature vector; and

designating images that include faces that correspond to the selected search feature vectors as the set of search result images.

3. The method of claim 2 wherein forming a set of search feature vectors includes:

detecting a class for the query face;

determining faces in a set of faces that have the same class as the query face; and

forming the set of search feature vectors by selecting feature vectors that correspond to the determined faces.

4. The method of claim 2 wherein determining distances between the query feature vector and the search feature vectors comprises determining a weighted Euclidean distance between the query feature vector and each of the search feature vectors.

5. The method of claim 2 wherein creating the query feature vector comprises:

locating the query face within a query image;

detecting facial features of the query face;

warping the query face based on the detected facial features;

extracting information regarding the detected facial features from the warped face; and

forming the query feature vector based on the extracted information.

6. The method of claim 5 further comprising:

detecting non-face based features from the query image;

extracting information regarding the non-face based features from the query image; and

forming the query feature vector based on the extracted information regarding the detected non-face based features and the extracted information regarding the detected facial features.

7. The method of claim 2 wherein:

accessing at least one query face comprises accessing multiple query faces;

creating a query feature vector for the query face comprises creating a query feature vector for each of the multiple faces; and

determining distances between the query feature vector and the search feature vectors includes determining distances between the multiple query feature vectors and the search feature vectors.

8. The method of claim 2 further comprising:

identifying negative faces that are determined to not be the same as the query face; and

creating negative feature vectors for the negative faces, wherein;

accessing at least one query face comprises accessing multiple query faces;

creating a query feature vector for the query face comprises creating a query feature vector for each of the multiple faces; and

determining distances between the query feature vector and the search feature vectors includes determining distances between the multiple query feature vectors and the search feature vectors and determining distances between the negative feature vectors and the search feature vectors.

9. The method of claim 1 wherein the inferred additional feedback includes information about faces in the search result images other than the application-selected face.

10. The method of claim 1 wherein the inferred additional feedback includes information about faces in the search result images other than a face selected by the user as being the same as the query face.

11. The method of claim 1 wherein the feedback related to the accuracy of the search result images includes at least one of positive feedback or negative feedback.

12. The method of claim 1 further comprising:

accessing a set of images that include faces that have not been labeled;

sorting the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and

accessing the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.

13. The method of claim 1 wherein the search result images comprise images within an online photograph album.

14. The method of claim 13 wherein the indication of the search result images provided to the user is an indication that the search result images have been associated with the query face, wherein the query face is in a labeled image.

15. The method of claim 13 wherein the indication of the search result images provided to the user includes the search result images.

16. The method of claim 1 wherein the search result images comprise images within a social network.

23. The computer-useable medium of claim 13 wherein:

the instructions for causing the computer to access at least one query face include instructions for causing the computer to access multiple query faces;

the instructions for causing the computer to create a query feature vector for the query face include instructions for causing the computer to create a query feature vector for each of the multiple faces; and

the instructions for causing the computer to determine distances between the query feature vector and the search feature vectors include instructions for causing the computer to determine distances between the multiple query feature vectors and the search feature vectors.

24. The computer-useable medium of claim 13 further includes instructions for causing the computer to:

identify negative faces that are determined to not be the same as the query face; and

create negative feature vectors for the negative faces,

wherein;

the instructions for causing the computer to access at least one query face includes instructions for causing the computer to access multiple query faces;

the instructions for causing the computer to create a query feature vector for the query face include instructions for causing the computer to create a query feature vector for each of the multiple faces; and

the instructions for causing the computer to determine distances between the query feature vector and the search feature vectors include instructions for causing the computer to determine distances between the multiple query feature vectors and the search feature vectors and determine distances between the negative feature vectors and the search feature vectors.

17. A computer-useable medium storing a computer program, the computer program including instructions for causing a computer to perform the following operations:

access at least one query image that includes at least one query face;

determine a set of search result images, wherein a search result image includes an application-selected face that is determined to be the same as the query face;

provide an indication of the search result images to a user;

receive feedback related to the accuracy of the search result images;

infer additional feedback based on the received feedback; and

update the set of search result images based on the received feedback and the inferred additional feedback.

18. The computer-useable medium of claim 17 wherein the instructions for causing the computer to determine a set of search result images include instructions for causing the computer to:

create a query feature vector for the query face;

form a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces;

determine distances between the query feature vector and the search feature vectors;

select one or more search feature vectors that are within a selected distance of the query face feature vector; and

designate images that include faces that correspond to the selected search feature vectors as the set of search result images.

19. The computer-useable medium of claim 17 wherein the instructions for causing the computer to determine distances between the query feature vector and the search feature vectors include instructions for causing the computer to determine a weighted Euclidean distance between the query face feature vector and each of the search feature vectors.

20. The computer useable medium of claim 17 wherein the inferred additional feedback includes information about faces in the search result images other than the application-selected face or information about faces in the search result images other than a face selected by the user as being the same as the query face.

21. The computer useable medium of claim 17 wherein the feedback related to the accuracy of the search result images includes at least one of positive feedback or negative feedback.

22. The computer useable medium of claim 17 further includes instructions for causing the computer to:

access a set of images that include faces that have not been labeled;

sort the set of images into groups based on the faces within the images, wherein the images in a group include faces that are determined to be the same; and

access the query face after the set of images have been sorted, wherein the query face is a face that is included in an image in one of the groups.

25. A system comprising:

a feature extractor configured to;

access at least one query face,

create a query feature vector for the query face, and

form a set of search feature vectors by creating one or more search feature vectors for faces within a set of search faces;

a face classifier configured to;

determine distances between the query feature vector and the search feature vectors,

select one or more search feature vectors that are within a selected distance of the query feature vector, and

designate images that include faces that correspond to the selected search feature vectors as a set of search result images;

an interface manager configured to;

provide an indication of the search result images to a user, and

receive feedback related to the accuracy of the search result images; and

a feedback manager configured to infer additional feedback based on the received feedback,

wherein the face classifier is also configured to update the set of search result images based on the received feedback and the inferred additional feedback.