Scalable face image retrieval

• US 8,498,455 B2
• Filed: 06/03/2010
• Issued: 07/30/2013
• Est. Priority Date: 06/03/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ke, Qifa, Li, Yi, Shum, Heung-Yeung, Sun, Jian, Wu, Zhong
• Primary Examiner(s): Wang, Claire X
• Assistant Examiner(s): JAMES, THOMAS AIDEN
• Application Number:
  US12/792,750
• Publication Number:
  US 20110299743A1
• Time in Patent Office: 1,153 Days
• Field of Search: None
• US Class Current: 382/118
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A system for identifying individuals in digital images and for providing matching digital images is provided. A set of images that include faces of known individuals is received. Faces are detected in the images and facial components are identified in each face. Visual words corresponding to the facial components are generated, stored, and associated with identifiers of the individuals. At a later time, a user may provide an image that includes the face of one of the known individuals. Visual words are determined from the face of the individual in the provided image and matched against the stored visual words. Images associated with matching visual words are ranked and presented to the user.

Claims

1. A method comprising:

receiving an image at a computing device;

detecting a face in the image by the computing device;

identifying a plurality of facial components of the detected face in the image by the computing device;

generating a plurality of visual words from the identified plurality of facial components by the computing device;

comparing the generated plurality of visual words from the identified facial components with a plurality of stored visual words by the computing device, wherein each visual word is associated with one of a plurality of stored images;

determining a plurality of candidate matching images from the plurality of stored images based on the comparison by the computing device;

selecting a plurality of reference images from the plurality of candidate matching images by the computing device;

for each candidate matching image of the plurality of candidate matching images, determining a distance between the candidate matching image and each reference image of the plurality of reference images by the computing device; and

ranking the candidate matching images based on the determined distances by the computing device.

2. The method of claim 1, further comprising identifying at least one individual associated with the plurality of candidate matching images.

3. The method of claim 2, further comprising one of providing information about the individual associated with the plurality of candidate matching images, and providing images of the individual associated with the plurality of candidate matching images.

4. The method of claim 1, further comprising defining a grid of elements for each facial component in the plurality of facial components.

5. The method of claim 4, wherein generating a plurality of visual words from the identified plurality of facial components comprises:

extracting an image patch from each grid element of an identified facial component; and

computing a descriptor for each extracted image patch.

6. The method of claim 1, wherein selecting a plurality of reference images from the plurality of candidate matching images comprises:

generating a hamming signature of the received image and each of the plurality of candidate matching images; and

selecting the candidate matching images that have hamming signatures that have a closest distance to the hamming signature of the received image and one or more previously selected candidate matching images.

18. The method of claim 1, further comprising receiving an indication of a scenario with the received image, and wherein determining the plurality of candidate matching images from the plurality of stored images based on the comparison further comprises determining the plurality of candidate matching images based on the scenario.

7. A method comprising:

receiving a plurality of images by a computing device, wherein each image includes a face associated with an individual;

detecting the face in each of the plurality of images by the computing device;

identifying a plurality of facial components for each detected face by the computing device;

generating a plurality of visual words from the identified plurality of facial components of each detected face by the computing device;

for each generated plurality of visual words, storing the generated plurality of visual words by the computing device;

receiving an image by the computing device, wherein the image is not part of the plurality of images;

determining a plurality of candidate matching images from the plurality of images for the received image by the computing device;

selecting a plurality of reference images from the plurality of candidate matching images by the computing device;

for each candidate matching image of the plurality of matching images, determining a distance between the candidate matching image and each reference image of the plurality of reference images by the computing device; and

ranking the candidate matching images based on the determined distances by the computing device.

8. The method of claim 7, wherein storing the generated plurality of visual words comprises storing the generated plurality of visual words as an inverted index.

9. The method of claim 7, wherein generating a plurality of visual words from the identified plurality of facial components of each detected face comprises:

defining a grid of elements for each facial component from the plurality of facial components;

extracting an image patch from each grid element; and

computing a descriptor for each extracted image patch.

10. The method of claim 7, wherein determining a plurality of candidate matching images from the plurality of images for the received image comprises:

detecting a face in the image;

identifying a plurality of facial components of the detected face in the image;

generating a plurality of visual words from the identified plurality of facial components;

comparing the generated plurality of visual words from the identified facial components with the plurality of stored visual words; and

determining the plurality of candidate matching images from the plurality of images for the received image based on the comparison.

11. The method of claim 7, wherein selecting a plurality of reference images from the plurality of candidate matching images comprises:

generating a hamming signature of the received image and each of the plurality of candidate matching images; and

selecting the candidate matching images that have hamming signatures that have a closest determined distance to the hamming signature of the first image and one or more previously selected candidate matching images.

12. The method of claim 11, wherein the candidate matching images are manually selected.

13. A system comprising:

at least one computing device;

a training component that;

receives a plurality of images, wherein each image includes a face associated with an individual;

detects the face in each of the plurality of images;

for each detected face, identifies a plurality of facial components of the detected face;

for each identified facial component, generates a plurality of visual words from the identified facial component of the detected face; and

for each generated plurality of visual words, stores the generated plurality of visual words with an identifier of the individual associated with the detected face included in the image and an identifier of the image; and

a matching component that;

receives an image, wherein the image is not part of the plurality of images;

detects a face in the image;

identifies a plurality of facial components of the detected face in the image;

generates a plurality of visual words from the identified plurality of facial components;

compares the generated plurality of visual words from the identified facial components with the plurality of stored visual words;

determines a plurality of candidate matching images from the plurality of images for the received image based on the comparison;

selects a plurality of reference images from the plurality of candidate matching images;

for each candidate matching image of the plurality of matching images, determines a distance between the candidate matching image and each reference image of the plurality of matching images; and

ranks the candidate matching images based on the determined distances.

14. The system of claim 13, wherein the training component stores the generated plurality of visual words as an inverted index.

15. The system of claim 13, wherein the training component further normalizes the identified plurality of facial components.

16. The system of claim 13, wherein the plurality of facial components comprises a nose, two eyes, a left mouth region, and a right mouth region.

17. The system of claim 13, wherein the matching component selects a plurality of reference images from the plurality of candidate matching images by:

generating a hamming signature of the received image and each of the plurality of candidate matching images; and

selecting the candidate matching images that have hamming signatures that have a closest distance to the hamming signature of the first image and one or more previously selected candidate matching images.