Manually-assisted automated indexing of images using facial recognition

• US 8,306,284 B2
• Filed: 07/12/2006
• Issued: 11/06/2012
• Est. Priority Date: 07/18/2005
• Status: Active Grant

• Current Assignee: Centerframe, LLC
• Original Assignee: Hysterical Sunset Limited
• Inventors: Goldberg, David A., Gray, Robert, Angell, Joe, Rose, Jeffrey
• Primary Examiner(s): COUSO, JOSE L
• Application Number:
  US12/442,361
• Publication Number:
  US 20120008837A1
• Time in Patent Office: 2,309 Days
• Field of Search: 382/100, 382115-118, 358 77- 78, 358/143, 707706-707, 707/758, 707/915, 707/999.001
• US Class Current: 382/118
• CPC Class Codes: G06V 10/987 with the intervention of an operator
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

The present method relates to the manual assistance for the automated indexing (100) of a collection of images using facial recognition. In a first automated indexing step, automated indexing of faces within a collection of images is performed, creating sets of faces each of which comprises faces that are determined by the automated process to be representative of the same person. In a second splitting step (200), sets are displayed to an operator, who determines whether there are false-positive associations within a set. If false-positive associations are found, the faces representing different people are manually split into different sets, hi this way, there will be no false-positive associations within the collection of images, hi a third merging step (300), sets that have some degree of similarity are presented to the operator, who determines whether the two sets comprise representations of the same person. If so, the two sets are manually merged, thereby eliminating false-negative errors. In this way, all of the faces in the image collection can be completely and accurately indexed.

Claims

1. A method for indexing face images in a collection of images according to the persons that are represented by the face images, comprising:

automatically indexing, by a computer processor, the face images in the collection so as to create a multiplicity of sets of face images, wherein each set comprises face images that are determined to represent the same person, and wherein the sets comprise errors chosen from the group consisting of false-positive errors and false-negative errors;

presenting to an operator by a display, a set, wherein the operator manually determines whether all of the face images in the set represent the same person;

manually splitting a set representing more than one person into a multiplicity of sets, each of which consist of face images that represent a single person;

displaying to the operator by a display at least one face image from a first set and at least one face image from a second set, wherein the operator manually determines whether the first set and the second set comprise face images representing the same person; and

manually merging the first set and the second set if the sets comprise face images representing the same person;

wherein the resulting sets of face images now contain no false-positive or false negative errors.

2. The method of claim 1 wherein presenting additionally comprises exhibiting to the operator, by a display, a first source image from which a first selected face in the set was derived and a second source image from which a second selected face in the set was derived, wherein the first source image and the second source image assist the operator in manually determining whether the first face and the second face represent the same person.

3. The method of claim 1 wherein displaying additionally comprises exhibiting to the operator by a display a first source image from which a first chosen face from the first set was derived and a second source image from which a second chosen face from the second set was derived, wherein the first source image and the second source image assist the operator in manually determining whether the first face and the second face represent the same person.

4. The method of claim 1 wherein presenting comprises generating by a computer processor a thumbnail image of at least one face in the set.

5. The method of claim 4 wherein generating for each face comprises determining with the assistance of a computer processor the locations of the eyes within the image comprising the face, scaling by a computer processor the image so that the eyes are separated by a predetermined number of pixels, and cropping by a computer processor the image with respect to the eye locations.

6. The method of claim 1 wherein displaying comprises generating a thumbnail image on a display of at least one face from the first set and at least one face from the second set.

7. The method of claim 6 wherein generating for each face comprises determining with the assistance of a computer processor the locations of the eyes within the source image from which the face was derived, scaling by a computer processor the image so that the eyes are separated by a predetermined number of pixels, and cropping by a computer processor the image with respect to the eye locations.

8. The method of claim 1 wherein displaying additionally comprises displaying on a display to the operator one or more face images from a third set.

9. The method of claim 1 additionally comprising a second automated indexing performed by a computer processor on the collection to which incremental images have been added, utilizing the resulting sets of face images, wherein each individual resulting set is a subset of the sets produced in the second automated indexing.

10. The method of claim 9 wherein face images from the incremental images are added to at most one of the resulting sets.

11. A method for removing false-positive associations between face images that have been automatically indexed from a collection of images into a set representative of a unique person, comprising:

presenting to an operator by a display the set of face images;

manually splitting a set representing more than one person into a multiplicity of sets, each of which consist of face images that represent a single person;

assigning, by a computer processor, an association between the split images;

displaying to the operator by a display images from a first set;

manually selecting by the operator one or more face images from the first set that are false-positive associations to other face images within the first set;

removing by a computer processor the selected face images from the first set;

wherein the resulting set of face images now contain no false-positive errors.

12. The method of claim 11 wherein the face images are ordered by the time at which the images were captured.

13. The method of claim 11 wherein the face images are ordered by the similarity of the face images as determined by automated processor means.

14. The method of claim 11 additionally comprising creating by a computer processor a new set from the face images that have been removed, wherein the new set of face images are representative of a second person.

21. The method of claim 11, further comprising:

displaying to the operator by a display images from a second set;

manually selecting by the operator one or more face images from the second set that are false-positive associations to other face images within the second set;

removing by a computer processor the selected face images from the second set;

wherein the resulting second set of face images now contain no false-positive errors.

22. The method of claim 21, further comprising manually merging the first set and the second set if the sets comprise face images representing the same person.

15. A system for indexing face images in a collection of images according to the persons that are represented by the face images, comprising:

an automated indexer that associates face images into a multiplicity of sets, each set of which is determined to comprise face images representative of the same person, wherein the sets comprise errors chosen from the group consisting of false-positive errors and false negative errors;

an operator;

a splitting screen displaying face images from a first set;

a selection tool allowing the operator to select one or more face images from the first set that are false-positive associations with other face images within the first set;

a splitting function allowing the operator to remove the selected face images from the first set;

a merging screen presenting face images from a second set and face images from a third set, wherein one or more face images from the second set has similarity to one or more face images from the third set by some criterion of the automated indexer; and

a merging function allowing the operator to merge the face images from the third set with the second set;

wherein the application by the operator of the splitting function to face images selected with the selecting tool on the splitting screen removes false-positive associations, and the application by the operator of the merging function to face images presented on the merging screen removes false-negative associations, so that the resulting indexed face images will not contain false-positive and false-negative associations.

16. The system of claim 15 wherein the face images displayed in the splitting screen are ordered by the time at which the images from which the face images were derived were captured.

17. The system of claim 15 wherein the face images displayed in the splitting screen are ordered by the similarity of the face images as determined by the automated indexer.

18. The system of claim 15 wherein the face images displayed in the splitting screen are a subset of the sources images from which the face images were derived.

19. The system of claim 18 wherein the splitting screen further comprises one or more source images from which the face images were derived.

20. The system of claim 15 wherein the face images presented in the merging screen are a subset of the sources images from which the face images were derived.