Grouping and ranking images based on facial recognition data

• US 10,607,062 B2
• Filed: 08/31/2017
• Issued: 03/31/2020
• Est. Priority Date: 04/29/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chan, Chunkit Jacky, Lee, David, Light, Allison
• Primary Examiner(s): Hasan, Syed H
• Application Number:
  US15/692,494
• Publication Number:
  US 20170364737A1
• Time in Patent Office: 943 Days
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Technologies for grouping images, and ranking the images and the groupings, based on entities shown in the images. Images may be grouped based on faces shown in the images. Different images with faces that indicate the same entity (e.g., Adam) may be automatically grouped together. Different images with faces that indicate the same multiple entities (e.g., the people in my family) may also be automatically grouped together. Such automatic grouping may be based on facial recognition technologies. Further, images and groups of images may be automatically ranked based on the faces shown and entities represented. Such rankings may also be influenced by adjacent data that indicates family and friends and the like, and that can be used to identify such entities in the images.

Claims

1. A method performed on a computing device, the method comprising:

receiving, by the computing device for each image in a set of images, facial recognition data that, for each face in each image, comprises a set of facial feature descriptors, a face score that is based on an open/closed state of each face'"'"'s eyes and mouth and that indicates an overall quality of each face, and a face signature that, across the images in the set, uniquely identifies an entity that each face represents;

ranking, by the computing device based at least on the face signatures, images of the set into a relative ranking; and

outputting, by the computing device, the relative ranking.

2. The method of claim 1 where the ranking is further based on the face scores.

3. The method of claim 1 where a face in each image with a face score below a particular threshold is not included in the ranking.

4. The method of claim 1 where a multi-entity image in the set of images with a larger number of detected faces is ranked higher than a multi-entity image in the set of images with a smaller number of detected faces.

5. The method of claim 1 further comprising:

grouping, based on the face signatures and the face scores, at least a portion of the images in the set of images into a set of groups.

6. The method of claim 1 where an image in the set of images with a face signature indicating a first entity determined to have a larger number of friends or family is ranked higher than an image in the set of images with a face signature indicating a second entity determined to have a smaller number of friends or family.

7. The method of claim 1 where the ranking is further based on an accumulation of ranking information for all faces detected in each image.

8. A computing device comprising:

at least one processor;

memory that is coupled to the at least one processor and that includes computer-executable instructions that, based on execution by the computing device, configure the computing device to perform actions comprising;

receiving, by the computing device for each image in a set of images, facial recognition data that, for each face in each image, comprises a set of facial feature descriptors, a face score that is based on an open/closed state of each face'"'"'s eyes and mouth and that indicates an overall quality of each face, and a face signature that, across the images in the set, uniquely identifies an entity that each face represents;

ranking, by the computing device based at least on the face signatures, images of the set into a relative ranking; and

outputting, by the computing device, the relative ranking.

9. The computing device of claim 8 where the ranking is further based on the face scores.

10. The computing device of claim 8 where a face in each image with a face score below a particular threshold is not included in the ranking.

11. The computing device of claim 8 where a multi-entity image in the set of images with a larger number of detected faces is ranked higher than a multi-entity image in the set of images with a smaller number of detected faces.

12. The computing device of claim 8, the actions further comprising:

grouping, based on the face signatures and the face scores, at least a portion of the images in the set of images into a set of groups.

13. The computing device of claim 8 where an image in the set of images with a face signature indicating a first entity determined to have a larger number of friends or family is ranked higher than an image in the set of images with a face signature indicating a second entity determined to have a smaller number of friends or family.

14. The computing device of claim 8 where the ranking is further based on an accumulation of ranking information for all faces detected in each image.

15. At least one non-transitory computer-readable media that includes computer-executable instructions that, based on execution by a computing device, configure the computing device to perform actions comprising:

receiving, by the computing device for each image in a set of images, facial recognition data that, for each face in each image, comprises a set of facial feature descriptors, a face score that is based on an open/closed state of each face'"'"'s eyes and mouth and that indicates an overall quality of each face, and a face signature that, across the images in the set, uniquely identifies an entity that each face represents;

ranking, by the computing device based at least on the face signatures, images of the set into a relative ranking; and

outputting, by the computing device, the relative ranking.

16. The at least one non-transitory computer-readable media of claim 15 where the ranking is further based on the face scores, or where the ranking is further based on an accumulation of ranking information for all faces detected in each image.

17. The at least one non-transitory computer-readable media of claim 15 where a face in each image with a face score below a particular threshold is not included in the ranking.

18. The at least one non-transitory computer-readable media of claim 15 where a multi-entity image in the set of images with a larger number of detected faces is ranked higher than a multi-entity image in the set of images with a smaller number of detected faces.

19. The at least one non-transitory computer-readable media of claim 15, the actions further comprising:

grouping, based on the face signatures and the face scores, at least a portion of the images in the set of images into a set of groups.

20. The at least one non-transitory computer-readable media of claim 15 where an image in the set of images with a face signature indicating a first entity determined to have a larger number of friends or family is ranked higher than an image in the set of images with a face signature indicating a second entity determined to have a smaller number of friends or family.