Ranking based on facial image analysis

• US 10,296,811 B2
• Filed: 08/30/2016
• Issued: 05/21/2019
• Est. Priority Date: 03/01/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Krupka, Eyal, Abramovski, Igor, Kviatkovsky, Igor
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US15/251,649
• Publication Number:
  US 20160371565A1
• Time in Patent Office: 994 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

A user'"'"'s collection of images may be analyzed to identify people'"'"'s faces within the images, then create clusters of similar faces, where each of the clusters may represent a person. The clusters may be ranked in order of size to determine a relative importance of the associated person to the user. The ranking may be used in many social networking applications to filter and present content that may be of interest to the user. In one use scenario, the clusters may be used to identify images from a second user'"'"'s image collection, where the identified images may be pertinent or interesting to the first user. The ranking may also be a function of user interactions with the images, as well as other input not related to the images. The ranking may be incrementally updated when new images are added to the user'"'"'s collection.

Claims

1. A method comprising:

accessing an image collection of a user, the image collection comprising a plurality of images;

performing image processing on the plurality of images to identity face objects and create corresponding facial feature vectors;

processing the facial feature vectors to determine respective person identities associated with the face objects;

determining number of occurrences, within the image collection, of the face objects for the person identities;

ranking the person identities based at least in part on the numbers of occurrences of the face objects within the images of the image collection of the user;

accessing data items that relate to the person identities, the data items comprising different content than the images of the image collection;

prioritizing the data items that relate to the person identities based at least in part on the ranking of the person identities; and

sending a data feed comprising the prioritized data items to a client device of the user over a network.

2. The method of claim 1, further comprising:

presenting, via the data feed, first data items relating to a first person identify relatively more prominently than second data items relating to a second person identity when the first person identity is ranked relatively higher than the second person identity.

3. The method of claim 2, wherein presenting the first data items relatively more prominently than the second data items comprises causing the first data items to be displayed in a relatively more prominent position than the second data items.

4. The method of claim 1, the data items comprising messages, the method further comprising:

presenting, via the data feed, first messages relating to a first person identity relatively more prominently than second messages relating to a second person identity when the first person identity is ranked relatively higher than the second person identity.

5. The method of claim 1, the data items comprising status updates, the method further comprising:

presenting, via the data feed, first status updates relating to a first person identity relatively more prominently than second status updates relating to a second person identity when the first person identity is ranked relatively higher than the second person identity.

6. The method of claim 1, further comprising:

ranking an identity of a first person relatively higher than an identity of a second person when the face objects in the image collection include more occurrences of the first person'"'"'s face than the second person'"'"'s face.

7. A system comprising:

at least one hardware processor; and

at least one memory or non-volatile storage media storing computer-readable instructions which, when executed by the at least one hardware processor, cause the at least one hardware processor to;

access an image collection, the image collection comprising a plurality of images;

perform image processing on the plurality of images to create feature vectors;

process the feature vectors to identify occurrences of objects within the plurality of images of the image collection and cluster the objects into object clusters;

identify different sizes of the object clusters, the different sizes of the object clusters reflecting different numbers of occurrences of objects in the image collection;

obtain data items that relate to identifiers associated with the object clusters, the data items comprising different content than the image collection;

generate a data feed that reflects a prioritization of the data items based at least on the different sizes of the object clusters; and

send the data feed over a network to a client device.

8. The system of claim 7, wherein the objects comprise faces.

9. The system of claim 7, wherein the image collection is associated with a user and the client device is associated with the user.

10. The system of claim 9, wherein the image collection is associated with a social network account of the user for a social networking application and the data feed is provided by the social networking application to the user.

11. The system of claim 10, wherein the data feed is a news feed provided by the social networking application.

12. The system of claim 10, wherein the computer-readable instructions, when executed by the at least one hardware processor, cause the at least one hardware processor to:

identify status updates directed to the user of the social networking application;

prioritize the status updates based at least on the different sizes of the object clusters; and

include the prioritized status updates in the data feed sent over the network to the client device.

13. The system of claim 10, wherein the computer-readable instructions, when executed by the at least one hardware processor, cause the at least one hardware processor to:

identify messages directed to the user of the social networking application;

prioritize the messages based at least on the different sizes of the object clusters; and

include the prioritized messages in the data feed sent over the network to the client device.

14. A server comprising:

at least one hardware processor; and

at least one memory or non-volatile storage media storing computer-readable instructions which, when executed by the at least one hardware processor, cause the at least one hardware processor to;

provide a social network service, the social network service being configured to;

maintain a plurality of social networking accounts for a plurality of social network users;

maintain social networking relationships between individual social network users;

determine specific social network users that have existing social networking relationships with a particular user, the particular user having an image collection comprising a plurality of images;

perform image processing on the plurality of images to identify face objects and create corresponding facial feature vectors;

process the facial feature vectors to identify specific face objects associated with the specific social network users;

determine numbers of occurrences, within the image collection, of the specific face objects associated with the specific social network users;

determine respective importance levels of the specific social network users to the particular user based at least in part on the numbers of the occurrences of the specific face objects within the image collection of the particular user;

generate a data feed for the particular user, the data feed including data items other than the plurality of images of the image collection, the data items relating to the specific social network users that have the existing social networking relationship with the particular user; and

send the data feed over a network to a client device associated with the particular user, the data items in the data feed being prioritized based at least in part on the respective importance levels of the specific social network users to the particular user.

15. The server of claim 14, wherein the computer-readable instructions, when executed by the at least one hardware processor, cause the at least on hardware processor to:

identify clusters of different face objects based at least on the facial feature vectors, different clusters being associated with different social network users.

16. The server of claim 15, the respective importance levels being determined based at least in part on sizes of the clusters of different face objects.

17. The server of claim 14, the face objects occurring in one or more videos in the image collection of the particular user.

18. The server of claim 14, the face objects occurring in one or more still images in the image collection of the particular user.

19. The server of claim 15, wherein the computer-readable instructions, when executed by the at least one hardware processor, cause the at least one hardware processor to:

compare the facial feature vectors in a space to determine respective distances between the facial feature vectors; and

identify the clusters based at least on the respective distances.

20. The server of claim 14, wherein the computer-readable instructions, when executed by the at least one hardware processor, cause the at least one hardware processor to:

determine a ranking of the specific social network users such that a first social network user is ranked relatively higher than a second social network user based at least in part on the image collection including more occurrences of the first social network user'"'"'s face than the second social network user'"'"'s face, the data feed being prioritized based at least on the ranking.