Systems and methods for identifying a user's demographic characteristics based on the user's social media photographs

• US 9,589,205 B2
• Filed: 05/15/2014
• Issued: 03/07/2017
• Est. Priority Date: 05/15/2014
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Joshi, Dhiraj, Chen, Francine, WILCOX, Lynn Donelle
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): VANCHY JR, MICHAEL J
• Application Number:
  US14/279,184
• Publication Number:
  US 20150332087A1
• Time in Patent Office: 1,027 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/178 estimating age from face image; using age information for improving recognition

Abstract

A method identifies user demographic characteristics, such as age and gender. First, a plurality of photos are acquired from postings to a social media site by a specific user. A plurality of facial images are identified in the acquired photos. Using a selected subset of the facial images, the method estimates one or more demographic characteristics of the user. In some cases, the estimation uses a profile photograph of the user. In some cases, the facial images are clustered based on similarity. The cluster that most closely matches the profile photo is used for demographic analysis to estimate the characteristics of the user. In some cases, a cluster is selected based on cluster purity, and the images in the cluster are used to estimate the characteristics of the user. In some cases, other self-portrait features are used, such as large facial images or arms along the perimeter of photographs.

Claims

1. A method of identifying user demographic characteristics, comprising:

at a computing device with one or more processors, and memory storing one or more programs for execution by the one or more processors;

acquiring a plurality of photos posted to a social media site by a user;

identifying a plurality of facial images in the acquired photos;

inferring that each of a subset of the plurality of photos is a respective self-potrait taken by a respective person corresponding to a respective facial image in the respective photo, wherein each inference is based on a respective image of the respective person'"'"'s arm along the perimeter of the respective photo;

selecting a subset of facial images from the plurality of facial images, including selecting one or more of the facial images based on their presence in one or more of the inferred self-portraits, wherein selecting the subset of facial images comprises;

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of cluster that has a highest cluster purity;

wherein the selected subset of facial images comprises the facial images in the selected cluster; and

estimating one or more demographic characteristics of the user based on the selected subset of facial images.

2. The method of claim 1, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site;

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that most closely matches the acquired profile photo;

wherein the selected subset of facial images comprises the facial images in the selected cluster.

3. The method of claim 1, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site; and

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters, using the profile photo as a clustering seed;

wherein the selected subset of facial images comprises the facial images in the cluster corresponding to the profile photo.

4. The method of claim 1, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site; and

selecting a subset of the facial images whose similarity to the acquired profile photo is greater than a predefined threshold similarity.

5. The method of claim 4, wherein similarity is measured using spatial pyramid matching.

6. The method of claim 1, wherein estimating the one or more demographic characteristics of the user further comprises:

determining demographic characteristics for a statistically significant percentage of the identified facial images;

computing a distribution of the determined demographic characteristics; and

using the distribution as feature inputs to a classifier trained on historical data of demographic characteristics for users'"'"' social media photos and demographic data for the users.

7. The method of claim 1, wherein estimating the one or more demographic characteristics of the user comprises:

applying N distinct evaluation methods to the facial images, each method identifying a respective set of facial images that are estimated to be images of the user, wherein N is an integer greater than one;

combining the identified respective sets of facial images to form a candidate set of facial images; and

for each facial image in the candidate set;

forming a respective N-dimensional feature vector, wherein each dimension of the respective feature vector corresponds to a unique one of the evaluation methods, and the values in the respective feature vector indicate which of the evaluation methods estimate the respective facial image to be an image of the user; and

using the respective feature vector as input to a trained classifier to estimate whether the respective facial image is an image of the user;

wherein the selected subset of the facial images includes facial images determined by the trained classifier to be images of the user.

8. The method of claim 1, wherein estimating the one or more demographic characteristics of the user comprises:

applying a plurality of evaluation methods to the facial images, each method identifying a respective set of facial images that are estimated to be images of the user;

combining the identified respective sets of facial images to form a candidate set of facial images;

scoring each facial image in the candidate set using a weighted average of scores provided by the evaluation methods;

wherein the selected subset of the facial images includes facial images whose weighted average scores exceed a predefined threshold value.

9. The method of claim 1, wherein the one or more demographic characteristics include age and gender.

10. A computer system for identifying user demographic characteristics, comprising:

one or more processors;

memory; and

one or more programs stored in the memory, the one or more programs comprising instructions for;

acquiring a plurality of photos posted to a social media site by a user;

identifying a plurality of facial images in the acquired photos;

inferring that each of a subset of the plurality of photos is a respective self-portrait taken by a respective person corresponding to a respective facial image in the respective photo, wherein each inference is based on a respective image of the respective person'"'"'s arm along the perimeter of the respective photo and each inference is further based on the respective photo having a facial image that exceeds a threshold percentage coverage of the respective photo;

selecting a subset of facial images from the plurality of facial images, including selecting one or more of the facial images based on their presence in one or more of the inferred self-portraits; and

estimating one or more demographic characteristics of the user based on the selected subset of facial images.

11. The computer system of claim 10, wherein the instructions for selecting the subset of facial images further comprise instructions for:

acquiring a profile photo of the user from the social media site;

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that most closely matches the acquired profile photo;

wherein the selected subset of facial images comprises the facial images in the selected cluster.

12. The computer system of claim 10, wherein the instructions for selecting the subset of facial images further comprise instructions for:

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that has a highest cluster purity;

wherein the selected subset of facial images comprises the facial images in the selected cluster.

13. The computer system of claim 10, wherein the instructions for estimating the one or more demographic characteristics of the user further comprise instructions for:

determining demographic characteristics for a statistically significant percentage of the identified facial images;

computing a distribution of the determined demographic characteristics; and

using the distribution as feature inputs to a classifier trained on historical data of demographic characteristics for users'"'"' social media photos and demographic data for the users.

14. The computer system of claim 10, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site; and

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters, using the profile photo as a clustering seed;

wherein the selected subset of facial images comprises the facial images in the cluster corresponding to the profile photo.

15. The computer system of claim 10, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site; and

selecting a subset of the facial images whose similarity to the acquired profile photo is greater than a predefined threshold similarity.

16. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer with an image sensor, the one or more programs comprising instructions for:

acquiring a plurality of photos posted to a social media site by a user;

identifying a plurality of facial images in the acquired photos;

inferring that each of a subset of the plurality of photos is a respective self-portrait taken by a respective person corresponding to a respective facial image in the respective photo, wherein each inference is based on a respective image of the respective person'"'"'s arm along the perimeter of the respective photo;

selecting a subset of facial images from the plurality of facial images, including selecting one or more of the facial images based on their presence in one or more of the inferred self-portraits; and

estimating one or more demographic characteristics of the user based on the selected subset of facial images and based on images of identified non-human background objects in the scenes of the acquired photos, wherein the identified non-human background objects have a skew with respect to the one or more demographic characteristics.

17. The computer readable storage medium of claim 16, wherein the instructions for selecting the subset of facial images further comprise instructions for:

acquiring a profile photo of the user from the social media site; and

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters, using the profile photo as a clustering seed;

wherein the selected subset of facial images comprises the facial images in the cluster corresponding to the profile photo.

18. The computer readable storage medium of claim 16, wherein the images of identified objects appear in photos that are identified as self-portraits.

19. The computer readable storage medium of claim 16, wherein the instructions for selecting the subset of facial images further comprise instructions for:

acquiring a profile photo of the user from the social media site;

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that most closely matches the acquired profile photo;

wherein the selected subset of facial images comprises the facial images in the selected clusters.

20. The computer readable storage medium of claim 16, wherein the instructions for estimating the one or more demographic characteristics of the user further comprise instructions for:

determining demographic characteristics for a statistically significant percentage of the identified facial images;

computing a distribution of the determined demographic characteristics; and

using the distribution as feature inputs to a classifier trained on historical data of demographic characteristics for users'"'"' social media photos and demographic data for the users.