Systems and Methods for Identifying a User's Demographic Characteristics Based on the User's Social Media Photographs

• US 20150332087A1
• Filed: 05/15/2014
• Published: 11/19/2015
• Est. Priority Date: 05/15/2014
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Joshi, Dhiraj, Chen, Francine, Wilcox, Lynn Donelle
• Granted Patent: US 9,589,205 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/178 estimating age from face image; using age information for improving recognition

Abstract

A method identifies user demographic characteristics, such as age and gender. First, a plurality of photos are acquired from postings to a social media site by a specific user. A plurality of facial images are identified in the acquired photos. Using a selected subset of the facial images, the method estimates one or more demographic characteristics of the user. In some cases, the estimation uses a profile photograph of the user. In some cases, the facial images are clustered based on similarity. The cluster that most closely matches the profile photo is used for demographic analysis to estimate the characteristics of the user. In some cases, a cluster is selected based on cluster purity, and the images in the cluster are used to estimate the characteristics of the user. In some cases, other self-portrait features are used, such as large facial images or arms along the perimeter of photographs.

Claims

1. A method of identifying user demographic characteristics, comprising:

at a computing device with one or more processors, and memory storing one or more programs for execution by the one or more processors;

acquiring a plurality of photos posted to a social media site by a user;

identifying a plurality of facial images in the acquired photos;

estimating one or more demographic characteristics of the user based on a selected subset of the facial images.

2. The method of claim 1, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site;

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that most closely matches the acquired profile photo;

wherein the selected subset of facial images comprises the facial images in the selected cluster.

3. The method of claim 1, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site; and

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters, using the profile photo as a clustering seed;

wherein the selected subset of facial images comprises the facial images in the cluster corresponding to the profile photo.

4. The method of claim 1, wherein selecting the subset of facial images comprises:

acquiring a profile photo of the user from the social media site; and

selecting a subset of the facial images whose similarity to the acquired profile photo is greater than a predefined threshold similarity.

5. The method of claim 4, wherein similarity is measured using spatial pyramid matching.

6. The method of claim 1, wherein selecting the subset of facial images comprises:

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that has the highest cluster purity;

wherein the selected subset of facial images comprises the facial images in the selected cluster.

7. The method of claim 1, wherein selecting the subset of facial images comprises selecting one or more of the facial images based on their presence in acquired photographs that have one or more self-portrait features.

8. The method of claim 7, wherein the self-portrait features for a photograph include having a facial image that exceeds a threshold percentage coverage of the photograph.

9. The method of claim 7, wherein the self-portrait features for a photograph include detecting an image of a person'"'"'s arm in the photograph along the perimeter of the photograph.

10. The method of claim 1, wherein estimating the one or more demographic characteristics of the user is further based on images of identified objects in the acquired photos, wherein the identified objects have a skew with respect to the one or more demographic characteristics.

11. The method of claim 10, wherein the images of identified objects appear in photos that are identified as self-portraits.

12. The method of claim 1, wherein estimating the one or more demographic characteristics of the user further comprises:

determining demographic characteristics for a substantial number of the identified facial images;

computing a distribution of the determined demographic characteristics; and

using the distribution as feature inputs to a classifier trained on historical data of demographic characteristics for users'"'"' social media photos and demographic data for the users.

13. The method of claim 1, wherein estimating the one or more demographic characteristics of the user comprises:

applying N distinct evaluation methods to the facial images, each method identifying a respective set of facial images that are estimated to be images of the user, wherein N is an integer greater than one;

combining the identified respective sets of facial images to form a candidate set of facial images; and

for each facial image in the candidate set;

forming a respective N-dimensional feature vector, wherein each dimension of the respective feature vector corresponds to a unique one of the evaluation methods, and the values in the respective feature vector indicate which of the evaluation methods estimate the respective facial image to be an image of the user; and

using the respective feature vector as input to a trained classifier to estimate whether the respective facial image is an image of the user;

wherein the selected subset of the facial images includes facial images determined by the trained classifier to be images of the user.

14. The method of claim 1, wherein estimating the one or more demographic characteristics of the user comprises:

applying a plurality of evaluation methods to the facial images, each method identifying a respective set of facial images that are estimated to be images of the user;

combining the identified respective sets of facial images to form a candidate set of facial images;

scoring each facial image in the candidate set using a weighted average of scores provided by the evaluation methods;

wherein the selected subset of the facial images includes facial images whose weighted average scores exceed a predefined threshold value.

15. The method of claim 1, wherein the one or more demographic characteristics include age and gender.

16. A computer system for identifying user demographic characteristics, comprising:

one or more processors;

memory; and

one or more programs stored in the memory, the one or more programs comprising instructions for;

acquiring a plurality of photos posted to a social media site by a user;

identifying a plurality of facial images in the acquired photos;

estimating one or more demographic characteristics of the user based on a selected subset of the facial images.

17. The computer system of claim 16, wherein the instructions for selecting the subset of facial images further comprise instructions for:

acquiring a profile photo of the user from the social media site;

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that most closely matches the acquired profile photo;

wherein the selected subset of facial images comprises the facial images in the selected cluster.

18. The computer system of claim 16, wherein the instructions for selecting the subset of facial images further comprise instructions for:

applying a clustering algorithm to group the plurality of facial images into a plurality of clusters; and

selecting a cluster of the plurality of clusters that has the highest cluster purity;

wherein the selected subset of facial images comprises the facial images in the selected cluster.

19. The computer system of claim 16, wherein the instructions for estimating the one or more demographic characteristics of the user further comprise instructions for:

determining demographic characteristics for a substantial number of the identified facial images;

computing a distribution of the determined demographic characteristics; and

using the distribution as feature inputs to a classifier trained on historical data of demographic characteristics for users'"'"' social media photos and demographic data for the users.

20. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer with a depth sensor, the one or more programs comprising instructions for:

acquiring a plurality of photos posted to a social media site by a user;

identifying a plurality of facial images in the acquired photos;

estimating one or more demographic characteristics of the user based on a selected subset of the facial images.