Predicting household demographics based on image data

• US 10,277,714 B2
• Filed: 05/10/2017
• Issued: 04/30/2019
• Est. Priority Date: 05/10/2017
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Bullock, William, Xu, Liang, Zhou, Li
• Primary Examiner(s): Keehn, Richard G
• Application Number:
  US15/592,108
• Publication Number:
  US 20180332140A1
• Time in Patent Office: 720 Days
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/53 using third party service providers
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context

Abstract

An online system predicts household features of a user, e.g., household size and demographic composition, based on image data of the user, e.g., profile photos, photos posted by the user and photos posted by other users socially connected with the user, and textual data in the user'"'"'s profile that suggests relationships among individuals shown in the image data of the user. The online system applies one or more models trained using deep learning techniques to generate the predictions. For example, a trained image analysis model identifies each individual depicted in the photos of the user; a trained text analysis model derive household member relationship information from the user'"'"'s profile data and tags associated with the photos. The online system uses the predictions to build more information about the user and his/her household in the online system, and provide improved and targeted content delivery to the user and the user'"'"'s household.

Claims

1. A method comprising:

receiving one or more photos associated with a user of an online system, each of the received photos including a plurality of visual features associated with each individual in the received photo;

receiving textual information describing the user and the one or more photos associated with the user;

applying a trained image analysis model to the received one or more photos to identify the plurality of visual features associated with each individual in the received photos;

applying a trained text analysis model to the textual information describing the user and the one or more photos to generate a plurality of textual features related to household features of the user;

generating one or more predictions of the household features of the user based on the plurality of visual features associated with each individual in the received one or more photos and the textual features related to the household features of the user;

evaluating a prediction of a number of members associated with a household of the user based on information describing a plurality of household devices and corresponding household device users associated with the user; and

storing the generated predictions associated with a profile of the user within the online system, the stored predictions used in targeting content to the user and to other members of the user'"'"'s household.

2. The method of claim 1, wherein the household features of the user comprise size of the user'"'"'s household and demographic composition of the user'"'"'s household.

3. The method of claim 1, wherein storing the generated predictions associated with a profile of the user within the online system comprises:

supplementing the profile of the user within the online system with the predicted household features of the user; and

generating a comprehensive profile of the user based on the supplementing of the profile of the user.

4. The method of claim 3, further comprising:

sharing the comprehensive profile of the user within the online system with another online system where the user has another user profile.

5. The method of claim 1, further comprising:

selecting one or more content items for display to the user on the online system based on the predictions of the household features of the user; and

displaying the selected one or more content items to the user on the online system.

6. The method of claim 1, wherein the image analysis model is trained based on a machine learning scheme associated with image processing on a corpus of image training data.

7. The method of claim 1, wherein the text analysis model is trained based on a machine learning scheme associated with natural language processing on a corpus of textual training data.

8. The method of claim 1, further comprising:

receiving information describing a plurality of household devices and corresponding household device users associated with the user; and

applying the trained text analysis model to the received information to generate information describing the household features of the user.

9. The method of claim 8, wherein the information describing the household features of the user includes

an Internet Protocol address shared by the plurality of household devices of user, the method further comprising determining at least one of a size and a demographic composition of the household of the user based on the plurality of visual features, the plurality of textual features, and the internet protocol address shared by the plurality of household devices.

10. The method of claim 1, further comprising reevaluating a prediction of a number of members residing in a household of the user based on information describing a plurality of household devices and corresponding household device users associated with the user.

11. A non-transitory computer-readable medium comprising computer program instructions, the computer program instructions when executed by a processor of a computer device causes the processor to perform the steps including:

receiving one or more photos associated with a user of an online system, each of the received photos including a plurality of visual features associated with each individual in the received photo;

receiving textual information describing the user and the one or more photos associated with the user;

applying a trained image analysis model to the received one or more photos to identify the plurality of visual features associated with each individual in the received photos;

applying a trained text analysis model to the textual information describing the user and the one or more photos to generate a plurality of textual features related to household features of the user;

generating one or more predictions of the household features of the user based on the plurality of visual features associated with each individual in the received one or more photos and the textual features related to the household features of the user;

evaluating a prediction of a number of members associated with a household of the user based on information describing a plurality of household devices and corresponding household device users associated with the user; and

storing the generated predictions associated with a profile of the user within the online system, the stored predictions used in targeting content to the user and to other members of the user'"'"'s household.

12. The non-transitory computer-readable storage medium of claim 11, wherein the household features of the user comprise size of the user'"'"'s household and demographic composition of the user'"'"'s household.

13. The non-transitory computer-readable storage medium of claim 11, wherein storing the generated predictions associated with a profile of the user within the online system comprises:

supplementing the profile of the user within the online system with the predicted household features of the user; and

generating a comprehensive profile of the user based on the supplementing of the profile of the user.

14. The non-transitory computer-readable storage medium of claim 13, further comprising computer program instructions, the computer program instructions when executed by the processor of the computer device causes the processor to perform the steps including:

sharing the comprehensive profile of the user within the online system with another online system where the user has another user profile.

15. The non-transitory computer-readable storage medium of claim 11, further comprising computer program instructions, the computer program instructions when executed by the processor of the computer device causes the processor to perform the steps including:

selecting one or more content items for display to the user on the online system based on the predictions of the household features of the user; and

displaying the selected one or more content items to the user on the online system.

16. The non-transitory computer-readable storage medium of claim 11, wherein the image analysis model is trained based on a machine learning scheme associated with image processing on a corpus of image training data.

17. The non-transitory computer-readable storage medium of claim 11, wherein the text analysis model is trained based on a machine learning scheme associated with natural language processing on a corpus of textual training data.

18. The non-transitory computer-readable storage medium of claim 11, further comprising computer program instructions, the computer program instructions when executed by the processor of the computer device causes the processor to perform the steps including:

receiving information describing a plurality of household devices and corresponding household device users associated with the user; and

applying the trained text analysis model to the received information to generate information describing the household features of the user.

19. The non-transitory computer-readable storage medium of claim 18, wherein the information describing the household features of the user includes

an Internet Protocol address shared by the plurality of household devices of user, the non-transitory computer-readable storage medium further comprising instructions for determining a number of members belonging to the household of the user based upon the Internet Protocol address shared by the plurality of household devices.

20. The non-transitory computer-readable storage medium of claim 11, further comprising reevaluating a prediction of a number of members residing in a household of the user based on information describing a plurality of household devices and corresponding household device users associated with the user.