Iterative classifier training on online social networks

• US 10,552,759 B2
• Filed: 12/01/2014
• Issued: 02/04/2020
• Est. Priority Date: 12/01/2014
• Status: Expired due to Fees

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Rich, Mark Andrew
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Zidanic, Michael
• Application Number:
  US14/556,854
• Publication Number:
  US 20160155063A1
• Time in Patent Office: 1,891 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

In one embodiment, a method includes accessing a first set of objects associated with an online social network, each object being associated with one or more comments. The method also includes generating a second set of objects from the first set of objects by applying a first filtering criteria to the first set of objects and scoring each object in the second set of objects based on the comments associated with each object. The method further includes generating a training set of objects from the second set of objects by selecting each object from the second set of objects having a score greater than a first threshold score, each object in the training set being associated with a first object-classification. The method further includes determining an object-classifier algorithm for the first object-classification, the object-classifier algorithm being determined through an iterative training process performed one or more times.

Claims

1. A method comprising, by one or more computing devices associated with an online social network:

accessing, by one or more of the computing devices, a first set of objects posted to the online social network by a plurality of client systems associated with a plurality of users of the online social network, the objects being stored in one or more data stores associated with the online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

generating, by one or more of the computing devices, a second set of objects by adding one or more objects in the first set of objects meeting a first filtering criteria to the second set;

scoring, by one or more of the computing devices, each object in the second set of objects by comparing the comments associated with each object to a pre-determined set of terms associated with a first object-classification;

generating, by one or more of the computing devices, a training set of objects from the second set of objects by selecting each object from the second set of objects having a score greater than a first threshold score, each object in the training set being pre-classified with the first object-classification;

generating, by one or more of the computing devices, an object-classifier algorithm for the first object-classification, the object-classifier algorithm comprising one or more features each comprising a text expression and stored in association with a feature score indicating a level of correlation between the corresponding feature and the first object-classification and being determined through an iterative training process performed one or more times, each iteration of the iterative training process comprising;

training an initial object-classifier algorithm based on the comments associated with the objects in the training set of objects;

accessing a third set of objects posted to the online social network by a plurality of client systems associated with a plurality of users of the online social network, the objects being stored in one or more data stores associated with the online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

classifying, using the initial object-classifier algorithm, each object in the third set of objects based on an analysis of the comments associated with each object, one or more of the objects in the third set of objects being classified with the first object-classification, wherein the analysis comprises comparing the comments with the features;

training a revised object-classifier algorithm based on the comments associated with the objects in the third set of objects having the first object-classification, wherein the training comprises revising one or more features of the initial object-classifier algorithm;

accessing a fourth set of objects stored in one or more data stores associated with the online social network, the fourth set of objects being generated by applying a second filtering criteria to a fifth set of objects posted to the online social network by a plurality of client systems associated with a plurality of users, the objects being stored in one or more data stores associated with the online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

classifying, using the revised object-classifier algorithm, each object in the fourth set of objects based on an analysis of the comments associated with each object, one or more objects in the fourth set of objects being classified with the first object-classification, wherein the analysis comprises comparing the comments with the modified features;

determining a level of precision associated with the object-classifier algorithm; and

if the determined level of precision does not satisfy a specified threshold level of precision;

then, generating a sixth set of objects from the fourth set of objects by selecting each object from the fourth set of objects having a score greater than a second threshold score, each object in the sixth set of objects being classified with the first object-classification, wherein the sixth set of objects is to be used as the training set in a next iteration of the iterative training process;

else, terminating the iterative training process and storing the revised object-classifier algorithm; and

classifying, using the generated object-classifier algorithm, a particular object posted by a client system associated with a particular user, wherein the particular object is classified with the first object-classification; and

determining a status of the particular user posting the particular object based at least in part on the classification of the particular object.

2. The method of claim 1, further comprising:

accessing a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, each node corresponding to an object associated with the online social network.

3. The method of claim 2, wherein, for each object in the first set of objects, the object is a post by a first user of the online social network corresponding to a first node of the plurality of nodes, and wherein each comment associated with the object is a comment on the post by a second user of the online social network, each second user corresponding to a second node, each second node being within a single degree of separation from the first node in the social graph.

4. The method of claim 1, wherein the objects of the first set of objects are each of an object-type selected from a group consisting of:

photos, posts, pages, applications, events, locations, or user groups.

5. The method of claim 1, wherein:

adding one or more objects in the first set of objects meeting a first filtering criteria to the second set comprises comparing metadata associated with each object in the first set of objects to a set of filtering criteria.

6. The method of claim 5, wherein the metadata comprises one or more of:

social-graph information associated with the object, other objects associated with the object, a language associated with the object, a time or date associated with the object, or any combination thereof.

7. The method of claim 6, wherein the social-graph information comprises an affinity coefficient between first and second users of the online social network, the first user being associated with an object in the first set of objects.

8. The method of claim 1, wherein scoring each object in the second set of objects comprises determining a score for each comment associated with the object by comparing the comment to the pre-determined set of terms associated with the first object-classification.

9. The method of claim 1, wherein scoring each object in the second set of objects comprises:

determining, for each object in the second set of objects, a score for each comment associated with the object; and

combining, for each object in the second set of objects, the scores for the comments to produce an object score.

10. The method of claim 9, wherein the object score is an average of the scores for the comments.

11. The method of claim 1, wherein the first threshold score is greater than 90% of the scores of the objects in the second set of objects.

12. The method of claim 1, wherein training the initial object-classifier algorithm comprises:

analyzing the comments associated with the objects in the training set to determine an initial set of features;

scoring each feature in the initial set of features based on a frequency of occurrence of the feature in the comments associated with the objects in the training set; and

modifying the initial object-classifier algorithm to associate at least in part the first object-classification with one or more of the features in the initial set of features having a score greater than a threshold feature score.

13. The method of claim 1, wherein classifying each object in the third set of objects comprises comparing the comments associated with an object in the third set to features associated with the initial object-classifier algorithm to determine whether the object is classified with the first object-classification.

14. The method of claim 1, wherein training the revised object-classifier algorithm comprises:

analyzing the comments associated with the objects in the third set of objects having the first object-classification to determine a set of revised features;

scoring each revised feature in the set of revised features based on a frequency of occurrence of the feature in the comments associated with the objects in the third set of objects having the first object-classification; and

modifying the revised object-classifier algorithm to associate at least in part the first object-classification with one or more of the features in the set of revised features having a score greater than a threshold feature score.

15. The method of claim 1, wherein classifying each object in the fourth set of objects comprises comparing the comments associated with an object in the fourth set to features associated with the revised object-classifier algorithm to determine whether the object is classified with the first object-classification.

16. The method of claim 1, wherein training the initial object-classifier algorithm is further based on the revised object-classifier algorithm trained in a prior iteration of the iterative training process.

17. The method of claim 1, wherein one or more of the first, third, or fifth sets of objects each comprise objects that are randomly selected from objects associated with the online social network.

18. The method of claim 1, wherein the first, third, and fifth sets of objects each comprise greater than or equal to one million objects.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access, by one or more of the computing devices, a first set of objects posted to the online social network by a plurality of client systems associated with a plurality of users of the online social network, the objects being stored in one or more data stores associated with an online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

generate, by one or more of the computing devices, a second set of objects by adding one or more objects in the first set of objects meeting a first filtering criteria to the second set;

score, by one or more of the computing devices, each object in the second set of objects by comparing the comments associated with each object to a pre-determined set of terms associated with a first object-classification;

generate, by one or more of the computing devices, a training set of objects from the second set of objects by selecting each object from the second set of objects having a score greater than a first threshold score, each object in the training set being pre-classified with the first object-classification;

generate, by one or more of the computing devices, an object-classifier algorithm for the first object-classification, the object-classifier algorithm comprising one or more features each comprising a text expression and stored in association with a feature score indicating a level of correlation between the corresponding feature and the first object-classification and being determined through an iterative training process performed one or more times, each iteration of the iterative training process comprising;

train an initial object-classifier algorithm based on the comments associated with the objects in the training set of objects;

access a third set of objects posted to the online social network by a plurality of client systems associated with a plurality of users of the online social network, the objects being stored in one or more data stores associated with the online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

classify, using the initial object-classifier algorithm, each object in the third set of objects based on an analysis of the comments associated with each object, one or more of the objects in the third set of objects being classified with the first object-classification, wherein the analysis comprises comparing the comments with the features;

train a revised object-classifier algorithm based on the comments associated with the objects in the third set of objects having the first object-classification, wherein the training comprises revising one or more features of the initial object-classifier algorithm;

access a fourth set of objects stored in one or more data stores associated with the online social network, the fourth set of objects being generated by applying a second filtering criteria to a fifth set of objects posted to the online social network by a plurality of client systems associated with a plurality of users, the objects being stored in one or more data stores associated with the online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

classify, using the revised object-classifier algorithm, each object in the fourth set of objects based on an analysis of the comments associated with each object, one or more objects in the fourth set of objects being classified with the first object-classification, wherein the analysis comprises comparing the comments with the modified features;

determine a level of precision associated with the object-classifier algorithm; and

if the determined level of precision does not satisfy a specified threshold level of precision;

then, generate a sixth set of objects from the fourth set of objects by selecting each object from the fourth set of objects having a score greater than a second threshold score, each object in the sixth set of objects being classified with the first object-classification, wherein the sixth set of objects is to be used as the training set in a next iteration of the iterative training process;

else, terminate the iterative training process and storing the revised object-classifier algorithm; and

classify, using the generated object-classifier algorithm, a particular object posted by a client system associated with a particular user, wherein the particular object is classified with the first object-classification; and

determine a status of the particular user posting the particular object based at least in part on the classification of the particular object.

20. A system comprising:

one or more processors; and

a memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to;

access, by one or more of the computing devices, a first set of objects posted to the online social network by a plurality of client systems associated with a plurality of users of the online social network, the objects being stored in one or more data stores associated with an online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

generate, by one or more of the computing devices, a second set of objects by adding one or more objects in the first set of objects meeting a first filtering criteria to the second set;

score, by one or more of the computing devices, each object in the second set of objects by comparing the comments associated with each object to a pre-determined set of terms associated with a first object-classification;

generate, by one or more of the computing devices, a training set of objects from the second set of objects by selecting each object from the second set of objects having a score greater than a first threshold score, each object in the training set being pre-classified with the first object-classification;

generate, by one or more of the computing devices, an object-classifier algorithm for the first object-classification, the object-classifier algorithm comprising one or more features each comprising a text expression and stored in association with a feature score indicating a level of correlation between the corresponding feature and the first object-classification and being determined through an iterative training process performed one or more times, each iteration of the iterative training process comprising;

train an initial object-classifier algorithm based on the comments associated with the objects in the training set of objects;

access a third set of objects posted to the online social network by a plurality of client systems associated with a plurality of users of the online social network, the objects being stored in one or more data stores associated with the online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

classify, using the initial object-classifier algorithm, each object in the third set of objects based on an analysis of the comments associated with each object, one or more of the objects in the third set of objects being classified with the first object-classification, wherein the analysis comprises comparing the comments with the features;

train a revised object-classifier algorithm based on the comments associated with the objects in the third set of objects having the first object-classification, wherein the training comprises revising one or more features of the initial object-classifier algorithm;

access a fourth set of objects stored in one or more data stores associated with the online social network, the fourth set of objects being generated by applying a second filtering criteria to a fifth set of objects posted to the online social network by a plurality of client systems associated with a plurality of users, the objects being stored in one or more data stores associated with the online social network, each object being associated with one or more comments, wherein each comment is posted to the online social network with respect to the corresponding object by a client system associated with a user of the online social network;

classify, using the revised object-classifier algorithm, each object in the fourth set of objects based on an analysis of the comments associated with each object, one or more objects in the fourth set of objects being classified with the first object-classification, wherein the analysis comprises comparing the comments with the modified features;

determine a level of precision associated with the object-classifier algorithm; and

if the determined level of precision does not satisfy a specified threshold level of precision;

then, generate a sixth set of objects from the fourth set of objects by selecting each object from the fourth set of objects having a score greater than a second threshold score, each object in the sixth set of objects being classified with the first object-classification, wherein the sixth set of objects is to be used as the training set in a next iteration of the iterative training process;

else, terminate the iterative training process and storing the revised object-classifier algorithm; and

classify, using the generated object-classifier algorithm, a particular object posted by a client system associated with a particular user, wherein the particular object is classified with the first object-classification; and

determine a status of the particular user posting the particular object based at least in part on the classification of the particular object.