Machine-assisted publisher classification

• US 9,201,954 B1
• Filed: 03/01/2013
• Issued: 12/01/2015
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Azari, David Rezazadeh, Mahalingam, Divya, Bhadbhade, Tanvi Milind, Payet, Nadia
• Primary Examiner(s): THAI, HANH B
• Application Number:
  US13/782,535
• Time in Patent Office: 1,005 Days
• Field of Search: 707737-738, 707/740, 707/748
• US Class Current: 1/1
• CPC Class Codes: G06F 16/90 Details of database functions independent of the retrieved data types

Abstract

One or more computing systems can implement a classifier to classify content publishers as being likely to provide appropriate content or as being likely to provide inappropriate content. The classifier can gather information from previously classified publishers. The information from the previously classified publishers can used to train the classifier. Based on the training, the classifier can learn about traits, characteristics, and/or behavioral patterns, etc., associated with publishers that have been previously classified as being good as well as publishers previously classified as being bad. The classifier can then process information about an unclassified publisher to determine a classification for the unclassified publisher, as being good (and likely to provide appropriate content) or bad (and likely to provide inappropriate content).

Claims

1. A computer-implemented method for classifying content publishers, comprising:

receiving data associated with a plurality of classified content publishers, the data including training data and testing data;

training a classifier model using at least a portion of the training data;

defining a first category of the classifier model, the first category identifying allowable content publishers;

defining a second category of the classifier model, the second category identifying disallowable content publishers;

defining a third category of the classifier model, the third category identifying content publishers needing further review;

testing the classifier model using at least a portion of the testing data;

determining that a precision level associated with the classifier model has at least met a specified precision threshold;

receiving data associated with an unclassified content publisher;

calculating a score for the unclassified content publisher based at least in part on processing the data associated with the unclassified content publisher using the classifier model; and

classifying the unclassified content publisher as belonging to one of the first category, the second category, or the third category, the classifying being based upon the score for the unclassified content publisher.

2. The computer-implemented method of claim 1, further comprising:

publishing content provided by the allowable content publisher in a content library.

3. The computer-implemented method of claim 1, further comprising:

denying publication of content provided by the disallowable content publisher in the content library.

4. The computer-implemented method of claim 1, wherein each of the training data and the testing data comprises a publisher behavior data, associated with the plurality of classified content publishers, and wherein the data associated with the unclassified content publisher comprises publisher behavior data associated with the unclassified content publisher.

5. A computer-implemented method comprising:

receiving information about one or more classified publishers;

modifying a classifier model based at least in part on the information about the one or more classified publishers;

defining a first category of the classifier model, the first category identifying allowable content publishers;

defining a second category of the classifier model, the second category identifying disallowable content publishers;

defining a third category of the classifier model, the third category identifying content publishers needing further review;

determining that a quality level associated with the classifier model at least meets a specified quality threshold;

receiving information about at least one publisher, the at least one publisher being unclassified;

processing at least a portion of the information about the at least one publisher using the modified classifier model; and

classifying each of the at least one publisher as belonging to one of the first category, the second category or the third category, based at least in part on the processing at least the portion of the information by the modified classifier model.

6. The computer-implemented method of claim 5, wherein the classifying each of the at least one publisher comprises calculating a confidence score for the classifying of each of the at least one publisher based at least in part on the processing at least the portion of the information by the modified classifier model.

7. The computer-implemented method of claim 6, wherein the classifying each of the at least one publisher comprises classifying a first one of the at least one publisher as being good when the confidence score for the classifying of the first one of the at least one publisher at least meets a specified good publisher threshold and classifying a second one of the at least one publisher as being bad when the confidence score for the classifying of the second one of the at least one publisher at least meets a specified bad publisher threshold.

8. The computer-implemented method of claim 7, wherein the specified good specified good publisher threshold and the specified bad publisher threshold are within an allowable deviation from being equivalent.

9. The computer-implemented method of claim 7, wherein the classifying each of the at least one publisher further comprises classifying a third one of the at least one publisher as being neutral when the confidence score for the third one of the at least one publisher is below the specified good publisher threshold and below the specified bad publisher threshold.

10. The computer-implemented method of claim 7, wherein each of the at least one publisher classified as being good is associated with a whitelist and wherein each of the at least one publisher classified as being bad is associated with a blacklist.

11. The computer-implemented method of claim 7, wherein content provided by each of the at least one publisher classified as being good is allowed for publication in a content library and wherein content provided by each of the at least one publisher classified as being bad is denied for publication in the content library.

12. The computer-implemented method of claim 5, wherein the information about the one or more classified publishers comprises at least one of pre-publish data, publisher behavior data, or post-publish data associated with the one or more classified publishers, and wherein the information about the at least one publisher comprises at least one of pre-publish data or publisher behavior data associated with the at least one publisher.

13. The computer-implemented method of claim 5, further comprising:

receiving post-publish data associated with the at least one publisher subsequent to the classifying; and

classifying the at least one publisher based at least in part on the post-publish data.

14. The computer-implemented method of claim 13, wherein the post-publish data is associated with a user feedback of content provided by the at least one publisher.

15. The computer-implemented method of claim 5, wherein the classifier model is associated with at least one of a decision tree or a K-nearest-neighbor model.

16. The computer-implemented method of claim 5, wherein the quality level associated with the classifier model includes a precision level indicative of how accurate the classifier model is.

17. The computer-implemented method of claim 16, wherein the determining that the quality level associated with the classifier model meets the specified quality threshold further comprises:

testing the classifier model using at least in part the information about the one or more classified publishers, wherein the testing results in the precision level; and

determining that the precision level resulting from the testing at least meets a specified precision level.

18. The computer-implemented method of claim 5, wherein the information received about the one or more publishers includes classification information resulting from manual review of the one or more publishers.

19. A system comprising:

a processor; and

a memory device including instructions that, when executed by the processor, cause the system to;

receive information about one or more classified publishers;

modify a classifier model based at least in part on the information about the one or more classified publishers;

define a first category of the classifier model, the first category identifying allowable content providers;

define a second category of the classifier model, the second category identifying disallowable content providers;

define a third category of the classifier model, the third category identifying content publishers needing further review;

determine that a quality level associated with the classifier model at least meets a specified quality threshold;

receive information about at least one publisher, the at least one publisher being unclassified;

process at least a portion of the information about the at least one publisher using the modified classifier model; and

classify each of the at least one publisher as belonging to one of the first category, the second category or the third category, based at least in part on the processing at least the portion of the information by the modified classifier model.

20. The system of claim 19, wherein the classifying each of the at least one publisher comprises calculating a confidence score for the classifying of each of the at least one publisher based at least in part on the processing at least the portion of the information by the modified classifier model.

21. The system of claim 20, wherein the classifying each of the at least one publisher comprises classifying a first one of the at least one publisher as being good when the confidence score for the classifying of the first one of the at least one publisher at least meets a specified good publisher threshold and classifying a second one of the at least one publisher as being bad when the confidence score for the classifying of the second one of the at least one publisher at least meets a specified bad publisher threshold.

22. The system of claim 21, wherein each of the at least one publisher classified as being good is associated with a whitelist and wherein each of the at least one publisher classified as being bad is associated with a blacklist.

23. The system of claim 21, wherein content provided by each of the at least one publisher classified as being good is allowed for publication in a content library and wherein content provided by each of the at least one publisher classified as being bad is denied for publication in the content library.

24. A non-transitory computer-readable storage medium including instructions for identifying elements, the instructions when executed by a processor of a computing system causing the computing system to:

receive information about one or more classified publishers;

modify a classifier model based at least in part on the information about the one or more classified publishers;

define a first category of the classifier model for allowable content publishers;

define a second category of the classifier model for disallowable content providers;

define a third category of the classifier model, the third category identifying content publishers needing further review;

determine that a quality level associated with the classifier model at least meets a specified quality threshold;

receive information about at least one publisher, the at least one publisher being unclassified;

process at least a portion of the information about the at least one publisher using the modified classifier model; and

classify each of the at least one publisher as belonging to one of the first category, the second category, or the third category, based at least in part on the processing at least the portion of the information by the modified classifier model.

25. The non-transitory computer-readable storage medium of claim 24, wherein the classifier model is associated with at least one of a decision tree or a K-nearest-neighbor Mode.