SPAM PROCESSING WITH CONTINUOUS MODEL TRAINING

• US 20170222960A1
• Filed: 02/01/2016
• Published: 08/03/2017
• Est. Priority Date: 02/01/2016
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: LinkedIn Corporation (Microsoft Corporation)
• Inventors: Agarwal, Siddharth, Gupta, Anindita, Sodhani, Siddharth, Gaurav, Nishant, Shacham, Dan, Raman, Shivakumar Edayathumangalam
• Application Number:
  US15/012,357
• Publication Number:
  US 20170222960A1
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

In various example embodiments, a system and method for generating a filtering spam content using machine learning are presented. One or more electronic content is received. The one or more electronic content is labeled as spam or not spam by a current spam filtering system. An associated accuracy score for each of the one or more labeled content is calculated. Potential errors in the one or more labeled content is identified based on the label of the one or more labeled content being inconsistent with information associated with the source of the one or more labeled content. The one or more labeled content with identified potential errors is sent for assessment. The one or more electronic content labeled as spam with an associated accuracy score within a predetermined range is filtered, excluding labeled content with identified potential errors.

Claims

1. A system comprising:

a processor, and a memory including instructions, which when executed by the processor, cause the processor to;

receive one or more electronic content label, by a current spam filtering system, the one or more electronic content as spam or not spam;

calculate an associated accuracy score for each of the one or more labeled content;

identify potential errors in the one or more labeled content based on the label of the one or more labeled content being inconsistent with information associated with the source of the one or more labeled content;

send the one or more labeled content with identified potential errors for assessment; and

filter the one or more electronic content labeled as spam with an associated accuracy score within a predetermined range, excluding labeled content with identified potential errors.

2. The system of claim 1, further comprising:

receive an assessment for the one or more labeled content with identified potential errors, the assessment comprising updating the label of the one or more labeled content with identified potential errors; and

filter the one or more updated labeled content being labeled as spam.

3. The system of claim 2, further comprising:

generate a general sampling data set based on randomly selecting a percentage of the one or more labeled content;

generate a positive sampling data set based on randomly selecting a percentage of the one or more electronic content labeled as spam; and

send the general sampling data set, the positive sampling data set, and the one or more electronic content with an associated accuracy score within a second predetermined range for assessment.

4. The system of claim 3, further comprising:

receive an assessment for the one or more labeled content with an associated accuracy score within a second predetermined range, the assessment comprising updating the label of the one or more labeled content.

5. The system of claim 4, further comprising:

receive electronic content being labeled as spam or not spam from individual users.

6. The system of claim 5, further comprising:

train a potential spam filtering system using the updated labeled content with potential errors, general sampling data set, positive sampling data set, updated labeled content with an associated accuracy score within a second predetermined range, and labeled content from individual users.

7. The system of claim 6, further comprising:

calculate a performance score for the potential spam filtering system using precision and recall measurements.

8. The system of claim 7, further comprising:

calculate a performance score for the current spam filtering system using precision and recall measurements;

compare the performance score of the current spam filtering system and the performance score of the potential spam filtering system; and

based on the performance score of the potential spam filtering system exceeding the performance score of the current spam filtering system, implement the potential spam filtering system for filtering incoming content.

9. A method comprising:

using one or more computer processors;

receiving one or more electronic content

labeling, by a current spam filtering system, the one or more electronic content as spam or not spam;

calculating an associated accuracy score for each of the one or more labeled content;

identifying potential errors in the one or more labeled content based on the label of the one or more labeled content being inconsistent with information associated with the source of the one or more labeled content;

sending the one or more labeled content with identified potential errors for assessment; and

filtering the one or more electronic content labeled as spam with an associated accuracy score within a predetermined range, excluding labeled content with identified potential errors.

10. The method of claim 9, further comprising:

receiving an assessment for the one or more labeled content with identified potential errors, the assessment comprising updating the label of the one or more labeled content with identified potential errors; and

filtering the one or more updated labeled content being labeled as spam.

11. The method of claim 10, further comprising:

generating a general sampling data set based on randomly selecting a percentage of the one or more labeled content;

generating a positive sampling data set based on randomly selecting a percentage of the one or more electronic content labeled as spam; and

sending the general sampling data set, the positive sampling data set, and the one or more electronic content with an associated accuracy score within a second predetermined range for assessment.

12. The method of claim 11, further comprising:

receiving an assessment for the one or more labeled content with an associated accuracy score within a second predetermined range, the assessment comprising updating the label of the one or more labeled content.

13. The method of claim 12, further comprising:

receiving electronic content being labeled as spam or not spam from individual users.

14. The method of claim 13, further comprising:

training a potential spam filtering system using the updated labeled content with potential errors, general sampling data set, positive sampling data set, updated labeled content with an associated accuracy score within a second predetermined range, and labeled content from individual users.

15. The method of claim 14, further comprising:

calculating a performance score for the potential spam filtering system using precision and recall measurements.

16. The method of claim 15, further comprising:

calculating a performance score for the current spam filtering system using precision and recall measurements;

comparing the performance score of the current spam filtering system and the performance score of the potential spam filtering system; and

based on the performance score of the potential spam filtering system exceeding the performance score of the current spam filtering system, implementing the potential spam filtering system for filtering incoming content.

17. A machine-readable medium not having any transitory signals and storing instructions that, when executed by at least one processor of a machine, cause the machine to perform operations comprising:

receiving one or more electronic content

labeling, by a current spam filtering system, the one or more electronic content as spam or not spam;

calculating an associated accuracy score for each of the one or more labeled content;

identifying potential errors in the one or more labeled content based on the label of the one or more labeled content being inconsistent with information associated with the source of the one or more labeled content;

sending the one or more labeled content with identified potential errors for assessment; and

filtering the one or more electronic content labeled as spam with an associated accuracy score within a predetermined range, excluding labeled content with identified potential errors.

18. The machine-readable medium of claim 17, wherein the operations further comprise:

receiving an assessment for the one or more labeled content with identified potential errors, the assessment comprising updating the label of the one or more labeled content with identified potential errors; and

filtering the one or more updated labeled content being labeled as spam.

19. The machine-readable medium of claim 18, wherein the operations further comprise:

generating a general sampling data set based on randomly selecting a percentage of the one or more labeled content;

generating a positive sampling data set based on randomly selecting a percentage of the one or more electronic content labeled as spam; and

sending the general sampling data set, the positive sampling data set, and the one or more electronic content with an associated accuracy score within a second predetermined range for assessment.

20. The machine-readable medium of claim 19, wherein the operations further comprise:

receiving an assessment for the one or more labeled content with an associated accuracy score within a second predetermined range, the assessment comprising updating the label of the one or more labeled content.