Method for classifying unknown electronic documents based upon at least one classificaton

• US 9,171,070 B2
• Filed: 09/13/2012
• Issued: 10/27/2015
• Est. Priority Date: 02/11/2004
• Status: Expired due to Fees

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Alspector, Joshua, Chowdhury, Abdur R., Kolcz, Aleksander
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Khakhar, Nirav K
• Application Number:
  US13/612,841
• Publication Number:
  US 20130173562A1
• Time in Patent Office: 1,139 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

A classification system includes a signature-based duplicate detector and an inductive classifier that share attribute information. To perform the duplicate detection and the classification, the duplicate detector and inductive classifier are first initialized by generating a lexicon of attributes for the duplicate detector and a classification model for the classifier. To develop a classification model, a training set of documents of known class are used by the classifier to determine the attributes of the documents that are most useful in classifying an unknown document. The model is developed from these attributes. Attribute information containing the attributes determined by the classifier is then passed to the duplicate detector and the duplicate detector uses the attribute information to generate the lexicon of attributes.

Claims

1. A method comprising:

receiving an electronic document;

analyzing, using at least one processor, the electronic document to extract a number of first attributes from the electronic document, the number of first attributes selected from a first attribute set;

obtaining a first classification output for the electronic document based upon the number of first attributes extracted from the electronic document;

if the first classification output for the electronic document is above a first threshold, withholding delivery of the electronic document;

if the first classification output for the electronic document is below the first threshold, analyzing the electronic document to extract a number of second attributes from the electronic document, the number of second attributes selected from a second attribute set, and obtaining a second classification output for the electronic document based upon the number of second attributes extracted from the electronic document;

if the second classification output for the electronic document is above a second threshold, withholding delivery of the electronic document; and

if the second classification output for the electronic document is below the second threshold, delivering the electronic document.

2. The method of claim 1, further comprising receiving a plurality of known spam electronic documents.

3. The method of claim 2, further comprising:

analyzing each of the plurality of known spam electronic documents; and

determining a plurality of unique attributes for each of the known spam electronic documents.

4. The method of claim 3, wherein the plurality of unique attributes comprise at least one of textual identifiers, hypertext markup language identifiers, or non-textual identifiers.

5. The method of claim 3, further comprising creating the first attribute set and the second attribute set based on the plurality of unique attributes determined from the plurality of known spam electronic documents.

6. The method of claim 3, further comprising associating a ranking with each of the plurality of unique attributes based on how indicative each of the plurality of unique attributes is of the likelihood of an electronic document being spam.

7. The method of claim 6, further comprising obtaining the first classification output and the second classification output for the electronic document based on the ranking associated with each of the number of attributes present in the electronic document.

8. The method of claim 1, further comprising:

determining a weight associated with each of the number of first attributes;

determining a weight associated with each of the number of second attributes;

generating the first classification output for the electronic document based at least in part on the weight associated with each of the number of first attributes; and

generating the second classification output for the electronic document based at least in part on the weight associated with each of the number of second attributes.

9. The method of claim 1, further comprising enabling a user to modify the first classification output and the second classification output for the electronic document.

10. The method of claim 1, wherein the electronic document comprises a video file.

11. A method comprising:

generating, using at least one processor, a first attribute set and a second attribute set for use in classifying electronic documents;

receiving an electronic document having an unknown query signature;

analyzing the electronic document to determine whether the electronic document contains one or more attributes selected from the first attribute set;

making a first determination as to whether the electronic document contains a number of attributes selected from the first attribute set above a first threshold;

classifying the electronic document based on the first determination;

if the electronic document does not contain a number of attributes selected from the first attribute set above the first threshold, analyzing the electronic document to determine whether the electronic document contains one or more attributes selected from the second attribute set;

making a second determination as to whether the electronic document contains a number of attributes selected from the second attribute set above the second threshold; and

classifying the electronic document based on the second determination.

12. The method of claim 11, further comprising assigning a query signature to the electronic document based on the classification of the electronic document.

13. The method of claim 12, further comprising:

comparing the query signature of the electronic document with one or more query signatures of known spam electronic documents; and

forwarding the electronic document based on whether the query signature of the electronic document matches the one or more query signatures of the known spam electronic documents.

14. The method of claim 11, further comprising associating a weight with each attribute in the first attribute set and with each attribute in the second attribute set, wherein each weight is indicative of the likelihood that an electronic document that contains the attribute is spam.

15. The method of claim 14, further comprising modifying the classification of the electronic document based on the weight of each attribute contained in the electronic document.

16. The method of claim 11, wherein the electronic document comprises an audio file.

17. The method of claim 11, further comprising enabling a user to modify the classification of the electronic document.

18. A system comprising:

at least one processor; and

at least one non-transitory computer readable medium storing instructions thereon that, when executed by at least one process, cause the system to;

receive an electronic document;

analyze the electronic document to extract a first number of attributes from the electronic document, the number of first attributes selected from a first attribute set;

obtain a first classification output for the electronic document based upon the number of first attributes extracted from the electronic document;

if the first classification output for the electronic document is above a first threshold, withhold delivery of the electronic document based;

if the first classification output for the electronic document is below the first threshold, analyze the electronic document to extract a number of second attributes from the electronic document, the number of second attributes selected from a second attribute set, and obtain a second classification output for the electronic document based upon the number of second attributes extracted from the electronic document;

if the second classification output for the electronic document is above a second threshold, withhold delivery of the electronic document; and

if the second classification output for the electronic document is below the second threshold, deliver the electronic document.

19. The system of claim 18, wherein the at least one processor is further caused to:

receive a plurality of known spam electronic documents; and

determine a plurality of unique attributes from the plurality of known spam electronic documents.

20. The system of claim 19, wherein the first attribute set and the second attribute set each comprise at least a portion of the plurality of unique attributes determined from the plurality of known spam electronic documents.

21. The system of claim 18, wherein the at least one processor is further caused to enable a user to modify the first classification output and the second classification output for the electronic document.