Method, apparatus, and system for clustering and classification

• US 8,010,466 B2
• Filed: 06/10/2009
• Issued: 08/30/2011
• Est. Priority Date: 11/04/2004
• Status: Active Grant

• Current Assignee: Sysxnet Ltd.
• Original Assignee: TW Vericept Corp. (Singapore Telecommunications Limited)
• Inventors: Patinkin, Seth
• Primary Examiner(s): HOLMES, MICHAEL B
• Application Number:
  US12/482,183
• Publication Number:
  US 20100017487A1
• Time in Patent Office: 811 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

The invention provides a method, apparatus and system for classification and clustering electronic data streams such as email, images and sound files for identification, sorting and efficient storage. The method further utilizes learning machines in combination with hashing schemes to cluster and classify documents. In one embodiment hash apparatuses and methods taxonomize clusters. In yet another embodiment, clusters of documents utilize geometric hash to contain the documents in a data corpus without the overhead of search and storage.

Claims

1. A computer method for detecting a document having identified attributes comprising:

(a) converting a binary coded message into numeric values;

(b) computing a hashing vector based upon the numeric values provided to a mathematical function;

(c) comparing a difference between a hashing vector and a stored vector.

2. The computer method as in claim 1 further comprising:

(d) appending a header to the message that indicates said differences.

3. The computer method as in claim 2 wherein the numeric values consist of at least two bytes.

4. A computer method for comparing a plurality of documents comprising the steps of:

(a) receiving a first document having coded elements into a random access memory;

(b) converting the coded elements into a number between two limits;

(c) loading a data register serially from the random access memory with at least two adjacent data elements from the document;

(d) computing a vector corresponding to at least two associated adjacent data elements and a uniform filter;

(e) loading the one data register serially from a means for storing with a next adjacent data element from the document;

(f) computing a vector corresponding to at least two associated adjacent data elements and a uniform filter;

(g) repeating the steps (e) through (f) until elements from the first document have a corresponding vector;

(h) summing each associated vector element to form an associated hashing vector elements; and

(i) comparing the hashing vector with an archive of hashing vectors to determine similarity.

5. A computer method for detecting transmission of a cluster of email, comprising the steps of:

(a) receiving one or more email messages;

(b) generating hash values, based on one or more portions of the plurality of email messages;

(c) generating an associated bit mask value based on one or more portions of the plurality of email messages;

(d) determining whether the generated hash values and the associated bit mask values match corresponding hash values and associated bit mask values related to one or more prior email messages in the cluster.

6. The method of claim 5, further comprising generating a salience score for the plurality of email messages based on a result of the determination of whether the generated hash values and the associated bit mask values match corresponding hash values and the associated bit mask values related to prior email messages of the class.

7. The method of claim 6 comprising the further step of taking remedial action when the one of the plurality of email messages is a potentially unwanted email.

8. A system for detecting transmission of potentially unwanted e-mails, comprising:

means for observing a plurality of e-mails;

a means for creating a hashing vector for one or more portions of the plurality of emails,

a means to generate hash values, a means to generate bit masks and a means for determining whether the generated hash values and associated bit mask values match hash values and associated bit mask values related to prior emails; and

a means for determining that the plurality of emails are potentially unwanted e-mails.

9. A computer method for improving the accuracy of text classification by operating within an unsure region comprising the steps of:

utilizing a K-NN processor to determine the document having the greatest similarity to the text.

10. The computer method in claim 9 for improving the accuracy of text classification by operating within an unsure region further comprising the step of:

utilizing a hash generating means to determine the cluster having the greatest similarity to the text.

11. The computer method in claim 9 improving the accuracy of text classification by operating within an unsure region further comprising the step of:

utilizing a stackable hash process to determine the cluster having the greatest similarity to the text.

12. A computer method for storing email messages comprising the steps of utilizing a stackable hash process to determine the cluster wherein said cluster determines a delta-storage of the email.

13. The computer method in claim 12 for storing email messages further comprising the steps of utilizing a uniform filter process to determine the cluster wherein said cluster enables a delta-storage of the email.

14. A method for creating an accumulation of documents stored as a cluster by utilizing a process to create a hashing vector to determine whether to add a document to a cluster.

15. The computer method for creating an accumulation of documents stored as a cluster as in claim 14, further comprising one or more steps selected from the group consisting of

(i) adjusting by an aging function,

(ii) utilizing a mask to identify document clusters,

(iii) auto-labeling emails according to instantiation of labels on the basis of whether these labels are pre-defined or user-defined,

(iv) determining whether text inputs are related or not related, by assigning a score relating to this determination,

(v) further comprising forming email clustering and displaying in graphical form long hash and small hash,

(vi) forming email clustering utilizing a small hash threshold, (vii) forming email clustering utilizing small hash length, (viii) forming email clustering utilizing small hash average,

(ix) ascribing retention rates to clusters of email based upon clustering,

(x) creating hash methods to minimize computation time for clustering and maximize accuracy of classification,

(xi) using a clustering method for the purpose of identifying emails containing one of a malicious code, a phishing, a virus or a worm,

(xii) using a clustering method for the purpose of identifying an email for purposes of delta storage,

(xiii) using a mask on a time frame for the purpose of identifying a cluster member,

(xiv) using an aging function to identify a cluster member,

(xv) using a cluster to populate a central repository of hash data,

(xvi) populating a hash table for anonymous sharing of information between organization,.

(xvii) selecting a hash function with small collisions,

(xviii) sorting by class a indexing technique to route queries,

(xix) using classes to route queries as part of a cluster,

(xx) choosing a prime number P for the purpose of determining equivalence classes (mod P) in a stackable hash method, and

(xxi) using a brightness and an aging function in a hash method for creating a cluster.

16. A computer method for creating an accumulation of documents stored as a set of clusters comprising the step of utilizing a stackable ash to determine whether to add a document to the set of clusters.

17. The computer method for creating an accumulation of documents stored as a set of clusters as in claim 16, adjusted by an aging function.

18. The computer method for creating an accumulation of documents stored as a set of clusters as in claim 16, utilizing a mask to identify document clusters.