METHOD FOR REVEALING A TYPE OF DATA

• US 20140258185A1
• Filed: 03/07/2013
• Published: 09/11/2014
• Est. Priority Date: 03/07/2013
• Status: Abandoned Application

• Current Assignee: Raam Sharon
• Original Assignee: Raam Sharon
• Inventors: Sharon, Raam
• Application Number:
  US13/788,808
• Publication Number:
  US 20140258185A1
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A method for revealing a type of data is provided herein. The method is comprising

• • the following steps (i) receiving at least one training data set of objects to create a signature for that data type; (ii) creating the signature, by: (a) scanning all objects in
    • each training data set, to create a list of unique N-Grams of any size and their statistics; and (b) adding each N-Gram in the list in case it appears in a minimum of predefined threshold objects to a repository; (iii) receiving data of unknown type for examination; (iv) scanning said received data of unknown type to score each N-Gram
    • in the signature of each object type when it is found in the scanned data; and (v) determining the type of the unknown type data according to the signature of the object type that accumulated highest score by the N-Grams of the signature.

Claims

1. A method for revealing a type of data, the method comprising the steps of:

receiving at least one training data set of objects to create a signature for that data type;

creating the signature, by;

(i) scanning all objects in each training data set, to create a list of unique N-Grams of any size and their statistics; and

(ii) adding each N-Gram in the list in case it appears in a minimum of predefined threshold objects to a repository,

wherein each signature includes at least one N-Gram, wherein size of the at least one N-Gram is variable having a minimum limitation,

wherein a temp maximum value is specified for the size of the at least one N-Gram, and

wherein the temp maximum value for the size of the at least one N-Gram is increased after the creating of the signature when at least one identified N-Gram in the creation process reach the temp maximum value, and

receiving data of unknown type for examination;

scanning said received data of unknown type to score each N-Gram in the signature of each object type when it is found in the scanned data; and

determining the type of the unknown type data according to the signature of the object type that accumulated highest score by the N-Grams of the signature.

2. The method of claim 1, further comprising the step of defining a range of thresholds for maximum standard deviation and removing objects not in the range of threshold in the signature creating step from a training data set.

3. The method of claim 1, wherein in the creating of the signature, order of the objects that are being scanned is determined according to size of the objects.

4. The method of claim 1, wherein the scanning of:

(i) each object in the training data set to create a signature; and

(ii) said received data of unknown type, is applied on fractional sections of the object by using a predefined scanning pattern which defines the relative location and size of each section.

5. The method of claim 1, wherein an N-gram that appears in more than one type signature receives a low score.

6. The method of claim 1, wherein the creating of one type signature is from multiple signatures.

7. The method of claim 1, wherein settings are applied for specified object types.

8. The method of claim 1, wherein the creating of the signature is utilizing a data structure of a tree structure, wherein each node includes one byte and represents one potential N-Gram beginning at the root of the tree.

9. The method of claim 8, wherein through the signature creation processing new nodes are added to the tree in case the number of appearances of the corresponding N-gram in all scanned objects is with the limits of predefined threshold.

10. The method of claim 9 wherein after scanning each object, trimming nodes which their corresponding N-gram number of appearances in all scanned objects is not with the limits of predefined threshold.

11. The method of claim 1, wherein the preparing of the training data set is comprising the steps of:

creating one group of objects (i.e. centroid) for each object in the training data set and keeping it in the first data storage;

for each centroid pairs in the first data storage, calculating the signature of an intersection of the centroids,

repeatedly taking the following steps, while there is more than one centroid left in the first data storage;

(a) selecting an ordered pair of centroids from the first data storage yielding the best signature by intersection,

(b) merging the ordered pair of centroids to yield a new centroid;

(c) keeping the new centroid in the first data storage;

(d) copying the new centroid with its related signature to the second data storage;

(e) removing the ordered pair of centroids that was used to create the new centroid from the first data storage; and

(f) repeatedly calculating the signatures for the intersection of the new centroid and each of the rest of the centroids in the first data storage;

(g) selecting centroids from the second data storage, in reverse order of creation, as training data sets in case;

(i) the signature of the centroid meets predefined criteria; and

(ii) the centroid is not included in another centroid that was selected earlier.

12. The method of claim 1, wherein during the signature creation after scanning each object checking each N-Gram that was added to the repository and removing all N-Grams that appear in less objects in the training data set than the N-Grams in objects'"'"' threshold.

13. The method of claim 1, during the signature creation wherein for each object type, N-Grams are sorted by specified criteria for contribution to the type signature and N-Grams having low contribution to the type signature are removed.

14. The method of claim 1, further comprising the steps of scanning each training data set of each type signature using signatures of all types and, calculating an average score and standard deviation of all objects in the training data sets and saving it as an attribute of the signature used for scanning the training data set.

15. The method of claim 14, further comprising the step of:

calculating distance parameters of the score of examined object by signature from the average scores taking into account standard deviations received by scanning each type training data set by the alleged signature.

16. The method of claim 15, further comprising the step of:

summing up the distance parameters for each type training data set and selecting the type where the result of the respective type training data set is the lowest.

17. The method of claim 1, wherein the method is further recreating the signature after temp maximum value for the size of the at least one N-Gram is increased.