Handling Noise in Training Data for Malware Detection

• US 20130097704A1
• Filed: 10/13/2012
• Published: 04/18/2013
• Est. Priority Date: 10/13/2011
• Status: Abandoned Application

• Current Assignee: Bitdefender IPR Management Limited (Bitdefender LLC)
• Original Assignee: Bitdefender IPR Management Limited (Bitdefender LLC)
• Inventors: GAVRILUT, Dragos, BARAT, Marius, CIORTUZ, Liviu V.
• Application Number:
  US13/651,409
• Publication Number:
  US 20130097704A1
• Time in Patent Office: Days
• US Class Current: 726/23
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06N 3/02 Neural networks

Abstract

Described systems and methods allow the reduction of noise found in a corpus used for training automatic classifiers for anti-malware applications. Some embodiments target pairs of records, which have opposing labels, e.g. one record labeled as clean/benign, while the other labeled as malware. When two such records are found to be similar, they are identified as noise and are either discarded from the corpus, or relabeled. Two records may be deemed similar when, in a simple case, they share a majority of features, or, in a more sophisticated case, they are sufficiently close in a feature space according to some distance measure.

Claims

1. A computer system comprising at least one processor configured to form a set of noise detectors, each noise detector of the set of noise detectors configured to de-noise a corpus of records, wherein the corpus is pre-classified into a subset of clean records and a subset of malware records prior to de-noising, and wherein de-noising the corpus comprises:

selecting a first record and a second record from the corpus, the first record being labeled as clean and the second record being labeled as malware;

in response to selecting the first and second records, determining whether the first and second records are similar according to a set of features; and

in response, when the first and second records are similar, determine that the first and second records are noise.

2. The computer system of claim 1, further configured, in response to determining that the first and second records are noise, to remove the first and second records from the corpus.

3. The computer system of claim 1, wherein the at least one processor is further configured, in response to determining that the first and second records are noise, to re-label the second record as clean.

4. The computer system of claim 1, wherein determining whether the first and second records are similar comprises:

determining a distance in a feature hyperspace, the distance separating a first feature vector from a second feature vector, the first feature vector determined for the first record and the second feature vector determined for the second record;

in response to determining the distance, comparing the distance to a pre-determined threshold; and

determining whether the first and second records are similar according to a result of the comparison.

5. The computer system of claim 4, wherein the distance is determined according to w|F¹−

F²|, wherein F¹ is an element of the first feature vector, F¹ indicating whether the first record has a selected feature, wherein F² is an element of the second feature vector, F² indicating whether the second record has the selected feature, and wherein w denotes a weight of the selected feature, the weight determined according to a count of clean-labeled records of the corpus having the selected feature, and further according to a count of malware-labeled records of the corpus having the selected feature.

6. The computer system of claim 4, further configured to determine a first set of values of a selected feature and a second set of values of the selected feature, wherein each value of the first set of values is determined for a malware record of the corpus and indicates whether the malware record has the selected feature, wherein each value of the second set of values is determined for a clean record of the corpus and indicates whether the clean record has the selected feature, and wherein the distance is determined according to

7. The computer system of claim 1, further configured to:

determine a hyperplane dividing the corpus into a clean region of a feature hyperspace and a malware region of the feature hyperspace;

in response to determining the hyperplane, select a third record of the corpus, the third record being labeled as malware, the third record being located in the clean region of feature hyperspace;

determine whether the third record is close to the hyperplane according to a comparison between a distance separating the third record from the hyperplane and a predetermined threshold; and

in response, when the third record is close to the hyperplane, determine that the third record is noise.

8. The computer system of claim 1, further configured to:

determine a hyperplane dividing the corpus into a clean region of a feature hyperspace and a malware region of the feature hyperspace;

in response to determining the hyperplane, select a third record of the corpus, the third record being labeled as clean, the third record being located in the malware region of feature hyperspace;

determine whether the third record is close to the hyperplane according to a comparison between a distance separating the third record from the hyperplane and a predetermined threshold; and

in response, when the third record is close to the hyperplane, determine that the third record is noise.

9. The computer system of claim 1, further configured to form a filter training engine connected to the set of noise detectors and configured to train an automated classifier to discriminate between malware and clean records according to an output of the set of noise detectors.

10. The computer system of claim 1, wherein de-noising the corpus further comprises:

before selecting the first and second records, dividing the corpus of records into a plurality of clusters, wherein all members of a cluster of the plurality of clusters share a selected set of features; and

in response to dividing the corpus into the plurality of clusters, selecting the first and second records from a first cluster of the plurality of clusters.

11. The computer system of claim 10, wherein dividing the corpus into the plurality of clusters comprises selecting a feature i of the selected set of features according to a proximity between a first count and a first reference, and further according to a proximity between a second count and a second reference, wherein the first count is a count of malware-labeled records of the corpus having feature i, wherein the second count is a count of clean-labeled records of the corpus having feature i, wherein the first reference is equal to half of the cardinality of the subset of malware records, and wherein the second reference is equal to half of the cardinality of the subset of clean records.

12. The computer system of claim 11, wherein dividing the corpus into the plurality of clusters comprises selecting a feature i of the selected set of record features according to:

|freq_i^malicious−

0.5|+|freq_i^clean−

0.5|,

wherein freq_i^malicious denotes a frequency of appearance of feature i within the subset of malware records of the corpus, and wherein freq_i^clean denotes a frequency of appearance of feature i within the subset of clean records of the corpus.

13. The computer system of claim 10, wherein a first noise detector of the plurality of noise detectors executes on a first processor of the computer system, wherein a second noise detector of the plurality of noise detectors executes on a second processor of the computer system, the second processor distinct from the first processor, wherein the first noise detector is configured to de-noise the first cluster, and wherein the second noise detector is configured to de-noise a second cluster of the plurality of clusters, the second cluster distinct from the first cluster.

14. The computer system of claim 10, wherein the distance is determined according to

15. A method comprising:

employing at least one processor of a computer system to select a first record and a second record from a corpus, wherein the corpus is pre-classified into a subset of clean records and a subset of malware records prior to selecting the first and second records, and wherein the first record is labeled as clean and the second record is labeled as malware;

in response to selecting the first and second records, employing the at least one processor to determine whether the first and second records are similar according to a set of features; and

in response, when the first and second records are similar, employing the at least one processor to determine that the first and second records are noise.

16. The method of claim 15, further comprising, in response to determining that the first and second records are noise, removing the first and second records from the corpus.

17. The method of claim 15, further comprising, in response to determining that the first and second records are noise, re-labeling the second record as clean.

18. The method of claim 15, wherein determining whether the first and second records are similar comprises:

employing the at least one processor to determine a distance in a feature hyperspace, the distance separating a first feature vector from a second feature vector, the first feature vector determined for the first record and the second feature vector determined for the second record;

in response to determining the distance, employing the at least one processor to compare the distance to a pre-determined threshold; and

employing the at least one processor to determine whether the first and second records are similar according to a result of the comparison.

19. The method of claim 18, comprising determining the distance according to w|F¹−

F²|, wherein F¹ is an element of the first feature vector, F¹ indicating whether the first record has a selected feature, wherein F² is an element of the second feature vector, F² indicating whether the second record has the selected feature, and wherein w denotes a weight of the selected feature, the weight determined according to a count of clean records of the corpus having the selected feature, and further according to a count of malware records of the corpus having the selected feature.

20. The method of claim 18, further comprising determining a first set of values of a selected feature and a second set of values of the selected feature, wherein each value of the first set of values is determined for a malware record of the corpus, indicating whether the malware record has the selected feature, wherein each value of the second set of values is determined for a clean record of the corpus, indicating whether the clean record has the selected feature, and wherein the distance is determined according to

21. The method of claim 15, further comprising:

employing the at least one processor to determine a hyperplane dividing the corpus into a clean region of a feature hyperspace and a malware region of the feature hyperspace;

in response to determining the hyperplane, employing the at least one processor to select a third record of the corpus, the third record being labeled as malware, the third record being located in the clean region of feature hyperspace;

employing the at least one processor to determine whether the third record is close to the hyperplane according to a comparison between a distance separating the third record from the hyperplane and a predetermined threshold; and

in response, when the third record is close to the hyperplane, employing the at least one processor to determine that the third record is noise.

22. The method of claim 15, further comprising:

employing the at least one processor to determine a hyperplane dividing the corpus into a clean region of a feature hyperspace and a malware region of the feature hyperspace;

in response to determining the hyperplane, employing the at least one processor to select a third record of the corpus, the third record being labeled as clean, the third record being located in the malware region of feature hyperspace;

employing the at least one processor to determine whether the third record is close to the hyperplane according to a comparison between a distance separating the third record from the hyperplane and a predetermined threshold; and

in response, when the third record is close to the hyperplane, employing the at least one processor to determine that the third record is noise.

23. The method of claim 15, further comprising training an automated classifier to discriminate between malware and clean records according to a result of determining that the first and second records are noise.

24. The method of claim 15, further comprising:

before selecting the first and second records, employing the at least one processor to divide the corpus of records into a plurality of clusters, wherein all members of a cluster of the plurality of clusters share a selected set of features; and

in response to dividing the corpus into the plurality of clusters, employing the at least one processor to select the first and second records from a first cluster of the plurality of clusters.

25. The method of claim 24, wherein dividing the corpus into the set of clusters comprises selecting feature i of the selected set of features according to a proximity between a first count and a first reference, and further according to a proximity between a second count and a second reference, wherein the first count is a count of malware-labeled records of the corpus having feature i, wherein the second count is a count of clean-labeled records of the corpus having feature i, wherein the first reference is equal to half of the cardinality of the subset of malware records, and wherein the second reference is equal to half of the cardinality of the subset of clean records.

26. The method of claim 25, wherein dividing the corpus into the set of clusters comprises selecting a feature i of the selected set of features according to:

|freq_i^malicious−

0.5|+|freq_i^clean−

0.5|,

wherein freq_i^malicious denotes a frequency of appearance of feature i within the subset of malware records of the corpus, and wherein freq_i^clean denotes a frequency of appearance of feature i within the subset of clean records of the corpus.

27. The method of claim 24, further comprising, in response to dividing the corpus into the plurality of clusters:

employing a first processor of the computer system to select the first and second records from the first cluster, to determine whether the first and second records are noise; and

employing a second processor of the computer system to select a third and a fourth records from a second cluster of the plurality of clusters, to determine whether the third and fourth records are noise,

wherein the second cluster is distinct from the first cluster, and wherein the second processor is distinct from the first processor.

28. The method of claim 24, wherein the distance is determined according to

29. A computer readable medium storing a set of instructions, which, when executed by a computer system, cause the computer system to form a record aggregator and a noise detector connected to the record aggregator, wherein the record aggregator is configured to:

assign records of a corpus to a plurality of clusters, wherein each record of the corpus is pre-labeled as either clean or malware prior to assigning records to the plurality of clusters, and wherein all members of a cluster of the plurality of clusters share a selected set of record features; and

in response to assigning the records to the plurality of clusters, send a target cluster of the plurality of clusters to the noise detector for de-noising;

and wherein the noise detector is configured, in response to receiving the target cluster, to;

select a first record and a second record from the target cluster, the first record being labeled as clean and the second record being labeled as malware;

in response to selecting the first and second records, determine whether the first and second records are similar according to a set of features; and

in response, when the first and second records are similar, determine that the first and second records are noise.