METHODS AND APPARATUS TO IDENTIFY MALICIOUS ACTIVITY IN A NETWORK

• US 20150135320A1
• Filed: 11/14/2013
• Published: 05/14/2015
• Est. Priority Date: 11/14/2013
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Coskun, Baris
• Granted Patent: US 9,503,465 B2
• Time in Patent Office: Days
• US Class Current: 726/24
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 63/0254 Stateful filtering
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Methods, apparatus, systems and articles of manufacture are disclosed to learn malicious activity. An example method includes assigning weights of a distance function to respective statistical features; iteratively calculating, with a processor, the distance function to adjust the weights (1) to cause a reduction in a first distance calculated according to the distance function for a first pair of entities in a reference group associated with malicious activity and (2) to cause an increase in a second distance calculated according to the distance function for a first one of the entities included in the reference group and a second entity not included in the reference group; and determining whether a first statistical feature is indicative of malicious activity based on a respective adjusted weight of the first statistical feature determined after calculating the distance function for a number of iterations.

Claims

1. A method comprising:

assigning weights of a distance function to respective statistical features, the distance function to calculate a distance between a pair of entities in a network based on respective calculated values of the statistical features corresponding to the pair of entities;

iteratively calculating, with a processor, the distance function to adjust the weights (1) to cause a reduction in a first distance calculated according to the distance function for a first pair of entities in a reference group associated with malicious activity and (2) to cause an increase in a second distance calculated according to the distance function for a first one of the entities included in the reference group and a second entity not included in the reference group; and

determining whether a first statistical feature is indicative of malicious activity based on a respective adjusted weight of the first statistical feature determined after calculating the distance function for a number of iterations.

2. The method according to claim 1, further comprising:

parsing network log records into fields based on communication information in the network logs;

determining categories of the fields based on the communication information in the respective fields; and

generating the statistical features from the network log records based on the categories of the fields.

3. The method according to claim 2, further comprising generating the statistical features by:

generating a first tier of statistical features from a first set of the fields identified as storing counter information and a second set of the fields identified as storing identity information,

generating a second tier of statistical features from the first tier of statistical features and a third set of fields identified as storing communication type information; and

generating a third tier of statistical features from the first tier of statistical features and the second tier of statistical features by generating ratios of respective pairs of statistical features from the first and second tiers of statistical features.

4. The method according to claim 1, wherein the weights are iteratively adjusted using a stochastic gradient descent.

5. The method according to claim 1, wherein the first statistical feature is determined to be indicative of malicious activity if the respective adjusted weight has a value greater than a second respective adjusted weight for a second statistical feature.

6. The method according to claim 1, further comprising identifying a third entity not in the reference group as malicious based on the first statistical feature.

7. The method according to claim 6, further comprising identifying the third entity as malicious by:

identifying a respective value of the first statistical feature for the third entity;

comparing the respective value of the first statistical feature for the third entity to a malicious statistical feature value; and

determining the third entity is malicious if the respective value of the first statistical feature for the third entity is within a threshold value of the malicious statistical feature value.

8. An apparatus comprising:

a memory to store machine readable instructions; and

a processor to execute the instructions to perform operations comprising;

assigning weights of a distance function to respective statistical features, the distance function to calculate a distance between a pair of entities in a network based on respective calculated values of the statistical features corresponding to the pair of entities;

iteratively calculating the distance function to adjust the weights (1) to cause a reduction in a first distance calculated according to the distance function for a first pair of entities in a reference group associated with malicious activity and (2) to cause an increase in a second distance calculated according to the distance function for a first one of the entities included in the reference group and a second entity not included in the reference group; and

determining whether a first statistical feature is indicative of malicious activity based on a respective adjusted weight of the first statistical feature determined after calculating the distance function for a number of iterations.

9. The apparatus according to claim 8, wherein the operations further comprise:

parsing network log records into fields based on communication information in the network logs;

determining categories of the fields based on the communication information in the respective fields; and

generating the statistical features from the network log records based on the categories of the fields.

10. The apparatus according to claim 9, wherein the operations further comprise generating the statistical features by:

generating a first tier of statistical features from a first set of the fields identified as storing counter information and a second set of the fields identified as storing identity information,

generating a second tier of statistical features from the first tier of statistical features and a third set of fields identified as storing communication type information; and

generating a third tier of statistical features from the first tier of statistical features and the second tier of statistical features by generating ratios of respective pairs of statistical features from the first and second tiers of statistical features.

11. The apparatus according to claim 8, wherein the weights are iteratively adjusted using a stochastic gradient descent.

12. The apparatus according to claim 8, wherein the first statistical feature is determined to be indicative of malicious activity if the respective adjusted weight has a value greater than a second respective adjusted weight for a second statistical feature.

13. The apparatus according to claim 8, wherein the operations further comprise identifying a third entity not in the reference group as malicious based on the first statistical feature.

14. The apparatus according to claim 8, wherein the operations further comprise identifying the third entity as malicious by:

identifying a respective statistical feature value of the first statistical feature for the third entity;

comparing the respective statistical feature value of the first statistical feature for the third entity to a malicious statistical feature value; and

determining the third entity is malicious if the respective statistical feature value of the first statistical feature for the third entity is within a threshold value of the malicious statistical feature value.

15. A tangible machine readable storage medium comprising instructions which, when executed, cause a machine to perform operations comprising:

assigning weights of a distance function to respective statistical features, the distance function to calculate a distance between a pair of entities in a network based on respective calculated values of the statistical features corresponding to the pair of entities;

iteratively calculating the distance function to adjust the weights (1) to cause a reduction in a first distance calculated according to the distance function for a first pair of entities in a reference group associated with malicious activity and (2) to cause an increase in a second distance calculated according to the distance function for a first one of the entities included in the reference group and a second entity not included in the reference group; and

determining whether a first statistical feature is indicative of malicious activity based on a respective adjusted weight of the first statistical feature determined after calculating the distance function for a number of iterations.

16. The storage medium according to claim 15, wherein the operations further comprise:

parsing network log records into fields based on communication information in the network logs;

determining categories of the fields based on the communication information in the respective fields; and

generating the statistical features from the network log records based on the categories of the fields.

17. The storage medium according to claim 16, wherein the operations further comprise generating the statistical features by:

generating a first tier of statistical features from a first set of the fields identified as storing counter information and a second set of the fields identified as storing identity information,

generating a second tier of statistical features from the first tier of statistical features and a third set of fields identified as storing communication type information; and

generating a third tier of statistical features from the first tier of statistical features and the second tier of statistical features by generating ratios of respective pairs of statistical features from the first and second tiers of statistical features.

18. The method according to claim 15, wherein the first statistical feature is determined to be indicative of malicious activity if the respective adjusted weight has a value greater than a second respective adjusted weight for a second statistical feature.

19. The storage medium according to claim 15, wherein the operations further comprise identifying a third entity not in the reference group as malicious based on the first statistical feature.

20. The storage medium according to claim 19, wherein the operations further comprise identifying the third entity as malicious by:

identifying a respective value of the first statistical feature for the third entity;

comparing the respective value of the first statistical feature for the third entity to a malicious statistical feature value; and

determining the third entity is malicious if the respective value of the first statistical feature for the third entity is within a threshold value of the malicious statistical feature value.