Systems and methods for detection and mitigation of malicious encryption

• US 10,637,879 B2
• Filed: 10/06/2017
• Issued: 04/28/2020
• Est. Priority Date: 10/06/2017
• Status: Active Grant

• Current Assignee: Carbonite LLC (Open Text Corporation)
• Original Assignee: Carbonite Incorporated (Open Text Corporation)
• Inventors: Bailey, Daniel Vernon
• Primary Examiner(s): Doan, Trang T
• Assistant Examiner(s): South, Jessica J
• Application Number:
  US15/727,463
• Publication Number:
  US 20190109869A1
• Time in Patent Office: 935 Days
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1466 Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks
• CPC Class Codes: H04L 9/14 using a plurality of keys or algorithms

Abstract

The present disclosure describes systems and methods for detection and mitigation of malicious encryption. A security agent on an infected computing device may monitor data writes to disk, memory, or network transmission buffers for strings that may represent encryption keys or moduli. The security agent may apply one or more techniques to decode and parse the string to either identify or extract the keys, or rule out the string as containing an encryption key or modulus. If a key is identified, or its presence cannot be excluded, then the security agent may generate an alert and take mitigation actions.

Claims

1. A method for detecting an encryption key for malicious encryption, comprising:

detecting, by a security agent executed by a computing device, writing of a first item of data to memory of the computing device;

compressing, by the security agent, a first portion of the first item of data;

calculating, by the security agent, a ratio of a size of the first portion of the first item of data to a size of a compressed first portion of the first item of data;

determining that the ratio does not exceed a predetermined threshold;

responsive to the determination that the ratio does not exceed the predetermined threshold, identifying the first item of data as comprising an encryption key; and

responsive to identifying the first item of data as comprising an encryption key, generating an alert, by the security agent, indicating a likely malicious encryption attempt.

2. The method of claim 1, wherein compressing the first portion of the first item of data further comprises:

compressing a second portion of the first item of data, different from the first portion;

determining that a compression ratio of the second portion of the first item of data exceeds the predetermined threshold; and

compressing the first portion of the first item of data, responsive to the determination that the compression ratio of the second portion of the first item of data exceeds the predetermined threshold.

3. The method of claim 1, further comprising:

determining, by the security agent, that a numeric representation of the first portion of the first item of data is a composite integer; and

wherein generating the alert is further performed responsive to the determination that the numeric representation of the first portion of the first item of data is a composite integer.

4. The method of claim 3, further comprising generating the numeric representation of the first portion of the first item of data via a base-64 decoding.

5. The method of claim 3, wherein determining that the numeric representation of the first item of data is a composite integer further comprises:

iteratively, for each of a predetermined set of prime numbers, determining that a result of a primality test based on a prime number of the predetermined set of prime numbers, the first item of data, and a random number is negative.

6. The method of claim 3, further comprising:

determining, by the security agent, that the numeric representation of the first portion of the first item of data lacks factors within a predetermined range; and

wherein generating the alert is further performed responsive to the determination that the numeric representation of the first portion of the first item of data lacks factors within the predetermined range.

7. The method of claim 1, wherein detecting writing of the first item of data to memory further comprises detecting writing of the first item of data to a transmission buffer of a network interface of the computing device.

8. A method for detecting an encryption key for malicious encryption, comprising:

detecting, by a security agent executed by a computing device, writing of a first item of data to memory of the computing device;

identifying, by the security agent, that the first item of data meets a predetermined size requirement;

responsive to the identification that the first item of data meets the predetermined size requirement, decoding the first item of data according to a predetermined encryption key encoding system;

determining, by the security agent, whether a decoded first item of data includes an encryption key;

responsive to the decoded first item of data not including an encryption key performing at least one further key detection procedure on the first item of data; and

responsive to the decoded first item of data not including an encryption key;

compressing, by the security agent, a first portion of the first item of data;

calculating, by the security agent, a ratio of a size of the first portion of the first item of data to a size of a compressed first portion of the first item of data;

determining that the ratio does not exceed a predetermined threshold; and

responsive to the determination that the ratio does not exceed the predetermined threshold, generating the alert, by the security agent, indicating a likely malicious encryption attempt.

9. The method of claim 8, wherein determining that the first item of data meets the predetermined size requirement further comprises determining that a length of the first item of data is within a predetermined range of bytes.

10. A method for detecting an encryption key for malicious encryption, comprising:

receiving, by a factorization host executed by a first computing device, from a second computing device, a received encryption modulus;

calculating, by the factorization host, a greatest common denominator between the received encryption modulus and an additional encryption modulus, received from a third computing device and stored in a moduli database of the first computing device;

determining that the greatest common denominator between the received encryption modulus and the additional encryption modulus is less than a minimum of either of the received encryption modulus and the additional encryption modulus to detect the encryption key is not presented in the received encryption modulus; and

transmitting, by the first computing device, the greatest common denominator to the second computing device, receipt of the greatest common denominator triggering the second computing device to perform decryption of at least one file using the greatest common denominator.

11. The method of claim 10, further comprising:

receiving a plurality of encryption moduli from a plurality of additional computing devices, by the factorization host; and

storing the plurality of encryption moduli in the moduli database, by the factorization host; and

wherein calculating the greatest common denominator between the received encryption modulus and the additional encryption modulus further comprises calculating a plurality of greatest common denominators between the received encryption modulus and each of the plurality of encryption moduli stored in the moduli database.

12. A system for detecting an encryption key for malicious encryption, comprising:

a memory unit of a computing device, the memory unit storing a first item of data; and

a security agent, executed by a processor of a computing device, configured to detect writing of the first item of data to the memory unit, and responsive to the detection;

compress a first portion of the first item of data,

calculate a ratio of a size of the first portion of the first item of data to a size of a compressed first portion of the first item of data,

determine that the ratio does not exceed a predetermined threshold,

responsive to a determination that the ratio does not exceed the predetermined threshold, identify the first item of data as comprising an encryption key, and

responsive to identifying the first item of data as comprising an encryption key, generate an alert indicating a likely malicious encryption attempt.

13. The system of claim 12, wherein the security agent is further configured to:

determine that a numeric representation of the first portion of the first item of data is a composite integer, and

generate the alert, responsive to the determination that the numeric representation of the first portion of the first item of data is a composite integer.

14. The system of claim 13, wherein the security agent is further configured to, iteratively, for each of a predetermined set of prime numbers, determine that a result of a primality test based on a prime number of the predetermined set of prime numbers, the first item of data, and a random number is negative.

15. The system of claim 13, wherein the security agent is further configured to:

determine that the numeric representation of the first portion of the first item of data lacks factors within a predetermined range; and

generate the alert, responsive to the determination that the numeric representation of the first portion of the first item of data lacks factors within the predetermined range.

16. The system of claim 12, wherein the security agent is further configured to:

determine that the first item of data meets a predetermined size requirement;

responsive to a determination that the first item of data meets the predetermined size requirement, decode the first item of data according to a predetermined encryption key encoding system;

identify an absence of an encryption key in a decoded first item of data; and

compress the first portion of the first item of data, responsive to identification of the absence of the encryption key in the decoded first item of data.

17. The system of claim 12, wherein the security agent is further configured to detect writing of the first item of data to a transmission buffer of a network interface of the computing device.

18. The system of claim 12, wherein the security agent is further configured to, responsive to generation of the alert, generate a snapshot of system memory, and search the snapshot for an encryption key.

19. The system of claim 12, wherein the security agent is further configured to identify an encryption modulus, and further comprising:

a network interface of the computing device configured to;

transmit the encryption modulus to a remote server, receipt of the encryption modulus triggering the remote server to calculate, for each of one or more previously received encryption moduli in a database, a greatest common denominator, and

receive, from the remote server, a selected greatest common denominator; and

wherein the security agent is further configured to calculate, from the selected greatest common denominator and the encryption modulus, an encryption key.