Systems and methods for detecting malicious use of digital certificates

• US 9,407,644 B1
• Filed: 11/26/2013
• Issued: 08/02/2016
• Est. Priority Date: 11/26/2013
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Li, Ying, Cheng, Tao, Roundy, Kevin, Fu, Jie, Li, Zhi Kai
• Primary Examiner(s): Abedin, Shanto M
• Application Number:
  US14/089,999
• Time in Patent Office: 980 Days
• Field of Search: 726 22- 25, 713/168, 713/175, 713/156
• US Class Current: 1/1
• CPC Class Codes: H04L 63/0853 using an additional device, e.g. smartcard, SIM or a different communication terminal cryptographic mechanisms or cryptographic arrangements for entity authentication involving additional secure or trusted devices H04L9/3234
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

A computer-implemented method for detecting malicious use of digital certificates may include determining that a digital certificate is invalid. The method may further include locating, within the invalid digital certificate, at least one field that was previously identified as being useful in distinguishing malicious use of invalid certificates from benign use of invalid certificates. The method may also include determining, based on analysis of information from the field of the invalid digital certificate, that the invalid digital certificate is potentially being used to facilitate malicious communications. The method may additionally include performing a security action in response to determining that the invalid digital certificate is potentially being used to facilitate malicious communications. Various other methods, systems, and computer-readable media are disclosed.

Claims

1. A computer-implemented method for detecting malicious use of digital certificates, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:

deducing, by a deduction module, an effectiveness of a plurality of fields within digital certificates in distinguishing malicious use of invalid certificates from benign use of invalid certificates by using a machine learning algorithm to examine fields of invalid certificates that have been used maliciously and fields of invalid certificates that have been used legitimately;

weighting, by the deduction module, each of the plurality of fields according to the effectiveness of the fields;

identifying, by the deduction module and based on the weighting of each of the plurality of fields, at least one field within invalid digital certificates that is more useful than at least one other field in distinguishing malicious use of invalid certificates from benign use of invalid certificates;

before performing an analysis to determine whether a digital certificate is potentially being used to facilitate malicious communications, determining, by a determination module, that the digital certificate is invalid by determining that the digital certificate is not trusted by a certificate authority;

in response to determining that the digital certificate is invalid, locating, by a location module, within the invalid digital certificate, the field that was identified as being more useful in distinguishing malicious use of invalid certificates from benign use of invalid certificates;

determining, by an analysis module, based on an analysis of information from the field of the invalid digital certificate, that the invalid digital certificate is potentially being used to facilitate malicious communications;

performing, by a security module, a security action in response to determining that the invalid digital certificate is potentially being used to facilitate malicious communications.

2. The computer-implemented method of claim 1, wherein determining that the digital certificate is not trusted by the certificate authority comprises determining that the digital certificate is self-signed.

3. The computer-implemented method of claim 1, wherein determining that the invalid digital certificate is potentially being used to facilitate malicious communications comprises determining that the invalid digital certificate is being used to establish an encrypted communication session for a malicious purpose.

4. The computer-implemented method of claim 3, wherein performing the security action comprises interrupting the encrypted communication session.

5. The computer-implemented method of claim 1, wherein determining that the invalid digital certificate is potentially being used to facilitate malicious communications comprises detecting that the information within the field comprises at least one of:

a domain name that comprises a randomly generated character string;

a certificate chain length of 0;

a public key length that does not conform to a key-length standard.

6. The computer-implemented method of claim 1, wherein:

analyzing the information within the field of the invalid digital certificate comprises identifying a domain name within the field;

the computer-implemented method further comprises at least one of;

pinging the domain name to determine if it is a legitimate domain name;

identifying a reputation score for the domain name.

7. A system for detecting malicious use of digital certificates, the system comprising:

a memory;

a deduction module, stored in the memory, that;

deduces an effectiveness of a plurality of fields within digital certificates in distinguishing malicious use of invalid certificates from benign use of invalid certificates by using a machine learning algorithm to examine fields of invalid certificates that have been used maliciously and fields of invalid certificates that have been used legitimately;

weights each of the plurality of fields according to the effectiveness of the fields;

identifies, based on the weighting of each of the plurality of fields, at least one field within invalid digital certificates that is more useful than at least one other field in distinguishing malicious use of invalid certificates from benign use of invalid certificates;

a determination module, stored in the memory, that determines, before an analysis is performed to determine whether a digital certificate is being used to facilitate malicious communications, that the digital certificate is invalid by determining that the digital certificate is not trusted by a certificate authority;

a location module, stored in the memory, that locates, within the invalid digital certificate in response to the determination that the digital certificate is invalid, the field that was identified as being more useful in distinguishing malicious use of invalid certificates from benign use of invalid certificates;

an analysis module, stored in the memory, that determines, based on an analysis of information from the field of the invalid digital certificate, that the invalid digital certificate is potentially being used to facilitate malicious communications;

a security module, stored in the memory, that performs a security action in response to determining that the invalid digital certificate is potentially being used to facilitate malicious communications;

at least one hardware processor that is configured to execute the deduction module, the determination module, the location module, the analysis module, and the security module.

8. The system of claim 7, wherein the determination module determines that the digital certificate is not trusted by the certificate authority by determining that the digital certificate is self-signed.

9. The system of claim 7, wherein the analysis module determines that the invalid digital certificate is potentially being used to facilitate malicious communications by determining that the invalid digital certificate is being used to establish an encrypted communication session for a malicious purpose.

10. The system of claim 9, wherein the security action comprises interrupting the encrypted communication session.

11. The system of claim 7, wherein the analysis module determines that the invalid digital certificate is potentially being used to facilitate malicious communications by detecting that the information within the field comprises at least one of:

a domain name that comprises a randomly generated character string;

a certificate chain length of 0;

a public key length that does not conform to a key-length standard.

12. The system of claim 7, wherein:

the analysis module analyzes the information within the field of the invalid digital certificate by identifying a domain name within the field;

the analysis module further performs at least one of;

pinging the domain name to determine if it is a legitimate domain name;

identifying a reputation score for the domain name.

13. A non-transitory computer-readable storage medium comprising one or more computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

deduce, by a deduction module, an effectiveness of a plurality of fields within digital certificates in distinguishing malicious use of invalid certificates from benign use of invalid certificates by using a machine learning algorithm to examine fields of invalid certificates that have been used maliciously and fields of invalid certificates that have been used legitimately;

weight, by the deduction module, each of the plurality of fields according to the effectiveness of the fields;

identify, by the deduction module and based on the weighting of each of the plurality of fields, at least one field within invalid digital certificates that is more useful than at least one other field in distinguishing malicious use of invalid certificates from benign use of invalid certificates;

before performing an analysis to determine whether a digital certificate is potentially being used to facilitate malicious communications, determine, by a determination module, that the digital certificate is invalid by determining that the digital certificate is not trusted by a certificate authority;

in response to determining that the digital certificate is invalid, locate, by a location module, within the digital certificate, the field that was identified as being more useful in distinguishing malicious use of invalid certificates from benign use of invalid certificates;

determine, by an analysis module, based on an analysis of information from the field of the invalid digital certificate, that the invalid digital certificate is potentially being used to facilitate malicious communications;

perform, by a security module, a security action in response to determining that the invalid digital certificate is potentially being used to facilitate malicious communications.

14. The non-transitory computer-readable storage medium of claim 13, wherein the one or more computer-readable instructions cause the computing device to determine that the digital certificate is not trusted by the certificate authority by determining that the digital certificate is self-signed.

15. The non-transitory computer-readable storage medium of claim 13, wherein the one or more computer-readable instructions cause the computing device to determine that the invalid digital certificate is potentially being used to facilitate malicious communications by determining that the invalid digital certificate is being used to establish an encrypted communication session for a malicious purpose.

16. The non-transitory computer-readable storage medium of claim 15, wherein the one or more computer-readable instructions cause the computing device to, as part of the security action, interrupt the encrypted communication session.

17. The non-transitory computer-readable storage medium of claim 13, wherein the one or more computer-readable instructions cause the computing device to determine that the invalid digital certificate is potentially being used to facilitate malicious communications by detecting that the information within the field comprises at least one of:

a domain name that comprises a randomly generated character string;

a certificate chain length of 0;

a public key length that does not conform to a key-length standard.