METHOD AND SYSTEM FOR DETECTING AND MITIGATING ATTACKS PERFORMED USING CRYPTOGRAPHIC PROTOCOLS

• US 20130254879A1
• Filed: 03/21/2012
• Published: 09/26/2013
• Est. Priority Date: 03/21/2012
• Status: Active Grant

• Current Assignee: Radware Limited
• Original Assignee: Radware Limited
• Inventors: Chesla, Avi, Shulman, Yosefa, Ichilov, Ziv, Azoulay, Iko
• Granted Patent: US 8,832,831 B2
• Time in Patent Office: Days
• US Class Current: 726/22
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 67/141 Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A method and security system for detecting and mitigating encrypted denial-of-service (DoS) attacks. The system includes a DoS defense (DoSD) module configured to detect an encrypted DoS attack in an inbound traffic by analyzing attributes only in the inbound traffic that relate to at least one of a network layer and an application layer, wherein the DoSD module is further configured to mitigate a detected encrypted attack, the inbound traffic originates at a client and is addressed to a protected server; and a cryptographic protocol engine (CPE) configured to establish a new encrypted session between the client and the security system, decrypt requests included in the inbound traffic, and send encrypted responses to the client over the new encrypted session between the client and the security system.

Claims

1. A security system for detecting and mitigating encrypted denial-of-service (DoS) attacks, comprising:

a DoS defense (DoSD) module configured to detect an encrypted DoS attack in an inbound traffic by analyzing attributes only in the inbound traffic that relate to at least one of a network layer and an application layer, wherein the DoSD module is further configured to mitigate a detected encrypted attack, the inbound traffic originates at a client and is addressed to a protected server; and

a cryptographic protocol engine (CPE) configured to establish a new encrypted session between the client and the security system, decrypt requests included in the inbound traffic, and send encrypted responses to the client over the new encrypted session between the client and the security system.

2. The security system of claim 1, wherein the encrypted DoS attack is performed using an application layer cryptographic protocol, wherein the cryptographic protocol includes at least any one of a secure socket layer (SSL) protocol and a transport layer security (TLS), wherein the encrypted DoS attack also includes an encrypted distributed DoS (DDoS) attack.

3. The security system of claim 1, wherein the DoSD module comprises:

a traffic anomaly mechanism configured to detect at least the DoS encrypted attack by identifying abnormal traffic behavior in the network layer attributes of the inbound traffic;

a network-based DoS mechanism configured to detect at least the DoS encrypted attack by identifying known attack patterns in the network layer attributes, wherein the network layer attributes are not encrypted;

an application-based DoS mechanism configured to detect at least the DoS encrypted attack by analyzing application layer attributes only in the inbound traffic; and

a directed-application DoS mechanism configured to detect at least the DoS encrypted attack by identifying one or more known application layer attack patterns in the application layer attributes, wherein the application layer attributes are encrypted.

4. The security system of claim 3, wherein the application-based DoS mechanism and the directed-application mechanism utilize the CPE to decrypt application layer attributes, wherein the CPE is configured with encryption keys of the protected server.

5. The security system of claim 3, wherein at least one of the traffic anomaly mechanism and the network-based DoS mechanism attempt to detect the encrypted DoS attack at the network layer, wherein the network-based DoS mechanism is further configured to drop a current encrypted connection established between the client and the protected server, when the encrypted DoS attack is detected.

6. The security system of claim 5, wherein the security system is further configured to detect the encrypted DoS attack at the application layer, when the encrypted DoS attack at the network layer has not been detected.

7. The security system of claim 6, wherein the security system is configured to detect the encrypted DoS attack at the application layer by:

terminating the current TCP connection and encrypted session between the client and the protected server by means of the CPE;

establishing the new encrypted session between the client and the security system;

decrypting a first request from the client by means of the CPE;

matching the decrypted first request against the one or more known application layer attack patterns by means of the directed-application DoS mechanism, wherein the first request is part of application layer attributes; and

dropping the new encrypted session with the client, if the decrypted first request matches at least one of the one or more known application attack patterns.

8. The security system of claim 1, wherein the security system is further configured to:

generate a client web challenge, if the first request does not match at least one of the one or more known application attack patterns by means of the application-based DoS mechanism;

encrypting, by means of the CPE, the client web challenge using encryption keys exchanged with the client during the establishment of the new encrypted session with the client;

responding to the first request with the encrypted client web challenge over the new encrypted session with the client;

determining if the client correctly responds to the encrypted client web challenge; and

dropping the new encrypted session with the client, if the client does not correctly respond to the encrypted client web challenge.

9. The security system of claim 8, wherein the security system is further configured to cause the client to establish a new encrypted connection with the protected server if the client correctly responds to the encrypted client web challenge.

10. The security system of claim 1, wherein the security system is deployed in a secured datacenter and the inbound traffic is diverted to the security system when it is suspected that the inbound traffic includes malicious threats.

11. The security system of claim 1, wherein the security system is deployed between the client and the protected server as an in-line deployment, wherein the security system processes only inbound traffic.

12. The security system of claim 1, wherein during processing of the inbound traffic by the security system no traffic is relayed from the client to the protected server.

13. A method for detecting and mitigating encrypted denial-of-service (DoS) attacks, comprising:

detecting an encrypted DoS attack in an inbound traffic by analyzing network layer attributes of only the inbound traffic, the inbound traffic originates at a client and is addressed to a protected server and the network layer attributes are not encrypted;

dropping a connection between the client and the protected server if an encrypted DoS attack at the network layer has been detected;

terminating an encrypted connection with the client if an encrypted DoS attack at the network layer has not been detected;

detecting an encrypted DoS attack in the inbound traffic by analyzing application layer attributes of only the inbound traffic, wherein the network layer attributes are encrypted; and

causing to establish a new encrypted connection between the client and the protected server, if an encrypted DoS attack at the application layer has not been detected.

14. The method of claim 13, wherein detecting the encrypted DoS attack by analyzing network layer attributes of the inbound traffic further comprising:

identifying abnormal traffic behavior in the network layer attributes; and

identifying one or more known network layer attack patterns in the network layer attributes.

15. The method of claim 13, wherein detecting the encrypted DoS attack by analyzing application layer attributes further comprising:

establishing a new encrypted session with the client;

decrypting a request received from the client, wherein the request is part of the application layer attributes;

generating a client web challenge;

encrypting the client web challenge using encryption keys exchanged with the client during the establishment of the new encrypted session with the client;

responding to the client'"'"'s request with an encrypted client web challenge over the new encrypted session;

determining if the client correctly responds to the encrypted client web challenge; and

dropping the new encrypted session with the client, if the client does not correctly respond to the encrypted client web challenge.

16. The method of claim 15, further comprising:

matching the decrypted request against one or more known application layer attack patterns; and

dropping the encrypted connection with the client, if the decrypted request matches at least one of the one or more known application attack patterns.

17. The method of claim 13, wherein the encrypted DoS attack is performed using a cryptographic protocol, wherein the cryptographic protocol includes at least any one of a secure socket layer (SSL) protocol and a transport layer security (TLS), wherein the encrypted DoS attack also includes an encrypted distributed DoS (DDoS) attack.

18. The method of claim 13, wherein during attempts to detect the encrypted DoS attack there is no active encrypted connection between the client and the protected server in which harmful traffic is relayed to the protected server.

19. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 13.

21. The method of claim 13, further comprising:

generating a client web challenge; and

encrypting the client web challenge using an encryption keys exchange with the client during the establishment of the new encrypted connection with the client.

22. The method of claim 21, further comprising:

matching the decrypted request against one or more known application layer attack patterns; and

dropping the new encrypted session with the client, if the decrypted request matches at least one of the one or more known application attack patterns.

23. The method of claim 21, further comprising:

detecting an encrypted DoS attack in an inbound traffic by analyzing network layer attributes of the inbound traffic, wherein the network layer attributes are not encrypted; and

dropping a connection between the client and the protected server if an encrypted DoS attack at the network layer has been detected.

20. A method for detecting and mitigating encrypted denial-of-service (DoS) attacks, comprising:

terminating an encrypted connection between the client and a protected server, upon reception of only an inbound traffic diverted from a client, wherein the inbound traffic is suspected to include malicious threats;

establishing a new encrypted session with the client;

responding to a client'"'"'s request received over the new encrypted session with an encrypted client web challenge;

determining if the client correctly responds to the encrypted client web challenge;

dropping the new encrypted session with the client, if the client does not correctly respond to the encrypted client web challenge; and

causing the client to establish a new encrypted session with the protected server, if the client does correctly respond to the encrypted client web challenge.

24. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 20.