Method and system for detecting and mitigating attacks performed using cryptographic protocols

• US 8,832,831 B2
• Filed: 03/21/2012
• Issued: 09/09/2014
• Est. Priority Date: 03/21/2012
• Status: Active Grant

• Current Assignee: Radware Limited
• Original Assignee: Radware Limited
• Inventors: Chesla, Avi, Shulman, Yosefa, Ichilov, Ziv, Azoulay, Iko
• Primary Examiner(s): Hoffman, Brandon
• Assistant Examiner(s): Ambaye, Samuel
• Application Number:
  US13/425,978
• Publication Number:
  US 20130254879A1
• Time in Patent Office: 902 Days
• Field of Search: 726/22, 709/217, 713/168
• US Class Current: 726/22
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 67/141 Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A method and security system for detecting and mitigating encrypted denial-of-service (DoS) attacks. The system includes a DoS defense (DoSD) module configured to detect an encrypted DoS attack in an inbound traffic by analyzing attributes only in the inbound traffic that relate to at least one of a network layer and an application layer, wherein the DoSD module is further configured to mitigate a detected encrypted attack, the inbound traffic originates at a client and is addressed to a protected server; and a cryptographic protocol engine (CPE) configured to establish a new encrypted session between the client and the security system, decrypt requests included in the inbound traffic, and send encrypted responses to the client over the new encrypted session between the client and the security system.

Claims

1. A security system for detecting and mitigating encrypted denial-of-service (DoS) attacks, the security system is connected in a secured database, comprising:

a DoS defense (DoSD) module configured to detect an encrypted DoS attack in an inbound traffic by analyzing attributes only in the inbound traffic that relate to at least one of a network layer and an application layer, wherein the DoSD module is configured to initially analyze the network layer attributes in the inbound traffic, and analyze the application layer attributes when an encrypted DoS attack at the network layer has not been detected, wherein the DoSD module is further configured to mitigate a detected encrypted attack, the inbound traffic originates at a client and is addressed to a protected server; and

a cryptographic protocol engine (CPE) configured to establish a new encrypted session between the client and the security system, decrypt requests included in the inbound traffic, and send encrypted responses to the client over the new encrypted session between the client and the security system, wherein the CPE is further configured to establish a new network connection between the client and the protected server when an encrypted DoS attack at the application layer has not been detected, wherein the new network connection causes outbound traffic originating at the protected server to be directly routed to the client to eliminate reception and processing of the outbound traffic by the security system connected in the secured datacenter.

2. The security system of claim 1, wherein the encrypted DoS attack is performed using an application layer cryptographic protocol, wherein the cryptographic protocol includes at least any one of:

a secure socket layer (SSL) protocol, and a transport layer security (TLS), wherein the encrypted DoS attack also includes an encrypted distributed DoS (DDoS) attack.

3. The security system of claim 1, wherein the DoSD module comprises:

a traffic anomaly mechanism configured to detect at least the DoS encrypted attack by identifying abnormal traffic behavior in the network layer attributes of the inbound traffic;

a network-based DoS mechanism configured to detect at least the DoS encrypted attack by identifying known attack patterns in the network layer attributes, wherein the network layer attributes are not encrypted;

an application-based DoS mechanism configured to detect at least the DoS encrypted attack by analyzing application layer attributes only in the inbound traffic; and

a directed-application DoS mechanism configured to detect at least the DoS encrypted attack by identifying one or more known application layer attack patterns in the application layer attributes, wherein the application layer attributes are encrypted.

4. The security system of claim 3, wherein the application-based DoS mechanism and the directed-application mechanism utilize the CPE to decrypt application layer attributes, wherein the CPE is configured with encryption keys of the protected server.

5. The security system of claim 3, wherein at least one of the traffic anomaly mechanism and the network-based DoS mechanism attempt to detect the encrypted DoS attack at the network layer, wherein the network-based DoS mechanism is further configured to drop a current encrypted connection established between the client and the protected server, when the encrypted DoS attack is detected.

6. The security system of claim 1, wherein the security system is configured to:

terminate the current network connection and encrypted session between the client and the protected server by means of the CPE;

establish a new network connection with the new encrypted session between the client and the security system;

decrypt a first request from the client by means of the CPE;

match the decrypted first request against the one or more known application layer attack patterns by means of the directed-application DoS mechanism, wherein the first request is part of application layer attributes; and

drop the new encrypted session with the client when the decrypted first request matches at least one of the one or more known application attack patterns.

7. The security system of claim 1, wherein the security system is further configured to:

generate a client web challenge, if the first request does not match at least one of the one or more known application attack patterns by means of the application-based DoS mechanism;

encrypt, by means of the CPE, the client web challenge using encryption keys exchanged with the client during the establishment of the new encrypted session with the client;

respond to the first request with the encrypted client web challenge over the new encrypted session with the client;

determine if the client correctly responds to the encrypted client web challenge; and

drop the new encrypted session with the client when the client does not correctly respond to the encrypted client web challenge.

8. The security system of claim 1, wherein the inbound traffic is diverted to the security system when it is suspected that the inbound traffic includes malicious threats.

9. The security system of claim 1, wherein during processing of the inbound traffic by the security system no traffic is relayed from the client to the protected server.

10. A method for detecting and mitigating encrypted denial-of-service (DoS) attacks, comprising:

detecting by a security system connected in a secured datacenter, an encrypted DoS attack in an inbound traffic by analyzing network layer attributes of only the inbound traffic, the inbound traffic originates at a client and is addressed to a protected server and the network layer attributes are not encrypted;

dropping a network connection between the client and the protected server if an encrypted DoS attack at the network layer has been detected;

terminating an encrypted connection with the client if an encrypted DoS attack at the network layer has not been detected;

detecting, by the security system, an encrypted DoS attack in the inbound traffic by analyzing application layer attributes of only the inbound traffic, wherein the network layer attributes are encrypted; and

causing to establish a new network connection between the client and the protected server, if an encrypted DoS attack at the application layer has not been detected, wherein the new network connection causes outbound traffic originating at the protected server to be directly routed to the client to eliminate reception and processing of the outbound traffic by the security system connected in the secured datacenter.

11. The method of claim 10, wherein detecting the encrypted DoS attack by analyzing network layer attributes of the inbound traffic further comprising:

identifying abnormal traffic behavior in the network layer attributes; and

identifying one or more known network layer attack patterns in the network layer attributes.

12. The method of claim 10, wherein detecting the encrypted DoS attack by analyzing application layer attributes further comprising:

establishing a new encrypted session with the client;

decrypting a request received from the client, wherein the request is part of the application layer attributes;

generating a client web challenge;

encrypting the client web challenge using encryption keys exchanged with the client during the establishment of the new encrypted session with the client;

responding to the client'"'"'s request with an encrypted client web challenge over the new encrypted session;

determining if the client correctly responds to the encrypted client web challenge; and

dropping the new encrypted session with the client when the client does not correctly respond to the encrypted client web challenge.

13. The method of claim 12, further comprising:

matching the decrypted request against one or more known application layer attack patterns; and

dropping the encrypted connection with the client when the decrypted request matches at least one of the one or more known application attack patterns.

14. The method of claim 10, wherein the encrypted DoS attack is performed using a cryptographic protocol, wherein the cryptographic protocol includes at least any one of a secure socket layer (SSL) protocol and a transport layer security (TLS), wherein the encrypted DoS attack also includes an encrypted distributed DoS (DDoS) attack.

15. The method of claim 10, wherein during attempts to detect the encrypted DoS attack there is no active encrypted connection between the client and the protected server in which harmful traffic is relayed to the protected server.

16. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 10.

18. The method of claim 10, further comprising:

generating a client web challenge; and

encrypting the client web challenge using an encryption keys exchange with the client during the establishment of the new encrypted connection with the client.

19. The method of claim 18, further comprising:

matching the decrypted request against one or more known application layer attack patterns; and

dropping the new encrypted session with the client when the decrypted request matches at least one of the one or more known application attack patterns.

20. The method of claim 18, further comprising:

detecting an encrypted DoS attack in an inbound traffic by analyzing network layer attributes of the inbound traffic, wherein the network layer attributes are not encrypted; and

dropping a connection between the client and the protected server if an encrypted DoS attack at the network layer has been detected.

17. A method for detecting and mitigating encrypted denial-of-service (DoS) attacks, comprising:

terminating, by a security system connected in a secured datacenter, an encrypted connection between a client and a protected server, upon reception of only an inbound traffic diverted from the client, wherein the inbound traffic is suspected to include malicious threats;

establishing a new encrypted session with the client;

responding to a client'"'"'s request received over the new encrypted session with an encrypted client web challenge;

determining, by the security system, if the client correctly responds to the encrypted client web challenge;

dropping the new encrypted session with the client, if the client does not correctly respond to the encrypted client web challenge; and

causing the client to establish a new network connection with the protected server, if the client does correctly respond to the encrypted client web challenge, wherein the new networkcauses outbound traffic originating at the protected server to be directly routed to the client to eliminate reception and processing of the outbound traffic by the security system connected in the secured datacenter.

21. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 17.