Method and system for detecting and mitigating attacks performed using cryptographic protocols

• US 9,344,448 B2
• Filed: 09/04/2014
• Issued: 05/17/2016
• Est. Priority Date: 03/21/2012
• Status: Active Grant

• Current Assignee: Radware Limited
• Original Assignee: Radware Limited
• Inventors: Chesla, Avi, Shulman, Yosefa, Ichilov, Ziv, Azoulay, Iko
• Primary Examiner(s): Hoffman, Brandon
• Assistant Examiner(s): Ambaye, Samuel
• Application Number:
  US14/477,356
• Publication Number:
  US 20140373143A1
• Time in Patent Office: 621 Days
• Field of Search: 726/22
• US Class Current: 1/1
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 67/141 Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A method and system for detecting and mitigating attacks performed using a cryptographic protocol are provided. The method comprises establishing an encrypted connection with the client using the cryptographic protocol, upon receiving an indication about a potential attack; receiving an inbound traffic from a client, wherein the inbound traffic is originally directed to a protected entity; analyzing application layer attributes of only the inbound traffic received on the encrypted connection to detect at least one encrypted attack; and causing to establish a new encrypted connection between the client and the protected entity, if the at least one encrypted attack at the application layer has not been detected.

Claims

1. A method for detecting and mitigating attacks performed using a cryptographic protocol, comprising:

upon receiving an indication about a potential attack, establishing an encrypted connection with the client using the cryptographic protocol;

receiving an inbound traffic from a client, wherein the inbound traffic is originally directed to a protected entity;

analyzing application layer attributes of only the inbound traffic received on the encrypted connection to detect at least one encrypted attack; and

causing to establish a new encrypted connection between the client and the protected entity, if the at least one encrypted attack at the application layer has not been detected.

2. The method of claim 1, wherein an outbound traffic is relayed directly from the protected entity to the client.

3. The method of claim 1, wherein establishing the encrypted connection with the client further comprises:

establishing a network connection with the client; and

performing a handshake process with the client based on provisions of the cryptographic protocol.

4. The method of claim 3, wherein the network connection is established as a delayed binding connection.

5. The method of claim 1, wherein analyzing the application layer attributes further comprises:

decrypting application layer requests received in the inbound traffic to reveal the application layer attributes; and

matching the application layer attributes against at least one adaptive attack filter to detect the at least one encrypted attack.

6. The method of claim 5, wherein the at least one adaptive attack filter defines at least one of content patterns and signatures associated with the at least one encrypted attack.

7. The method of claim 1, wherein analyzing the application layer attributes further comprises:

generating an encrypted client web challenge;

in response to a client request included in the inbound traffic, sending the encrypted client web challenge to the client over the encrypted connection;

checking if the client correctly responds to the encrypted client web challenge; and

determining the client initiated the at least one encrypted attack, when the client does not correctly respond to the encrypted client web challenge.

8. The method of claim 7, further comprising:

dropping the encrypted connection with the client, when the client initiated the at least one encrypted attack.

9. The method of claim 7, wherein the encrypted client web challenge includes at least one of:

a 302 HTTP redirect message, a redirect script, and a CAPTCHA challenge.

10. The method of claim 1, further comprising:

receiving an indication that the at least one encrypted attack cannot be detected at a network layer, wherein the network layer detection is performed by analyzing network layer attributes.

11. The method of claim 1, further comprising:

dropping the encrypted connection with the client, when the at least one encrypted attack at the application layer has not been detected.

12. The method of claim 1, wherein the cryptographic protocol includes at least any one of:

a secure socket layer (SSL) protocol and a transport layer security (TLS).

13. The method of claim 1, wherein the at least one encrypted attack is any one of:

an encrypted denial-of-service (DoS) attack and an encrypted distributed DoS (DDoS) attack.

14. The method of claim 1, wherein during attempts to detect that at least one encrypted attack there is no active encrypted connection between the client and the protected entity in which harmful traffic is relayed to the protected entity.

15. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 1.

16. A system for detecting and mitigating attacks performed using a cryptographic protocol, comprising:

a processing unit;

a memory containing instructions that, when executed by the processing unit, configure the system to;

upon receiving an indication about a potential attack, establishing an encrypted connection with the client using the cryptographic protocol;

receiving an inbound traffic from a client, wherein the inbound traffic is originally directed to a protected entity;

analyzing application layer attributes of only the inbound traffic received on the encrypted connection to detect at least one encrypted attack; and

causing to establish a new encrypted connection between the client and the protected entity, if the at least one encrypted attack at the application layer has not been detected.

17. The system of claim 16, wherein an outbound traffic is relayed directly from the protected entity to the client.

18. The system of claim 16, wherein the system is further configured to:

establish a network connection with the client; and

perform a handshake process with the client based on provisions of the cryptographic protocol.

19. The system of claim 18, wherein the network connection is established as a delayed binding connection.

20. The system of claim 16, wherein the system is further configured to:

decrypt application layer requests received in the inbound traffic to reveal the application layer attributes; and

match the application layer attributes against at least one adaptive attack filter to detect the at least one encrypted attack.

21. The system of claim 20, wherein the at least one adaptive attack filter defines at least one of content patterns and signatures associated with the at least one encrypted attack.

22. The system of claim 16, wherein the system is further configured to:

generate an encrypted client web challenge;

send the encrypted client web challenge to the client over the encrypted connection, in response to a client request included in the inbound traffic;

check if the client correctly responds to the encrypted client web challenge; and

determine the client initiated the at least one encrypted attack, when the client does not correctly respond to the encrypted client web challenge.

23. The system of claim 22, wherein the system is further configured to:

drop the encrypted connection with the client, when the client initiated the at least one encrypted attack.

24. The system of claim 22, wherein the encrypted client web challenge includes at least one of:

a 302 HTTP redirect message, a redirect script, and a CAPTCHA challenge.

25. The system of claim 16, wherein the system is further configured to:

receive an indication that the at least one encrypted attack cannot be detected at a network layer, wherein the network layer detection is performed by analyzing network layer attributes.

26. The system of claim 16, wherein the system is further configured to:

drop the encrypted connection with the client, when the at least one encrypted attack at the application layer has not been detected.

27. The system of claim 16, wherein the cryptographic protocol includes at least any one of:

a secure socket layer (SSL) protocol and a transport layer security (TLS), and wherein the at least one encrypted attack is any one of;

an encrypted denial-of-service (DoS) attack and an encrypted distributed DoS (DDoS) attack.