Mitigating TCP SYN DDoS attacks using TCP reset

• US 10,158,666 B2
• Filed: 07/26/2016
• Issued: 12/18/2018
• Est. Priority Date: 07/26/2016
• Status: Active Grant

• Current Assignee: A10 Networks Incorporated
• Original Assignee: A10 Networks Incorporated
• Inventors: Jalan, Rajkumar, Kamat, Gurudeep, Szeto, Ronald Wai Lun
• Primary Examiner(s): Shiferaw, Eleni A
• Assistant Examiner(s): Khan, Sher A
• Application Number:
  US15/220,326
• Publication Number:
  US 20180034848A1
• Time in Patent Office: 875 Days
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1458 Denial of Service

Abstract

Provided are methods and systems for mitigating a denial of service attack. A system for mitigating a denial of service attack may include a network module, a storage module, and a processor module. The network module may be operable to receive a request from a network device to establish a data connection between the network device and a server based on a determination that the network device is trusted. The storage module may be operable to store a whitelist associated with a plurality of trusted network devices. The processor module may be operable to determine that the network device is trusted. Based on the determination, the processor module may associate the network device with the whitelist for a predetermined period of time.

Claims

1. A system for determining that a network device is trusted, the system comprising:

a network module configured to;

receive a request from a network device to establish a data connection between the network device and a server, wherein the receiving of the request from the network device includes receiving, from the network device, an initial synchronization (SYN) request to establish a Transmission Control Protocol (TCP) session;

determine that the data connection associated with the TCP session between the network device and the server is trusted based on a determination that the network device is trusted, the determination being reached by generating a SYN cookie, transmitting a SYN acknowledgment (ACK) including the SYN cookie to the network device, responsive to the SYN ACK, receiving from the network device a network device ACK, and determining that the network device ACK includes the SYN cookie, thereby validating authenticity of the SYN cookie;

transmit an unexpected identifier to the network device, the unexpected identifier causing the network device to terminate the TCP session and to start a new TCP session;

receive, from the network device, a further SYN request to establish the new TCP session; and

establish a trusted data connection associated with the new TCP session between the network device and the server, wherein data packets from the server are routed directly to the network device without being processed by the network module;

a storage device operable to store a whitelist associated with a plurality of trusted network devices; and

a processor operable to;

determine that the network device is trusted; and

based on the determination, associate the network device with the whitelist for a predetermined period of time.

2. The system of claim 1, wherein lack of the network device ACK is indicative of the network device being untrusted.

3. The system of claim 1, wherein the SYN cookie includes an encryption of an identifier associated with the SYN.

4. The system of claim 1, wherein the unexpected identifier is included in a reset cookie, wherein the network module is further configured to, responsive to the reset cookie, receive, from the network device a reset command, the reset command being generated responsive to the unexpected identifier.

5. The system of claim 4, wherein establishing the data connection between the network device and the server includes:

receiving, by the network module, a further SYN request from the network device; and

based on the determining that the network device is trusted, transmitting, by the network module, the further SYN request to the server.

6. The system of claim 1, wherein the processor is further operable to determine that the predetermined period of time has expired and, based on the determination, removing, by the processor, the network device from a white list, thereby flagging, by processor, the network device as untrusted.

7. The system of claim 1, wherein the network module is further configured to direct return data packets from the server directly to the network device without being processed by the network module.

8. The system of claim 1, wherein no SYN cookies are sent by the network module to the network device while the network device is whitelisted.

9. A method for determining that a network device is trusted, the method comprising:

receiving, by a network appliance, a request from a network device to establish a data connection with a server, wherein the receiving of the request from the network device includes receiving, by the network appliance, from the network device, an initial synchronization (SYN) request to establish a Transmission Control Protocol (TCP) session;

determining, by the network appliance, that the data connection associated with the TCP session between the network device and the server is trusted based on a determination that the network device is trusted, the determination being made by generating a SYN cookie, transmitting a SYN acknowledgement (ACK) including the SYN cookie to the network device, responsive to the SYN ACK, receiving from the network device a network device ACK, and determining that the network device ACK includes the SYN cookie, thereby validating authenticity of the SYN cookie;

responsive to the determination, whitelisting, by the network appliance, the network device for a predetermined period of time; and

transmitting, by the network appliance, an unexpected identifier to the network device, the unexpected identifier causing the network device to terminate the TCP session and to start a new TCP session;

receiving, by the network appliance, from the network device, a further SYN request to establish the new TCP session;

determining, by the network appliance, that the network device is whitelisted; and

establishing, by the network appliance, a trusted data connection associated with the new TCP session between the network device and the server, wherein data packets from the server are routed directly to the network device without being processed by the network appliance.

10. The method of claim 9 wherein lack of the network device ACK is indicative of the network device being untrusted.

11. The method of claim 9, wherein the unexpected identifier is included in a reset cookie,

the method further comprising:

responsive to the reset cookie, receiving, by the network appliance, from the network device, a reset command, the reset command being generated responsive to the unexpected identifier.

12. The method of claim 11, wherein the establishing the data connection between the network device and the server includes:

receiving, by the network appliance, a further SYN request from the network device; and

based on the determining that the network device is trusted, transmitting, by the network appliance, the further SYN request to the server.

13. The method of claim 9, further comprising determining that the predetermined period of time has expired and, based on the determination, removing the network device from a white list, thereby flagging the network device as untrusted.

14. The method of claim 9, further comprising directing, by the network appliance, return data packets from the server directly to the network device without being processed by the network appliance.

15. The method of claim 9, wherein no SYN cookies are sent to the network device while the network device is whitelisted.

16. A system for determining that a network device is trusted, the system comprising:

a network module configured to;

receive a request from a network device to establish a data connection between the network device and a server, wherein the receiving of the request from the network device includes receiving, by the network module, from the network device, an initial synchronization (SYN) request to establish a Transmission Control Protocol (TCP) session; and

determine that the data connection associated with the TCP session between the network device and the server is trusted based on a determination that the network device is trusted;

a storage device operable to store a whitelist associated with a plurality of trusted network devices; and

a processor operable to;

determine that the network device is trusted;

based on the determination, associate the network device with the whitelist for a predetermined period of time;

wherein the determining, by the processor, that the network device is trusted includes;

receiving, by the network module, from the network device, an initial synchronization (SYN) request to establish a Transmission Control Protocol (TCP) session;

generating, by the processor module, a SYN cookie;

transmitting, by the network module, a SYN acknowledgement (ACK) including the SYN cookie to the network device;

responsive to the SYN ACK, receiving from the network device, by the network module, a network device ACK;

determining, by the processor, that the network device ACK includes the SYN cookie; and

based on the determining, validating, by the processor, authenticity of the SYN cookie; and

wherein the establishing, by the network module, the data connection by the network module between the network device and the server includes;

based on the determination that the network device is trusted, transmitting, by the network module, an unexpected identifier to the network device, the unexpected identifier causing the network device to terminate the TCP session and to start a new TCP session;

receiving, by the network module, from the network device, a further SYN request to establish the new TCP session;

determining, by the processor, that the network device is whitelisted; and

based on the determination, establishing, by the network module, a trusted data connection associated with the new TCP session between the network device and the server for the predetermined period of time.