System and method for managing denial of service attacks

• US 6,851,062 B2
• Filed: 09/27/2001
• Issued: 02/01/2005
• Est. Priority Date: 09/27/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Murphy, Thomas E. Jr., Krissell, Daniel L., Stevens, Jeffrey S., Orzel, Francine M., Hartmann, Richard G., Rieth, Paul F.
• Primary Examiner(s): HUA, LY
• Application Number:
  US09/965,074
• Publication Number:
  US 20030061510A1
• Time in Patent Office: 1,223 Days
• Field of Search: 713/200, 713/201, 713/153, 709/206, 709/207, 709/228, 709/227, 709/229, 709/226, 709/225, 380/287
• US Class Current: 726/22
• CPC Class Codes: H04L 63/04 for providing a confidential data exchange among entities communicating through data packet networks
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/0823 using certificates cryptographic mechanisms or cryptographic arrangements for entity authentication involving certificates H04L9/3263
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 63/18 using different networks or channels, e.g. using out of band channels cryptographic mechanisms or cryptographic arrangements for key distribution involving distinctive intermediate devices or communication paths H04L9/0827; cryptographic mechanisms or cryptographic arrangements for authentication using a plurality of channels H04L9/3215

Abstract

A system and method for monitoring and controlling the total number of SSL port resources that are allowed to be tied up by a malicious or inept client making multiple requests from a single IP address. Smart SSL handshake timeout detection is used to track and deny service to any SSL clients that do denial of service (DOS) attacks.

Claims

1. A method for operating an application in the application layer of a server, comprising the steps of:

maintaining in said application layer of said server a record of the number of pending service requests from a client; and

upon said application receiving a request for service from said client when said client has more than a configurable number of pending service requests denying service to said client.

2. The method of claim 1, further comprising the steps of:

determining when the number of times that negotiations for a service connection from said client exceed a timeout value; and

upon said number exceeding a trigger threshold, denying service to said client.

3. The method of claim 2, further comprising the steps of:

processing a request at said server from said client to enter into secure sockets layer (SSL) negotiations;

responsive to failure of said negotiations to complete, registering a denial of service to a denial of service list; and

responsive to registering a client to said denial of service list, closing any connection with said client while not processing any pending request from said client.

4. The method of claim 3, further comprising the steps of:

maintaining a cache copy and a server copy of said denial of service list;

selectively logging to said server copy clients restored to service by administrative action; and

responsive to said request from said client for server resources, checking said cache copy to determine if service is to be denied to said client and, responsive to determining from said cache copy that service is to be denied, checking said server copy of said denial of service list to determine if said client is to be granted service.

5. The method of claim 4, further comprising the steps of:

responsive to determining from said cache copy of said denial of service list that said client is to be denied service, refreshing said cache copy from said server copy of said denial of service list, and re-checking said cache copy to determine if service is to be granted to said client.

6. The method of claim 2, further comprising the steps of:

responsive to a request for service from said client which is executed to completion within a timeout period, decrementing the number of outstanding connection requests from said client.

7. The method of claim 6, further comprising the step of:

responsive to a request for service from said client which is not executed to completion within said timeout period, incrementing the number of outstanding connection requests from said client.

8. The method of claim 7, further comprising the steps of:

further responsive to a request for service from said client which does not complete within said timeout period, recalculating said timeout period.

9. The method of claim 8, said recalculating step further comprising:

determining for said client an average round trip time value (RTT); and

setting said timeout period as a function of said average round trip value.

10. The method of claim 9, further comprising the step of weighting said round trip value selectively to favor historical or more recent values of said round trip time.

11. A method for operating a server to manage denial of service attacks, comprising the steps of:

receiving from a client a connection request;

responsive to said connection request, determining if said client is identified for denial of service and, if so, closing said connection request;

responsive to said client not being identified for denial of service, incrementing a count M for said client in an outstanding negotiation list;

responsive to said count M being incremented to a value greater than a trigger value T, closing said connection request;

responsive to said count M being incremented to a value equal to or less than said trigger value T, entering into a service negotiation with said client;

responsive to said service negotiation successfully completing within a timeout period N, deleting said client from said outstanding negotiation list and processing said request for service;

responsive to said service negotiation not successfully completing within said timeout period N, incrementing said count M, recalculating said timeout period N and closing said connection request;

responsive to said count M exceeding a trigger value, identifying said client for denial of service.

12. A server system, comprising:

an application server;

an access denied list (ADL) of client addresses to be denied service by an application executing in said server;

a pending negotiation list for identifying clients negotiating server services with said application server;

a counter for maintaining a count M of the number of pending requests for services from a client in said pending negotiation list;

a trigger value T;

a timeout period value N;

said application server being responsive to a request for service from a client to be denied service for closing a secure sockets layer (SSL) connection request from said client;

said server being responsive to a request for service from a client not to be denied service for adding said request for service to said pending negotiation list; and

said server being responsive to a SSL request from a client having more than T pending requests for service for closing said connection request from said client.

13. The system of claim 12, further comprising:

said server being responsive to a request for service from a client having T or less pending requests for service for entering into negotiations with said client.

14. The system of claim 13, further comprising:

said server being responsive to said negotiations exceeding said timeout period value N, for incrementing said count M, recalculating said timeout period value N, and closing said connection request from said client; and

further responsive to said count M exceeding said trigger value T for identifying said client for denial of service.

15. The system of claim 12, said server further for negotiating a secure socket layer (SSL) connection with said client responsive to a SSL HELLO(SSL_INIT)request for service from said client that requires said client to respond.

16. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating an application in the application layer of a server, said method steps comprising:

maintaining in said application layer a record of the number of pending service requests from a client; and

upon said application receiving a request for service from said client when said client has more than a configurable number of pending service requests denying service to said client.

17. The program storage device of claim 16, said method steps further comprising:

determining when the number of times that negotiations for a service connection from said client exceed a timeout value; and

upon said number exceeding a trigger threshold, denying service to said client.

18. The program storage device of claim 17, said method steps further comprising:

registering a denial of service to a denial of service list;

responsive to registering a client to said denial of service list, closing any connection with said client while not processing any pending request from said client;

maintaining a cache copy and a server copy of said denial of service list, said server copy for keeping a master denial of service list for clients known to have too many requests for connection pending without responding and for selectively restoring access by administrator action, and said cache copy for tracking connection requests from a given client;

responsive to said request from said given client for server resources, checking said cache copy to determine if service is to be denied to said given client and, responsive to determining from said cache copy that service is to be denied, checking said server copy of said denial of service list to determine if said client is to be granted service;

responsive to determining from said cache copy of said denial of service list that said client is to be denied service, refreshing said cache copy from said server copy of said denial of service list, and re-checking said cache copy to determine if service is to be granted to said client;

responsive to a request for service from said client which completes within a timeout period, decrementing the number of outstanding connection requests from said client;

responsive to a request for service from said client which does not complete within said timeout period, incrementing the number of outstanding connection requests from said client;

further responsive to a request for service from said client which does not complete within said timeout period, recalculating said timeout period;

determining for said client an average round trip time value (RTT);

setting said timeout period as a function of said average round trip value; and

weighting said round trip value selectively to favor historical or more recent values of said round trip time.

19. A computer program product or computer program element for executing method steps comprising:

receiving from a client a connection request;

responsive to said connection request, determining if said client is identified for denial of service and, if so, closing said connection request;

responsive to said client not being identified for denial of service, incrementing a count M for said client in an outstanding negotiation list;

responsive to said count M being incremented to a value greater than a trigger value T, closing said connection request;

responsive to said count M being incremented to a value equal to or less than said trigger value T, entering into a service negotiation with said client;

responsive to said service negotiation successfully completing within a timeout period N, deleting said client from said outstanding negotiation list and processing said request for service;

responsive to said service negotiation not successfully completing within said timeout period N, incrementing said count M, recalculating said timeout period N and closing said connection request;

responsive to said count M exceeding a trigger value, identifying said client for denial of service.

20. A method for operating a server, comprising the steps of:

maintaining a record of the number of pending service requests from a client;

upon receiving a request for service from said client when said client has more than a configurable number of pending service requests denying service to said client;

determining when the number of times that negotiations for a service connection from said client exceed a timeout value, and upon said number exceeding a trigger threshold, denying service to said client;

registering a denial of service to a denial of service list;

responsive to registering a client to said denial of service list, closing any connection with said client while not processing any pending request from said client;

maintaining a cache copy and a server copy of said denial of service list;

said server copy for keeping a master denial of service list of clients known to have too many requests for connection pending without responding and for selectively restoring access by administrator action;

said cache copy for tracking connection requests from a given client;

responsive to said request from said given client for server resources, checking said cache copy to determine if service is to be denied to said given client and, responsive to determining from said cache copy that service is to be denied, checking said server copy of said denial of service list to determine if said given client is to be granted service; and

responsive to determining from said cache copy of said denial of service list that said client is to be denied service, refreshing said cache copy from said server copy of said denial of service list, and re-checking said cache copy to determine if service is to be granted to said client.

21. The method of claim 20, further comprising the steps of:

responsive to a request for service from said given client which is executed to completion within a timeout period, decrementing the number of outstanding connection requests from said given client.

22. The method of claim 21, further comprising the step of:

responsive to a request for service from said given client which is not executed to completion within said timeout period, incrementing the number of outstanding connection requests from said given client.

23. The method of claim 22, further comprising the steps of:

further responsive to a request for service from said given client which does not complete within said timeout period, recalculating said timeout period.

24. The method of claim 23, said recalculating step further comprising:

determining for said given client an average round trip time value (RTT); and

setting said timeout period as a function of said average round trip value.

25. The method of claim 24, further comprising the step of weighting said round trip value selectively to favor historical or more recent values of said round trip time.

26. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating a server, said method steps comprising:

maintaining a record of the number of pending service requests from a client;

upon receiving a request for service from said client when said client has more than a configurable number of pending service requests denying service to said client;

determining when the number of times that negotiations for a service connection from said client exceed a timeout value, and upon said number exceeding a trigger threshold, denying service to said client;

registering a denial of service to a denial of service list;

responsive to registering a client to said denial of service list, closing any connection with said client while not processing any pending request from said client;

maintaining a cache copy and a server copy of said denial of service list, said server copy for keeping a master denial of service list for clients known to have too many requests for connection pending without responding and for selectively restoring access by administrator action, and said cache copy for tracking connection requests from a given client;

responsive to said request from said given client for server resources, checking said cache copy to determine if service is to be denied to said given client and, responsive to determining from said cache copy that service is to be denied, checking said server copy of said denial of service list to determine if said client is to be granted service;

responsive to determining from said cache copy of said denial of service list that said client is to be denied service, refreshing said cache copy from said server copy of said denial of service list, and re-checking said cache copy to determine if service is to be granted to said client;

responsive to a request for service from said client which completes within a timeout period, decrementing the number of outstanding connection requests from said client;

responsive to a request for service from said client which does not complete within said timeout period, incrementing the number of outstanding connection requests from said client;

further responsive to a request for service from said client which does not complete within said timeout period, recalculating said timeout period;

determining for said client an average round trip time value (RTT);

setting said timeout period as a function of said average round trip value; and

weighting said round trip value selectively to favor historical or more recent values of said round trip time.