Blocking unidentified encrypted communication sessions

• US 8,341,724 B1
• Filed: 12/19/2008
• Issued: 12/25/2012
• Est. Priority Date: 12/19/2008
• Status: Active Grant

• Current Assignee: Juniper Networks Incorporated
• Original Assignee: Juniper Networks Incorporated
• Inventors: Burns, Bryan, Sukhanov, Vladimir
• Primary Examiner(s): Hoffman, Brandon
• Assistant Examiner(s): SONG, HEE K
• Application Number:
  US12/339,948
• Time in Patent Office: 1,467 Days
• Field of Search: 726/12, 726/13, 726/22, 726/23, 380/28, 713/151, 370/235, 370/252, 709/224
• US Class Current: 726/13
• CPC Class Codes: H04L 2463/144 Detection or countermeasures against botnets
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

Techniques are described for blocking unidentified encrypted communication sessions. In one embodiment, a device includes an interface to receive a packet, an application identification module to attempt to identify an application associated with the packet, an encryption detection module to determine whether the packet is encrypted when the application identification module is unable to identify an application associated with the packet, and an attack detection module to determine whether the packet is associated with a network attack, to forward the packet when the packet is not associated with a network attack, and to take a response when the packet is associated with a network attack, wherein the encryption detection module sends a message to the attack detection module that indicates whether the packet is encrypted, wherein when the message indicates that packet is encrypted, the attack detection module determines that the packet is associated with a network attack.

Claims

1. A method comprising:

receiving a network packet;

determining whether the packet represents a key exchange,

identifying a key exchange for a communication session for the packet,

making a record of the key exchange for the communication session after identifying the key exchange,

using an application-layer header of the packet, determining whether the packet is associated with an identifiable network application;

when the packet is not determined to be associated with an identifiable network application, determining whether data in the packet is encrypted by calculating a randomness value of the packet from a payload of the packet that includes the application-layer header and an application-layer payload and determining that the packet is encrypted when the randomness value exceeds a randomness threshold; and

when the data in the packet is determined to be encrypted, executing a programmed response, wherein executing a programmed response comprises;

determining whether a key exchange has been recorded for the communication session associated with the packet; and

when a key exchange has not been recorded, dropping the packet, and

when a key exchange has been recorded, forwarding the packet.

2. The method of claim 1, wherein determining whether data in the packet is encrypted comprises applying the Wald-Wolfowitz runs test to the payload of the packet that includes the application-layer header and the application-layer payload.

3. The method of claim 2, wherein applying the Wald-Wolfowitz runs test comprises:

determining a size value for the packet;

determining a mean value for the bytes in the packet;

determining a number of bytes with values in excess of the mean in the packet;

determining a number of bytes with values below and equal to the mean in the packet;

determining a number of runs of bytes in the packet;

determining an expected number of runs for a general packet having the size value of the packet, the number of bytes with values in excess of the mean, and the number of bytes with values below and equal to the mean;

determining a variance for the general packet; and

determining that the data in the packet is encrypted when the absolute value of the difference between the number of runs in the packet and the expected number of runs for the general packet, divided by the square root of the variance, is within a randomness threshold.

4. The method of claim 3,

wherein determining an expected number of runs comprises determining that the expected number of runs is equal to two times the number of bytes in excess of the mean times the number of bytes below and equal to the mean divided by the size value, plus one;

and

wherein determining a variance comprises determining that the variance is equal to a quantity, the quantity being the expected number of runs minus one times the expected number of runs minus two, the quantity then being divided by a second quantity, the second quantity being the size value minus one.

5. The method of claim 1, wherein determining whether data in the packet is encrypted comprises applying at least one filter to the data.

6. The method of claim 5, further comprising:

displaying an administrator interface; and

receiving, with the administrator interface, one or more instructions to add, remove, and edit the at least one filter.

7. The method of claim 1, wherein executing a programmed response comprises sending an alert.

8. The method of claim 1, further comprising identifying the packet as belonging to a communication session between a server and a client.

9. The method of claim 1, further comprising, when the data in the packet is determined to not be encrypted, forwarding the packet.

10. The method of claim 1, further comprising, when the packet is determined to be associated with an identifiable network application, forwarding the packet.

11. The method of claim 10, wherein forwarding the packet further comprises:

determining whether the identifiable network application is a legitimate network application;

when the identifiable network application is determined to be a legitimate network application, forwarding the packet; and

when the identifiable network application is determined not be a legitimate network application, dropping the packet.

12. A network device comprising:

an interface to receive a packet;

an application identification module to attempt to identify an application associated with the packet;

an encryption detection module to determine whether the packet is encrypted when the application identification module is unable to identify an application associated with the packet; and

an attack detection module to determine whether the packet is associated with a network attack, to forward the packet when the packet is not associated with a network attack, and to take a response when the packet is associated with a network attack,

wherein the encryption detection module sends a message to the attack detection module that indicates whether the packet is encrypted, wherein when the message indicates that packet is encrypted, the attack detection module determines that the packet is associated with a network attack,

wherein the attack detection module determines whether the packet represents a key exchange, records whether a key exchange has been detected for a communication session, and, when a key exchange has not been detected for a communication session associated with the packet and the encryption detection module has determined that the packet is encrypted, drops the packet, and when a key exchange has been detected for the communication session associated with the packet and the encryption detection module has determined that the packet is encrypted, forwards the packet.

13. The device of claim 12, wherein the encryption detection module implements the Wald-Wolfowitz runs test and applies the Wald-Wolfowitz runs test to the packet to determine whether the packet is encrypted.

14. The device of claim 12, wherein the encryption detection module applies at least one filter to the packet to determine whether the packet is encrypted.

15. A non-transitory computer-readable medium comprising instructions that cause a programmable processor to:

receive a network packet;

determine whether the packet represents a key exchange,

identify a key exchange for a communication session for the packet,

make a record of the key exchange for the communication session after identifying the key exchange,

determine whether the packet is associated with a network application using an application-layer header of the packet;

determine whether data in the packet is encrypted, when the packet is not determined to be associated with an identifiable network application, by calculating a randomness value of the packet from a payload of the packet that includes the application-layer header and an application-layer payload and determining that the packet is encrypted when the randomness value exceeds a randomness threshold; and

when the data in the packet is determined to be encrypted, execute a response, and when the data in the packet is determined not to be encrypted, forward the packet, wherein the response comprises;

determining, when the data in the packet is determined to be encrypted, whether a key exchange has been recorded for the communication session associated with the packet; and

when a key exchange has not been recorded, dropping the packet, and

when a key exchange has been recorded, forwarding the packet.

16. The non-transitory computer-readable medium of claim 15, further comprising instructions to apply the Wald-Wolfowitz runs test to the packet to determine whether the data in the packet is encrypted.

17. The non-transitory computer-readable medium of claim 15, further comprising instructions to apply at least one filter to the packet to determine whether the data in the packet is encrypted.