Automatically detecting malicious computer network reconnaissance by updating state codes in a histogram

• US 7,356,587 B2
• Filed: 07/29/2003
• Issued: 04/08/2008
• Est. Priority Date: 07/29/2003
• Status: Active Grant

• Current Assignee: Trend Micro Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Boulanger, Alan D., Danford, Robert W., Himberger, Kevin D., Jeffries, Clark D., Singh, Raj K.
• Primary Examiner(s): Avellino, Joseph E.
• Application Number:
  US10/629,175
• Publication Number:
  US 20050027854A1
• Time in Patent Office: 1,715 Days
• Field of Search: 709223-229, 709217-219, 726 22- 23
• US Class Current: 709/224
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 63/166 at the transport layer

Abstract

A detection and response system that generates an Alert if unauthorized scanning is detected on a computer network that includes a look-up table to record state value corresponding to the sequence in which SYN, SYN/ACK and RST packets are observed. A set of algorithms executed on a processing engine adjusts the state value in response to observing the packets. When the state value reaches a predetermined value indicating that all three packets have been seen, the algorithm generates an Alert.

Claims

1. A method to detect unauthorized reconnaissance or scanning of a computer network comprising:

monitoring communications within the network;

detecting a predefined sequential triplet of TCP/IP protocol set packets flowing within said communications, each of the predefined sequential triplet packets comprising a source address field, a target device address field, a source port field and a target device port field, comprising;

providing a histogram in which states of the predefined sequence of packets are maintained, the histogram including a table partitioned into a first field in which source addresses of network devices are kept and a second field concatenated to the first field;

dynamically updating said histogram as selected ones of the predefined sequence of packets is detected by initializing or incrementing a state code field in response to an order in which packets in the predefined sequence of packets are detected;

concatenating a source address field, a target device address field, a source port field and a target device port field of a packet of the predefined sequential triplet into the table first and second fields as an ordered four-tuples;

hashing the ordered four-tuple; and

using the hashed ordered four-tuple as a histogram location index;

observing an initial SYN packet originating from a source address;

detecting a next sequential SYN/ACK packet issuing from a target device address in response to the SYN packet; and

detecting a last sequential RST packet originating from the source address in response to the SYN/ACK packet; and

issuing an alert indicating unauthorized scanning if the predefined sequence of packets are each relevant to the source address and if the state code field has an alert value.

2. The method of claim 1 wherein the issuing further includes sending a message to an administrator.

3. The method of claim 1 wherein the issuing further includes blocking future packets comprising the source address, the target device address and a target device port address.

4. The method of claim 1 wherein issuing further includes rate-limiting flows of packets comprising the source address.

5. The method of claim 1, wherein detecting the predefined sequential triplet comprises:

concatenating source address, target device address, source port and target device port fields of the SYN packet in a source address-target device address-source port-target device port first order four-tuple and initializing the state code field;

concatenating source address, target device address, source port and target device port fields of the SYN/ACK packet in a reflection of the first order in a target device address-source address-target device port-source port reflected order four-tuple and incrementing the initialized state code field; and

concatenating source address, target device address, source port and target device port fields of the RST packet in a first order four-tuple and incrementing the incremented state code field into the alert value.

6. The method of claim 5, comprising:

starting a purge time period;

purging the state code field upon a lapse of the purge time period.

7. The method of claim 5, wherein detecting the next sequential SYN/ACK packet comprises matching a look-up table key source address to the SYN/ACK source address field.

8. A method to deploy an intrusion detection system on a network device comprising:

providing an algorithm to detect a predefined sequential triplet of TCP/IP protocol packets;

providing a table to record at least one characteristic to identify network devices and state code corresponding to a sequence in which the predefined sequential triplet of packets are received, wherein each of the predefined sequential triplet packets comprise a source address field, a target device address field, a source port field and a target device port field;

dynamically updating a histogram by concatenating a source address field, a target device address field, a source port field and a target device port field of a packet of the predefined sequential triplet into a histogram table field as an ordered four-tuple;

hashing the ordered four-tuple; and

using the hashed ordered four-tuple as a histogram location index; and

generating an alert if the predefined triplet of packets is detected and the triplet packets are each relevant to a source address;

wherein the triplet comprises an initial SYN packet originating from the source address, a next sequential SYN/ACK packet issuing from a target device address in response to the SYN packet, and a last sequential RST packet originating from the source address in response to the SYN/ACK packet.

9. The method of claim 8 further comprising blocking future packets comprising the source address, the target device address and a target device port address.

10. The method of claim 8 further comprising rate-limiting flows of packets comprising the source address.

11. The method of claim 8 wherein detecting the predefined sequential triplet comprises:

concatenating source address, target device address, source port and target device port fields of the SYN packet in a source address-target device address-source port-target device port first order four-tuple and initializing the state code;

concatenating source address, target device address, source port and target device port fields of the SYN/ACK packet in a reflection of the first order in a target device address-source address-target device port-source port reflected order four-tuple and incrementing the initialized state code; and

concatenating source address, target device address, source port and target device port fields of the RST packet in a first order four-tuple and incrementing the incremented state code into an alert value.

12. The method of claim 11, comprising:

starting a purge time period;

purging the state code upon a lapse of the purge time period.

13. The method of claim 11, wherein detecting the next sequential SYN/ACK packet comprises matching a look-up table key source address to the SYN/ACK source address field.

14. A method to protect devices from malicious attacks launched on a computer network comprising:

providing on a device to be protected a software program that monitors packets, the software program includes a table containing codes whose values represent detection of one of the predefined set of packets and at least one source address associated with at least one of the codes, each of the predefined sequential triplet packets comprising a source address field, a target device address field, a source port field and a target device port field;

dynamically updating a histogram by concatenating a source address field, a target device address field, a source port field and a target device port field of a packet of the predefined sequential triplet into a histogram table field as an ordered four-tuple;

hashing the ordered four-tuple and using the hashed ordered four-tuple as a histogram location index; and

issuing an alert if a predefined sequential triplet of TCP/IP protocol packets are detected and the triplet packets are each relevant to a source address;

wherein the triplet comprises an initial SYN packet originating from the source address, a next sequential SYN/ACK packet issuing from a target device address in response to the SYN packet, and a last sequential RST packet originating from the source address in response to the SYN/ACK packet.

15. The method of claim 14 further comprising blocking future packets comprising the source address, the target device address and a target device port address.

16. The method of claim 14 further comprising rate-limiting flows of packets comprising the source address.

17. The method of claim 14 wherein detecting the predefined sequential triplet comprises:

concatenating source address, target device address, source port and target device port fields of the SYN packet in a source address-target device address-source port-target device port first order four-tuple and initializing a state code;

concatenating source address, target device address, source port and target device port fields of the SYN/ACK packet in a reflection of the first order in a target device address-source address-target device port-source port reflected order four-tuple and incrementing the initialized state code; and

concatenating source address, target device address, source port and target device port fields of the RST packet in a first order four-tuple and incrementing the incremented state code into an alert value.

18. The method of claim 17, comprising:

starting a purge time period; and

purging the state code upon a lapse of the purge time period.

19. The method of claim 17, wherein detecting the next sequential SYN/ACK packet comprises matching a look-up table key source address to the SYN/ACK source address field.