Systems and methods for correlating and distributing intrusion alert information among collaborating computer systems

• US 7,779,463 B2
• Filed: 06/09/2004
• Issued: 08/17/2010
• Est. Priority Date: 05/11/2004
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Parekh, Janak, Misra, Vishal, Locasto, Michael, Malkin, Tal, Keromytis, Angelos D., Stolfo, Salvatore J.
• Primary Examiner(s): Moazzami; Nasser
• Assistant Examiner(s): Louie; Oscar A
• Application Number:
  US10/864,226
• Publication Number:
  US 20050257264A1
• Time in Patent Office: 2,260 Days
• Field of Search: 726/22, 726/23, 726/24
• US Class Current: 726/22
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 63/1475 Passive attacks, e.g. eavesdropping or listening without modification of the traffic monitored
• CPC Class Codes: H04L 63/1491 using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment

Abstract

Systems and methods for correlating and distributing intrusion alert information among collaborating computer systems are provided. These systems and methods provide an alert correlator and an alert distributor that enable early signs of an attack to be detected and rapidly disseminated to collaborating systems. The alert correlator utilizes data structures to correlate alert detections and provide a mechanism through which threat information can be revealed to other collaborating systems. The alert distributor uses an efficient technique to group collaborating systems and then pass data between certain members of those groups according to a schedule. In this way data can be routinely distributed without generating excess traffic loads.

Claims

1. A method of detecting a threat to a computer system in a plurality of collaborating computer systems, the method comprising:

grouping a first computer system, a collaborating second computer system and other collaborating systems into groups so that each collaborating system in a group occupies a position in that group;

receiving, at the first computer system, a first one-way data structure from the collaborating second computer system, the first one-way data structure representing first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating second computer system such that the first data is hidden in the first one-way data structure, the receiving comprising periodically exchanging one-way data structures between the first computer system and the collaborating second computer system when in the same position in the different groups;

detecting, using an intrusion detection system of the first computer system, a second intrusion attempt;

storing second data relating to the second intrusion attempt in a second one-way data structure of the first computer system such that the second data is hidden in the second one-way data structure;

determining whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first data structure and the second data structure;

indicating that a threat is present if the second intrusion attempt is determined to correlate with the data received from the collaborating second computer system relating to the first intrusion attempt;

rotating the position occupied by each member of at least one of the groups according to a schedule; and

changing the schedule,

wherein the first one-way data structure is a bloom filter.

2. The method of claim 1, wherein the first data forms at least part of a profile of the first intrusion attempt.

3. The method of claim 2, further comprising responding to the threat based on the profile of the first intrusion attempt.

4. The method of claim 1, wherein the one-way data structures are exchanged through at least one proxy.

5. The method of claim 1, wherein the second intrusion attempt is one of a scan and probe.

6. The method of claim 1, wherein the second intrusion attempt is a blocked worm.

7. The method of claim 1, wherein the intrusion detection system is a honey pot.

8. The method of claim 1, wherein storing data relating to the second intrusion attempt in the second one-way data structure of the first computer system comprises:

forming a hash of the second data relating to the second intrusion attempt;

using the hash as an index to the bloom filter; and

setting corresponding bits in the bloom filter based on the index.

9. The method of claim 8, wherein forming the hash comprises using at least one of the SHA-1 and the MD-5 hashing algorithms.

10. The method of claim 1, wherein storing data relating to the second intrusion attempt in the second one-way data structure of the first computer system comprises:

forming a hash of the second data relating to the second intrusion attempt;

selecting at least one portion of the hash;

using the at least one portion of the hash as an index to the bloom filter; and

setting corresponding bits in the bloom filter based on the index.

11. The method of claim 10, wherein forming the hash comprises using at least one of the SHA-1 and the MD-5 hashing algorithms.

12. The method of claim 1, wherein indicating that the threat is present comprises outputting an IP address of the source of the threat.

13. The method of claim 1, wherein indicating that the threat is present comprises causing a firewall to block traffic from an IP address of the source of the threat.

14. The method of claim 1, wherein indicating that the threat is present comprises outputting a signature associated with the threat.

15. The method of claim 1, wherein determining whether the second intrusion attempt correlates with the first intrusion attempt comprises:

comparing bits of the first one-way data structure to corresponding bits of the second one-way data structure; and

indicating a correlation if at least a predetermined number of corresponding bits are set to the same bit value.

16. A method of detecting a threat to a computer system in a plurality of collaborating computer systems, the method comprising:

grouping a first computer system, a collaborating second computer system and other collaborating systems into groups so that each collaborating system in a group occupies a position in that group;

receiving, at the first computer system, a first data structure from the collaborating second computer system, the first data structure representing first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating second computer system such that the first data is hidden in the first data structure, the receiving comprising periodically exchanging one-way data structures between the first computer system and the collaborating second computer system when in the same position in the different groups;

receiving an indication of a characteristic of the collaborating second computer system;

detecting, using an intrusion detection system of the first computer system, a second intrusion attempt;

storing second data relating to the second intrusion attempt in a second data structure of the first computer system such that the second data is hidden in the second data structure;

determining whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first data structure and the second data structure;

based on the characteristic of the collaborating second computer system, indicating that a threat is present if the second intrusion attempt is determined to correlate with the first intrusion attempt;

rotating the position occupied by each member of at least one of the groups according to a schedule; and

changing the schedule,

wherein the first data structure is a bloom filter.

17. The method of claim 16, wherein the first data and the characteristic received from the collaborating second computer system form at least part of a profile of the first intrusion attempt.

18. The method of claim 17, further comprising responding to the threat based on the profile of the first intrusion attempt.

19. The method of claim 16, wherein the characteristic comprises the type of user of the collaborating system.

20. The method of claim 19, wherein the characteristic indicates that the user is a bank.

21. The method of claim 19, wherein the characteristic indicates that the user is a university.

22. The method of claim 16, wherein the data structures are exchanged through at least one proxy.

23. The method of claim 16, wherein the second intrusion attempt is one of a scan and probe.

24. The method of claim 16, wherein the second intrusion attempt is a blocked worm.

25. The method of claim 16, wherein the intrusion detection system is a honey pot.

26. The method of claim 16, wherein storing second data relating to the second intrusion attempt in a second data structure of the first computer system comprises:

forming a hash of the second data relating to the second intrusion attempt;

using the hash as an index to the bloom filter; and

determining whether a bit setting corresponding bits in the bloom filter based on the index.

27. The method of claim 26, wherein forming the hash comprises using at least one of the SHA-1 and the MD-5 hashing algorithms.

28. The method of claim 16, wherein storing second data relating to the second intrusion attempt in the second data structure of the first computer system comprises:

forming a hash of the second data relating to the second intrusion attempt;

selecting at least one portion of the hash;

using the at least one portion of the hash as an index to the bloom filter; and

setting corresponding bits in the bloom filter based on the index.

29. The method of claim 28, wherein forming the hash comprises using at least one of the SHA-1 and the MD-5 hashing algorithms.

30. The method of claim 16, wherein indicating that the threat is present comprises outputting an IP address of the source of the threat.

31. The method of claim 16, wherein indicating that the threat is present comprises causing a firewall to block traffic from an IP address of the source of the threat.

32. The method of claim 16, wherein indicating that the threat is present comprises outputting a signature associated with the threat.

33. A system configured to detect intrusion attempts in a computer system among a plurality of collaborating computer systems, comprising:

at least one server that;

groups a collaborating computer system and other collaborating systems into groups so that each collaborating system in a group occupies a position in that group;

receives a first one-way data structure from the collaborating computer system, wherein the first one-way data structure represents first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating computer system such that the first data is hidden in the first one-way data structure, wherein the at least one server in receiving at least periodically exchanges one-way data structures with the collaborating computer system based on the position of the collaborating computer system;

detects a second intrusion attempt;

stores second data relating to the second intrusion attempt in a second one-way data structure such that the second data is hidden in the second one-way data structure;

determines whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first one-way data structure and the second one-way data structure; and

indicates that a threat is present if the second intrusion attempt is determined to correlate with the first intrusion attempt;

rotates the position occupied by each member of at least one of the groups according to a schedule; and

changes the schedule,

wherein the first one-way data structure is a bloom filter.

34. The system of claim 33, wherein the at least one server, in determining whether the second data relating to the second intrusion attempt correlates with the data received from the collaborating second computer system relating to the first intrusion attempt, also:

compares bits of the first data structure to corresponding bits of the second one-way data structure; and

indicates a correlation if at least a predetermined number of corresponding bits are set to the same bit value.

35. A system configured to detect intrusion attempts in a computer system among a plurality of collaborating computer systems, comprising:

at least one server that;

groups a collaborating computer system and other collaborating systems into groups so that each collaborating system in a group occupies a position in that group;

receives a first data structure from the collaborating computer system, wherein the first data structure represents first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating computer system such that the first data is hidden in the first data structure, wherein the at least one server in receiving at least periodically exchanges data structures with the collaborating computer system based on the position of the collaborating computer system;

receives an indication of a characteristic of the collaborating computer system;

detects a second intrusion attempt;

stores second data relating to the second intrusion attempt in a second data structure such that the second data is hidden in the second data structure;

determines whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first data structure and the second data structure; and

based on the characteristic of the collaborating computer system, indicates that a threat is present if the second intrusion attempt correlates with the first intrusion attempt;

rotates the position occupied by each member of at least one of the groups according to a schedule; and

changes the schedule,

wherein the first data structure is a bloom filter.

36. A method of detecting a threat to a computer system in a plurality of collaborating computer systems, the method comprising:

receiving, at a first computer system, a first one-way data structure from a collaborating second computer system, the first one-way data structure representing first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating second computer system such that the first data is hidden in the first one-way data structure, wherein the first one-way data structure is a bloom filter that is used to hide information regarding the first intrusion attempt;

detecting, using an intrusion detection system of the first computer system, a second intrusion attempt;

storing second data relating to the second intrusion attempt in a second one-way data structure of the first computer system such that the second data is hidden in the second one-way data structure, wherein the storing comprises;

forming a hash of the second data relating to the second intrusion attempt using at least one of the SHA-1 and the MD-5 hashing algorithms;

using the hash as an index to the bloom filter; and

setting corresponding bits in the bloom filter based on the index;

determining whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first data structure and the second data structure; and

indicating that a threat is present if the second intrusion attempt is determined to correlate with the data received from the collaborating second computer system relating to the first intrusion attempt.

37. The method of claim 36, wherein the first data forms at least part of a profile of the first intrusion attempt.

38. The method of claim 37, further comprising responding to the threat based on the profile of the first intrusion attempt.

39. The method of claim 36, wherein receiving the first one-way data structure from the collaborating second computer system comprises periodically exchanging one-way data structures with the collaborating second computer system.

40. The method of claim 39, wherein the one-way data structures are exchanged through at least one proxy.

41. The method of claim 39, further comprising:

grouping the collaborating second computer system and other collaborating systems into groups so that each collaborating system in a group occupies a position in that group;

exchanging one-way data structures between collaborating systems in the same position in the different groups; and

rotating the position occupied by each member of at least one of the groups according to a schedule.

42. The method of claim 41, further comprising changing the schedule.

43. The method of claim 36, wherein the second intrusion attempt is one of a scan and probe.

44. The method of claim 36, wherein the second intrusion attempt is a blocked worm.

45. The method of claim 36, wherein the intrusion detection system is a honey pot.

46. The method of claim 36, wherein the hash is used as an index to the bloom filter by:

selecting at least one portion of the hash; and

using the at least one portion of the hash as an index to the bloom filter.

47. The method of claim 36, wherein indicating that the threat is present comprises outputting an IP address of the source of the threat.

48. The method of claim 36, wherein indicating that the threat is present comprises causing a firewall to block traffic from an IP address of the source of the threat.

49. The method of claim 36, wherein indicating that the threat is present comprises outputting a signature associated with the threat.

50. The method of claim 36, wherein determining whether the second intrusion attempt correlates with the first intrusion attempt comprises:

comparing bits of the first one-way data structure to corresponding bits of the second one-way data structure; and

indicating a correlation if at least a predetermined number of corresponding bits are set to the same bit value.

51. A method of detecting a threat to a computer system in a plurality of collaborating computer systems, the method comprising:

receiving, at a first computer system, a first data structure from a collaborating second computer system, the first data structure representing first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating second computer system such that the first data is hidden in the first data structure, wherein the first data structure is a bloom filter that is used to hide information regarding the first intrusion attempt;

receiving an indication of a characteristic of the collaborating second computer system;

detecting, using an intrusion detection system of the first computer system, a second intrusion attempt;

storing second data relating to the second intrusion attempt in a second data structure of the first computer system such that the second data is hidden in the second data structure, wherein the storing comprises;

forming a hash of the second data relating to the second intrusion attempt using at least one of the SHA-1 and the MD-5 hashing algorithms;

using the hash as an index to the bloom filter; and

setting corresponding bits in the bloom filter based on the index;

determining whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first data structure and the second data structure; and

based on the characteristic of the collaborating second computer system, indicating that a threat is present if the second intrusion attempt is determined to correlate with the first intrusion attempt.

52. The method of claim 51, wherein the first data and the characteristic received from the collaborating second computer system form at least part of a profile of the first intrusion attempt.

53. The method of claim 52, further comprising responding to the threat based on the profile of the first intrusion attempt.

54. The method of claim 51, wherein the characteristic comprises the type of user of the collaborating system.

55. The method of claim 54, wherein the characteristic indicates that the user is a bank.

56. The method of claim 54, wherein the characteristic indicates that the user is a university.

57. The method of claim 51, wherein receiving the first data structure from the collaborating second computer system comprises periodically exchanging data structures with the collaborating second computer system.

58. The method of claim 57, wherein the data structures are exchanged through at least one proxy.

59. The method of claim 57, further comprising:

grouping the collaborating second computer system and other collaborating systems into groups so that each collaborating system in a group occupies a position in that group;

exchanging data structures between collaborating systems in the same position in the different groups; and

rotating the position occupied by each member of at least one of the groups according to a schedule.

60. The method of claim 59, further comprising changing the schedule.

61. The method of claim 51, wherein the second intrusion attempt is one of a scan and probe.

62. The method of claim 51, wherein the second intrusion attempt is a blocked worm.

63. The method of claim 51, wherein the intrusion detection system is a honey pot.

64. The method of claim 51, wherein the hash is used as an index to the bloom filter by:

selecting at least one portion of the hash; and

using the at least one portion of the hash as an index to the bloom filter.

65. The method of claim 51, wherein indicating that the threat is present comprises outputting an IP address of the source of the threat.

66. The method of claim 51, wherein indicating that the threat is present comprises causing a firewall to block traffic from an IP address of the source of the threat.

67. The method of claim 51, wherein indicating that the threat is present comprises outputting a signature associated with the threat.

68. A system configured to detect intrusion attempts in a computer system among a plurality of collaborating computer systems, comprising:

at least one server that;

receives a first one-way data structure from a collaborating computer system, wherein the first one-way data structure represents first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating computer system such that the first data is hidden in the first one-way data structure, wherein the first one-way data structure is a bloom filter that is used to hide information regarding the first intrusion attempt;

detects a second intrusion attempt;

stores second data relating to the second intrusion attempt in a second one-way data structure such that the second data is hidden in the second one-way data structure, wherein in storing the at least one server at least;

forms a hash of the second data relating to the second intrusion attempt using at least one of the SHA-1 and the MD-5 hashing algorithms;

uses the hash as an index to the bloom filter; and

sets corresponding bits in the bloom filter based on the index;

determines whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first one-way data structure and the second one-way data structure; and

indicates that a threat is present if the second intrusion attempt is determined to correlate with the first intrusion attempt.

69. The system of claim 68, wherein the at least one server, in determining whether the second data relating to the second intrusion attempt correlates with the data received from the collaborating second computer system relating to the first intrusion attempt, also:

compares bits of the first one-way data structure to corresponding bits of the second one-way data structure; and

indicates a correlation if at least a predetermined number of corresponding bits are set to the same bit value.

70. A system configured to detect intrusion attempts in a computer system among a plurality of collaborating computer systems, comprising:

at least one server that;

receives a first data structure from a collaborating computer system, wherein the first data structure represents first data relating to a first intrusion attempt detected by an intrusion detection system of the collaborating computer system such that the first data is hidden in the first data structure, wherein the first data structure is a bloom filter that is used to hide information regarding the first intrusion attempt;

receives an indication of a characteristic of the collaborating computer system;

detects a second intrusion attempt;

stores second data relating to the second intrusion attempt in a second data structure such that the second data is hidden in the second data structure, wherein in storing the at least one server at least;

forms a hash of the second data relating to the second intrusion attempt using at least one of the SHA-1 and the MD-5 hashing algorithms;

uses the hash as an index to the bloom filter; and

sets corresponding bits in the bloom filter based on the index;

determines whether the second intrusion attempt correlates with the first intrusion attempt by comparing the first data structure and the second data structure; and

based on the characteristic of the collaborating computer system, indicates that a threat is present if the second intrusion attempt correlates with the first intrusion attempt.