Systems and methods for detecting a compromised network

• US 20050157662A1
• Filed: 01/21/2005
• Published: 07/21/2005
• Est. Priority Date: 01/20/2004
• Status: Abandoned Application

• Current Assignee: Intrusic, Inc.
• Original Assignee: Intrusic, Inc.
• Inventors: Bingham, Justin, Zatko, Peiter
• Application Number:
  US11/041,772
• Publication Number:
  US 20050157662A1
• Time in Patent Office: Days
• US Class Current: 370/254
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Systems and methods are disclosed for monitoring data transmissions on a network and detecting compromised networks. The systems and methods monitor communications involving network hosts and analyze the communications in view of the business function of the hosts. In certain embodiments the analysis is performed by associating a set of rules of operation for the sessions, hosts, and/or environment, and analyzing data packet transmissions to ascertain violations of the rules.

Claims

1. A method for detecting a compromised host in a network, comprising:

identifying hosts on a network,

identifying model session rules expected to be followed during sessions in which one or more host participates,

monitoring data packet transmissions between hosts to identify violations of the model session rules, and

identifying a compromise if at least one violation is identified in a session involving a host.

2. The method of claim 1, wherein the at least one violation includes two or more violations.

22. The method of claim 1, wherein monitoring data packet transmissions includes using a tap or span port to copy data packets transmitted on the network, bundling the copied data packets into groups based on network protocol identified in the data packet headers, associating the data packets in the groups according to unique sessions in which the data packets were transmitted.

23. The method of claim 22, further comprising compiling a profile of session information for each host on the network based on the data packets transmitted in the sessions.

25. A method for validating a detected compromise on a network, comprising:

applying the method of claim 1 to identify a host involved in a session that violates a model session rule,

identifying model host rules of expected operation for the host,

analyzing the data packet transmissions involving the host to identify violations of the model host rules, and

validating an identified compromise if at least one violation of the model host rules is identified.

26. A method for validating a detected compromise on a network, comprising:

applying the method of claim 1 to identify a host involved in a session that violates a model session rule,

identifying a model environment rule for the host,

analyzing the data packet transmissions involving the host to identify violations of the model environment rule, and

validating an identified compromise if at least one violation of the model environment rule is identified.

27. A method for identifying a compromised network, comprising applying the method of claim 1 or claim 4, and applying validation studies to reduce at least one false positive, identify at least one false negative, or both.

3. A method for detecting a compromised host in a network, comprising:

identifying hosts on the network,

identifying model host rules of expected operation for one or more hosts within the network,

monitoring data packet transmissions involving a host to identify violations of the model host rules, and

identifying a compromise if at least one violation of the model host rules is identified.

4. A method for detecting a compromised host in a network, comprising:

collecting data packet transmissions involving hosts on the network,

identifying model session rules expected to be followed during sessions involving the hosts,

for each host identifying model host rules of expected operation for the host and an environment rule for the host,

using the data packet transmissions to identify violations of the model session rules, model host rules, and model environment rules, and

identifying a compromise if the host is involved in at least one rule violation.

5. The method of claim 4, wherein a compromise is identified if the host is involved in more than one rule violation.

6. The method of claim 4, wherein the network is an internal network.

7. The method of claim 4, further comprising providing a report setting forth one or more identified violations.

8. The method of claim 4, further comprising analyzing the data packet transmissions to identify other communication typical of an intruder.

9. The method of claim 4, wherein a violation of a host rule includes a host changing roles on a network.

10. The method of claim 4, wherein a violation of the environment rule includes participating in one or more mirrored sessions.

11. The method of claim 4, wherein the host is a server, client, or network device.

12. The method of claim 4, wherein the host is operated by a malicious insider.

13. The method of claim 4, wherein the compromise is caused by a party that has gained unauthorized accessed to the network.

14. The method of claim 4, further comprising monitoring data packets sent and data packets received by a host through the network after identifying the host as being compromised.

15. The method of claim 4, wherein network communications are monitored at a single source on the network.

24. A method for repairing a network having a compromised host, comprising

identifying a compromised host by the method of claim 4,

stopping network traffic in and out of the compromised host, and

allowing all uncompromised hosts on the network to continue functioning without interruption.

16. A method of reducing false positive results when identifying a network compromise, comprising:

monitoring data packet transmissions between hosts on a network,

identifying model session rules expected to be followed during sessions involving the hosts,

identifying model host rules of expected operation for the hosts,

using the data packet transmissions to identify violations of the model session rules,

using the data packet transmissions to identify violations of the model host rules, and

identifying a compromise if a particular host is involved in at least one rule violation.

17. The method of claim 16, wherein a compromise is identified if the particular host is involved in more than one rule violation.

18. The method of claim 16, further comprising identifying a model environment rule for each host and using the data packet transmissions to identify violations by a host of its model environment rule.

19. The method of claim 18, further comprising using the data packet transmissions to identify instances where a host engages in communication typical of an intruder.

20. The method of claim 19, wherein a compromise is detected if the host is either involved in more than one rule violation or is involved in one rule violation along with communication typical of an intruder.

21. The method of claim 19, wherein the communication typical of an intruder includes one or more of IRC Traffic, ICMP Routing, IDS Evasion and software known to be used by malicious users.

28. A system for detecting a compromised network, comprising:

a data monitoring device adapted to collect data packet transmissions on a network,

software programmed with model session rules expected to be followed during sessions involving hosts on the network and with rules for operation of a model host expected to be followed by one or more hosts on the network, and

a data analysis engine operably connected to the data monitoring device and the software, and adapted to analyze the data packet transmissions to identify a network host participating in a session with one or more session rule violations.

29. The system of claim 28, wherein the data analysis engine is adapted to analyze the data packet transmissions to identify a network host violating at least one rule of operation of a model host.

30. The system of claim 29, wherein the software is further programmed with a model environment rule for each host, and the data analysis engine is adapted to analyze the data packet transmissions to identify a host operating in violation of its model environment rule.

31. The system of claim 28, further comprising a reporting unit.