Detecting public network attacks using signatures and fast content analysis

• US 8,296,842 B2
• Filed: 12/01/2004
• Issued: 10/23/2012
• Est. Priority Date: 04/08/2004
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Singh, Sumeet, Varghese, George, Estan, Cristi, Savage, Stefan
• Primary Examiner(s): CERVETTI, DAVID GARCIA
• Application Number:
  US11/547,944
• Publication Number:
  US 20080307524A1
• Time in Patent Office: 2,883 Days
• Field of Search: 713/153, 726/22, 726/23, 726/11, 726/12, 726/13
• US Class Current: 726/22
• CPC Class Codes: G06F 21/55 Detecting local intrusion or implementing counter-measures
• CPC Class Codes: H04L 2209/60 Digital content management, e.g. content distribution
• CPC Class Codes: H04L 2463/141 Denial of service attacks against endpoints in a network
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 9/002 Countermeasures against attacks on cryptographic mechanisms network architectures or network communication protocols for protection against malicious traffic H04L63/1441
• CPC Class Codes: H04L 9/3236 using cryptographic hash functions
• CPC Class Codes: H04L 9/3247 involving digital signatures

Abstract

Network worms or viruses are a growing threat to the security of public and private networks and the individual computers that make up those networks. A content sifting method if provided that automatically generates a precise signature for a worm or virus that can then be used to significantly reduce the propagation of the worm elsewhere in the network or eradicate the worm altogether. The content sifting method is complemented by a value sampling method that increases the throughput of network traffic that can be monitored. Together, the methods track the number of times invariant strings appear in packets and the network address dispersion of those packets including variant strings. When an invariant string reaches a particular threshold of appearances and address dispersion, the string is reported as a signature for suspected worm.

Claims

1. A system, comprising at least one hardware module, for detecting a network attack, comprising:

a communication module configured to receive a plurality of packets on a network; and

a signature module configured to receive said plurality of packets from the communication module and analyze the content of said packets to detect common content among said packets to identify a network attack; and

a content analysis module configured to analyze the common content of said plurality of packets, including criteria not based on a known vulnerability, to identify a network attack;

wherein the content analysis module comprises a correlation module configured to determine whether packets sent in a first interval to a destination address are sent from said destination address in a second interval.

2. The system of claim 1, wherein the signature module further comprises:

a parser module configured to parse packets into a plurality of portions;

a key module configured to perform a data reduction on each of said plurality of portions to create a plurality of data reduced portions;

and a data module configured to store said data reduced portions in a data storage area,

wherein the reduced data portions in the plurality of data reduced portions have a smaller size and a constant predetermined relation with portions into which the packets are parsed and at least some of the portions into which the packets are parsed that differ are reduced to the same reduced data portion.

3. The system of claim 2, further comprising a filter module configured to reduce the number of said plurality of data reduced portions prior to storage.

4. The system of claim 1, wherein the content analysis module further comprises a code detection module configured to identify the presence of executable code in a packet.

5. A computer implemented method for analyzing network activity, comprising:

receiving, at a computer, a plurality of packets transiting a network;

analyzing, at the computer, the content of said plurality of packets to detect common content among said packets;

identifying, at the computer, network attacks based upon said analysis for common content, wherein criteria for the analysis of the common content of said plurality of packets includes criteria not based on a known vulnerability; and

comparing the destinations of said plurality of packets with destinations having known vulnerabilities.

6. The method of claim 5 wherein analyzing the content further comprises performing a data reduction on at least a portion of each of the packets of said plurality of packets to form a plurality of data reduced packets, wherein the reduced data packets in the plurality of data reduced packets have a smaller size and a constant predetermined relation with the packets transiting the network and at least some of the packets transiting the network that differ are reduced to the same reduced data packet.

7. The method of claim 6 wherein said data reduction comprises performing a hash function on at least a portion of each of the packets.

8. The method of claim 7 wherein said hash function is an incremental hash function.

9. The method of claim 6, wherein analyzing further comprises analyzing a subset of the data reduced portions, the subset identified by a common characteristic of each of the data reduced portions.

10. The method of claim 6, wherein analyzing further comprises analyzing a subset of the data reduced portions, the subset identified by a common characteristic of each of the data reduced portions.

11. The method of claim 10, wherein the common characteristic is the reduced data portion being equal to a value in a set of predefined values.

12. The method of claim 5 wherein identifying further comprises identifying one or more signatures of an attack.

13. The method of claim 5, wherein analyzing the content includes analyzing the content of the payloads in the plurality of packets.

14. The method of claim 5 further comprising determining whether there is an increasing number of sources and destinations of packets having common content.

15. The method of claim 5 further comprising analyzing the content of said plurality of packets for the presence of a specified type of code.

16. The method of claim 5 further comprising forming a plurality of portions from each of said plurality of packets, each portion comprising a specified subset of a packet.

17. A computer implemented method for analyzing network activity comprising:

obtaining a plurality of packets being transmitted across a network;

performing a data reduction on at least a portion of each of the packets of said plurality of packets to form a plurality of data reduced packets, wherein the reduced data packets in the plurality of data reduced packets have a smaller size and a constant predetermined relation with the packets being transmitted across the network and at least some of the packets being transmitted across the network that differ are reduced to the same reduced data packet;

detecting repetition of content among said plurality of data packets based on the reduced data packets;

comparing destinations of said plurality of packets with destinations having known vulnerabilities; and

identifying network attacks based upon said detection of repetition and said comparison of destinations.

18. The method of claim 17 wherein the destinations of said plurality of packets are only analyzed for packets wherein a repetition of content has been detected.

19. The method of claim 17, further comprising analyzing the content of the payloads in the plurality of packets.

20. The method of claim 19 further comprising analyzing the content of said plurality of packets for the presence of a specified type of code.

21. The method of claim 20 further comprising:

maintaining a first list of addresses;

forming a second list of sources that have sent to addresses on said first list; and

comparing a current source of common content to said second list.

22. The method of claim 17 wherein said data reduction comprises carrying out a hash function on at least a portion of each of the packets.

23. The method of claim 17 wherein said detecting common content comprises using at least first, second and third data reduction techniques on at least a portion of each of the packets, to obtain at least first, second and third results, and to count said first, second and third results, and to detect repetition when all of said at least first second and third results have a frequency of occurrence greater than a specified amount.

24. The method of claim 17 further comprising:

monitoring a first content sent to a destination;

monitoring a second content sent by said destination; and

determining whether there is a correlation between said first content and said second content.

25. The method of claim 17 further comprising forming a plurality of portions from each of said plurality of packets, each portion comprising a specified subset of a packet.

26. The method of claim 25, wherein:

forming a plurality of portions further comprises

identifying all portions comprising a specified subset of a packet, and

forming a plurality of portions from a subset of said all portions;

each portion in the subset has a common characteristic.

27. The method of claim 26, wherein the specified subset is a segment of data from the data payload having a predetermined byte size.

28. The method of claim 25, wherein:

a first portion of the plurality is from two or more adjacent packets; and

the first portion is obtained by storing a minimum length subsection comprising data from the two or more adjacent packets.

29. A method of analyzing network activity comprising:

obtaining, at a computer, a plurality of packets transiting a network;

performing, at a computer, a data reduction on at least a portion of each of the packets of said plurality of packets to form a plurality of data reduced packets, wherein the reduced data packets in the plurality of data reduced packets have a smaller size and a constant predetermined relation with the packets transiting the network and at least some of the packets transiting the network that differ are reduced to the same reduced data packet;

analyzing, at a computer, said plurality of data reduced packets to detect a repetition of at least a portion of content among said plurality of data packets based on criteria not necessarily based on a known vulnerability;

comparing the destinations of said plurality of packets with destinations having known vulnerabilities; and

analyzing, at a computer, said packets having repetitive content to determine if said packets are spreading.

30. The method of claim 29, wherein analyzing for spreading comprises determining whether there is an increasing number of sources and destinations of packets having common content.

31. The method of claim 29 wherein analyzing spreading comprises:

monitoring a first content sent to a destination;

monitoring a second content sent by said destination; and

determining a whether there is a correlation between said first content and said second content.

32. The method of claim 31 further comprising analyzing the content of said plurality of packets for the presence of a specified type of code.

33. The method of claim 29 further comprising determining a signature of an attack based upon said analyzing of said plurality of data reduced packets and said analyzing spreading.

34. The method of claim 29, wherein analyzing of said plurality of data reduced packets includes analyzing the content of the payloads in the plurality of packets.

35. The method of claim 29 wherein said data reduction comprises carrying out a hash function on at least a portion of each of the packets.

36. The method of claim 29 further comprising using at least first, second and third data reduction techniques on at least a portion of each of the packets, to obtain at least first, second and third results, and to count said first, second and third results, and to establish frequently occurring sections when all of said at least first second and third results have a frequency of occurrence greater than a specified amount.

37. The method of claim 29 further comprising forming a plurality of portions from each of said plurality of packets, each portion comprising a specified subset of a packet.

38. The method of claim 37, wherein a first portion of the plurality is from at least two packets.

39. The method of claim 37, wherein analyzing further comprises analyzing a subset of the plurality of portions, the subset identified by a common characteristic of each of said portions.

40. The method of claim 39, wherein the specified subset is a segment of data from the data payload having a predetermined byte size.

41. An apparatus comprising:

a signature generator, having a connection to a network, to obtain a portion of data from the network, operating to carry out a data reduction on said data portion by carrying out a hash function to reduce said data portion to a reduced data portion in a repeatable manner;

a memory, storing said reduced data portions,

wherein said signature generator also operates to detect common elements within said reduced data portion, and

a content analysis module configured to analyze the common elements using criteria not based on a known vulnerability, to identify a network attack;

wherein the content analysis module comprises a correlation module configured to determine whether packets sent in a first interval to a destination address are sent from said destination address in a second interval.

42. An apparatus as in claim 41, wherein said signature generator determining frequently occurring sections of message information within said reduced data portion.

43. An apparatus as in claim 42, further comprising a module that carries out an additional test on said frequently occurring sections of message information after said signature generator determines frequently occurring sections of message information.

44. An apparatus as in claim 43, wherein said additional test is a test to look for an increasing number of at least one of sources and destinations of said frequently occurring sections of message information.

45. An apparatus as in claim 44, wherein said module is a module to look for code within the frequently occurring sections.

46. An apparatus as in claim 43, wherein said module:

maintains a first list of unassigned addresses in said memory;

forms a second list of sources that have sent to addresses on said first list; and

compares a current source of a frequently occurring section to said second list.

47. An apparatus as in claim 46, wherein said module data reduces information prior to storing in said list.

48. An apparatus as in claim 47, wherein said portion of a network header is a port number indicating a service requested by a network packet.

49. An apparatus as in claim 46, wherein said module operates to:

first monitor a first content sent to a destination;

second monitor a second content sent by said destination; and

determine a correlation between said first content and said second content as said additional test.

50. An apparatus as in claim 49, wherein:

said first monitoring comprises monitoring multiple destinations; and

said second monitoring comprises monitoring multiple destinations during a different time period than said first monitoring.

51. An apparatus as in claim 46 wherein said portion of data further includes a portion of a network header.

52. An apparatus as in claim 46, wherein said portion of data comprises a first subset of a network packet including payload and header and wherein said data portion module further obtains a second subset of the same network packet for subsequent analysis.

53. An apparatus as in claim 52, wherein said data portion module forms a plurality of portions from each network packet, each of said plurality of portions comprising a specified subset of the network packet.

54. An apparatus as in claim 46, wherein said signature generator operates to form a hash function of at least one of the source or destination address, to form hash values using the hash function, to first determine a unique number of said hash values, and to second determine a number of said one of source or destination addresses based on said unique number from said first determine.

55. An apparatus as in claim 54, wherein said count carried out by said signature generator further comprises scaling the hash values prior to said second determine.

56. A apparatus as in claim 55, wherein said scaling comprises scaling by a first value during a first counting session, and scaling by a second value during a second measurement interval.

57. An apparatus as in claim 42, wherein said determining frequently occurring sections is done by using at least first, second and third data reduction techniques on each said portion, to obtain first, second and third results, and to count said first, second and third results, and to establish frequently occurring sections when all of said first second and third results have a frequency of occurrence greater than a specified amount.

58. An apparatus as in claim 41, wherein said memory stores information indicative of at least one of a number of sources sending the common content, and/or destinations that are receiving the common content, and said signature generator determines whether said number is increasing.

59. An apparatus as in claim 41, wherein said signature generator also analyzes for the presence of a specified type of code within said data portion.

60. An apparatus as in claim 41, further comprising forming a plurality of portions from each network packet, each of said plurality of portions comprising a continuous portion of payload, and information indicative of a port number requested by a network packet.

61. An apparatus as in claim 41, further comprising:

first and second hash generators, respectively forming first and second hash functions of said portions;

a first counter, with a plurality of stages, connected such that respective stages of said counter are incremented based on said first hash function;

a second counter, with a plurality of stages, and connected such that respective stages of said counter are incremented based on said first hash function.

62. An apparatus as in claim 61, further comprising a module that checks said one of said stages of said first counter and said one of said stages of said second counter against a threshold, and identifies said portion as frequent content only when both said one of said stages of said first counter and said one of said stages of said second counter are both above said threshold.

63. An apparatus as in claim 62, further comprising a frequent content buffer table storing specified frequent content.

64. An apparatus as in claim 63, further comprising at least a third counter, and a third hash generator, taking a third hash of said portion, and incrementing a stage of said third counter based on said third hash, where said module identifies said portion as frequent content only when all of said stages of each of said first, second and third counters are each above said threshold.

65. An apparatus as in claim 64, wherein said signature generator includes a sliding window portion that first obtains said portion by taking a first part of the message, and subsequently obtains said portion by taking a second part of the message.

66. A apparatus as in claim 65, wherein at least one of said hash functions is an incremental hash function.

67. An apparatus as in claim 41, wherein said memory stores a list of computers on the network, and stores an update level for each of said computers indicating which of said computers is susceptible to a specified kind of attack, and a module which monitors for said kind of attack only when the message is directed for a computer which is susceptible to said kind of attack.

68. An apparatus of claim 67 where said module checks comprises checking for a message that attempts to exploit a known vulnerability to which a computer is vulnerable, as said specified attack.

69. An apparatus as in claim 68, wherein said module checks for a field that is longer than a specified length.