Systems and methods for detecting and mitigating threats to a structured data storage system

• US 9,185,125 B2
• Filed: 01/09/2014
• Issued: 11/10/2015
• Est. Priority Date: 01/31/2012
• Status: Active Grant

• Current Assignee: QuantumXchange, Inc.
• Original Assignee: DB Networks Inc.
• Inventors: Varsanyi, Eric, Rosenberg, David, Paterson, Chuck, Schnetzler, Steve, Ruddick, Timothy
• Primary Examiner(s): Hailu, Teshome
• Assistant Examiner(s): Wysznski, Aubrey
• Application Number:
  US14/151,597
• Publication Number:
  US 20140201838A1
• Time in Patent Office: 670 Days
• Field of Search: 726/23
• US Class Current: 1/1
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Systems, methods, and computer-readable media for detecting threats on a network. In an embodiment, target network traffic being transmitted between two or more hosts is captured. The target network traffic comprises a plurality of packets, which are assembled into one or more messages. The assembled message(s) may be parsed to generate a semantic model of the target network traffic. The semantic model may comprise representation(s) of operation(s) or event(s) represented by the message(s). Score(s) for the operation(s) or event(s) may be generated using a plurality of scoring algorithms, and potential threats among the operation(s) or event(s) may be identified using the score(s).

Claims

1. A method for detecting threats on a network, the method comprising:

capturing target network traffic being transmitted between two or more hosts, wherein the target network traffic comprises a plurality of packets; and

using at least one hardware processor to

assemble the plurality of packets into one or more messages,

parse the assembled one or more messages to generate a semantic model of the target network traffic, wherein the semantic model comprises one or more representations of one or more operations or events represented by the one or more messages,

generate one or more scores for the one or more operations or events using a plurality of scoring algorithms, and

identify one or more potentially threatening ones of the one or more operations or events based on the one or more scores.

2. The method of claim 1, wherein generating the semantic model of the target network traffic comprises generating one or more language-independent representations of one or more operations or events represented by the one or more messages.

3. The method of claim 2, wherein each of the one or more language-independent representations of one or more operations or events identify one or more of a session, a user, a database server, a type of operation or event, a lexical structure of one or more messages associated with the operation or event, a parse structure of the one or more messages associated with the operation or event, a semantic structure of the one or more messages associated with the operation or event, and timing data related to the operation or event.

4. The method of claim 2, wherein parsing the one or more messages to generate a semantic model of the target network traffic comprises:

lexically analyzing the assembled one or more messages into a plurality of dialect-independent tokens;

parsing one or more sequences of the plurality of tokens into one or more parse trees comprising a plurality of parse nodes; and

semantically analyzing the one or more parse trees to generate one or more dialect-independent semantic representations of the one or more operations or events.

5. The method of claim 4, wherein generating one or more scores for the one or more operations or events using a plurality of scoring algorithms comprises:

traversing the one or more parse trees to identify one or more operations or events;

generating a first score for at least one of the one or more operations or events using a first one of the plurality of scoring algorithms;

generating a second score for the at least one operation or event using a second one of the plurality of scoring algorithms, wherein the second algorithm is different than the first algorithm; and

computing a total score for the at least one operation or event based, at least in part, on the first score and the second score.

6. The method of claim 1, further comprising:

receiving one or more representations of acceptable network traffic; and

training each of one or more of the plurality of scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic.

7. The method of claim 6, wherein the one or more representations of acceptable network traffic comprise a plurality of representations of acceptable operations or events, and wherein training at least one of the one or more scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic comprises:

parsing the plurality of representations of acceptable operations or events into a plurality of parse trees; and

generating a pattern-matching tree that is an isomorphism between two or more of the plurality of parse trees and represents a unification of the two or more parse trees.

8. The method of claim 7, wherein generating one or more scores for the one or more operations or events using a plurality of scoring algorithms comprises generating a score for a target operation or event using the at least one scoring algorithm by:

parsing a representation of the target operation or event into a target parse tree;

computing a tree-edit distance comprising a minimum number of edits necessary to unify the target parse tree with the pattern-matching tree; and

,

based on the tree-edit distance, generating a scalar value indicating a probability that the target operation or event represents a malicious attack or nominal application variability.

9. The method of claim 6, wherein training at least one of the one or more scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic comprises generating one or more profiles of normal network traffic, wherein the one or more profiles of normal network traffic comprise one or more of a normal number of rows returned by an operation, a normal execution time of an operation, one or more normal parameter values for an operation, one or more normal types of content returned by an operation, a normal execution time of an operation for a certain time period, a normal frequency of an operation for a certain time period, an identifier of an application, and a model of normal execution semantics for an operation.

10. The method of claim 6, wherein training the one or more scoring algorithms comprises, for each of the one or more scoring algorithms, generating a model, for scoring operations or events, using the one or more representations of acceptable network traffic.

11. The method of claim 6, wherein at least one of the trained one or more scoring algorithms determines whether a structural signature of a target operation within the target network traffic matches the structural signature of an acceptable operation, learned during training of the at least one scoring algorithm, to generate a score for the target operation.

12. The method of claim 11, wherein the at least one trained scoring algorithm determines a minimum edit distance between a structure of the target operation and a structure of the acceptable operation, and wherein the minimum edit distance represents a minimum number of insertions required to create the structure of the target operation from the structure of the acceptable operation.

13. The method of claim 11, wherein the target operation comprises a structured query language (SQL) statement.

14. The method of claim 13, wherein the at least one trained scoring algorithm maintains a set of one or more templates of acceptable SQL statements.

15. The method of claim 6, wherein the at least one scoring algorithm comprises a first scoring algorithm, and wherein a second one of the plurality of scoring algorithms:

determines a background frequency of lexical errors within one or more acceptable operations learned during training of the first scoring algorithm;

identifies one or more lexical errors within a target operation within the target network traffic; and

computes a probability that the one or more lexical errors within the target operation are in accordance with the background frequency of lexical errors within the one or more acceptable operations learned during the training of the first scoring algorithm.

16. The method of claim 1, wherein at least one of the plurality of scoring algorithms searches a target operation within the target network traffic for one or more segments of structured query language (SQL) that potentially indicate an attack.

17. The method of claim 16, wherein the one or more segments of SQL represent potentially one or more SQL injections.

18. The method of claim 16, wherein the one or more segments of SQL represent potentially one or more time-consuming SQL clauses.

19. The method of claim 18, wherein each of the one or more segments of SQL is associated with one or more performance parameters, and wherein the at least one scoring algorithm calculates an estimated performance metric for the target operation based on the one or more performance parameters associated with any of the one or more segments of SQL identified within the target operation.

20. The method of claim 1, wherein at least one of the plurality of scoring algorithms parses a structured query language (SQL) statement into a plurality of segments, and determines whether the plurality of segments satisfy one or more criteria.

21. The method of claim 1, wherein assembling the plurality of packets into one or more messages comprises:

synchronizing the plurality of packets;

sorting each of the plurality of packets into one of two host queues according to the transmission direction of the packet;

processing the two host queues into a single push queue by alternately processing the packets in one of the two host queues until a packet is encountered which cannot be disposed of or the host queue is empty and then processing the packets in the other one of the two host queues until a packet is encountered that cannot be disposed of or the host queue is empty;

if loss of a packet is detected, generating a synthetic gap packet to stand in for the lost packet; and

bundling packets in the single push queue into the one or more messages, wherein each of the one or more messages is a request message or a response message.

22. The method of claim 21, wherein the synthetic gap packet comprises an indication that it is a stand-in for a lost packet.

23. The method of claim 1, further comprising preventing one or more identified potentially threatening operations from being performed on a database that is accessible to one of the two or more hosts.

24. A system for detecting threats on a network, the system comprising:

at least one hardware processor; and

one or more executable modules that, when executed by the at least one hardware processor,

capture target network traffic being transmitted between two or more hosts, wherein the target network traffic comprises a plurality of packets,

assemble the plurality of packets into one or more messages,

parse the assembled one or more messages to generate a semantic model of the target network traffic, wherein the semantic model comprises one or more representations of one or more operations or events represented by the one or more messages,

generate one or more scores for the one or more operations or events using a plurality of scoring algorithms, and

identify one or more potentially threatening ones of the one or more operations or events based on the one or more scores.

25. The system of claim 24, wherein generating the semantic model of the target network traffic comprises generating one or more language-independent representations of one or more operations or events represented by the one or more messages.

26. The system of claim 25, wherein each of the one or more language-independent representations of one or more operations or events identify one or more of a session, a user, a database server, a type of operation or event, a lexical structure of one or more messages associated with the operation or event, a parse structure of the one or more messages associated with the operation or event, a semantic structure of the one or more messages associated with the operation or event, and timing data related to the operation or event.

27. The system of claim 25, wherein parsing the one or more messages to generate a semantic model of the target network traffic comprises:

lexically analyzing the assembled one or more messages into a plurality of dialect-independent tokens;

parsing one or more sequences of the plurality of tokens into one or more parse trees comprising a plurality of parse nodes; and

semantically analyzing the one or more parse trees to generate one or more dialect-independent semantic representations of the one or more operations or events.

28. The system of claim 27, wherein generating one or more scores for the one or more operations or events using a plurality of scoring algorithms comprises:

traversing the one or more parse trees to identify one or more operations or events;

generating a first score for at least one of the one or more operations or events using a first one of the plurality of scoring algorithms;

generating a second score for the at least one operation or event using a second one of the plurality of scoring algorithms, wherein the second algorithm is different than the first algorithm; and

computing a total score for the at least one operation or event based, at least in part, on the first score and the second score.

29. The system of claim 24, wherein the one or more executable modules further:

receive one or more representations of acceptable network traffic; and

train each of one or more of the plurality of scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic.

30. The system of claim 29, wherein the one or more representations of acceptable network traffic comprise a plurality of representations of acceptable operations or events, and wherein training at least one of the one or more scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic comprises:

parsing the plurality of representations of acceptable operations or events into a plurality of parse trees; and

generating a pattern-matching tree that is an isomorphism between two or more of the plurality of parse trees and represents a unification of the two or more parse trees.

31. The system of claim 30, wherein generating one or more scores for the one or more operations or events using a plurality of scoring algorithms comprises generating a score for a target operation or event using the at least one scoring algorithm by:

parsing a representation of the target operation or event into a target parse tree;

computing a tree-edit distance comprising a minimum number of edits necessary to unify the target parse tree with the pattern-matching tree; and

,

based on the tree-edit distance, generating a scalar value indicating a probability that the target operation or event represents a malicious attack or nominal application variability.

32. The system of claim 29, wherein training at least one of the one or more scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic comprises generating one or more profiles of normal network traffic, wherein the one or more profiles of normal network traffic comprise one or more of a normal number of rows returned by an operation, a normal execution time of an operation, one or more normal parameter values for an operation, one or more normal types of content returned by an operation, a normal execution time of an operation for a certain time period, a normal frequency of an operation for a certain time period, an identifier of an application, and a model of normal execution semantics for an operation.

33. The system of claim 29, wherein training the one or more scoring algorithms comprises, for each of the one or more scoring algorithms, generating a model, for scoring operations or events, using the one or more representations of acceptable network traffic.

34. The system of claim 29, wherein at least one of the trained one or more scoring algorithms determines whether a structural signature of a target operation within the target network traffic matches the structural signature of an acceptable operation, learned during training of the at least one scoring algorithm, to generate a score for the target operation.

35. The system of claim 34, wherein the at least one trained scoring algorithm determines a minimum edit distance between a structure of the target operation and a structure of the acceptable operation, and wherein the minimum edit distance represents a minimum number of insertions required to create the structure of the target operation from the structure of the acceptable operation.

36. The system of claim 34, wherein the target operation comprises a structured query language (SQL) statement.

37. The system of claim 36, wherein the at least one trained scoring algorithm maintains a set of one or more templates of acceptable SQL statements.

38. The system of claim 29, wherein the at least one scoring algorithm comprises a first scoring algorithm, and wherein a second one of the plurality of scoring algorithms:

determines a background frequency of lexical errors within one or more acceptable operations learned during training of the first scoring algorithm;

identifies one or more lexical errors within a target operation within the target network traffic; and

computes a probability that the one or more lexical errors within the target operation are in accordance with the background frequency of lexical errors within the one or more acceptable operations learned during the training of the first scoring algorithm.

39. The system of claim 24, wherein at least one of the plurality of scoring algorithms searches a target operation within the target network traffic for one or more segments of structured query language (SQL) that potentially indicate an attack.

40. The system of claim 39, wherein the one or more segments of SQL represent potentially one or more SQL injections.

41. The system of claim 39, wherein the one or more segments of SQL represent potentially one or more time-consuming SQL clauses.

42. The system of claim 41, wherein each of the one or more segments of SQL is associated with one or more performance parameters, and wherein the at least one scoring algorithm calculates an estimated performance metric for the target operation based on the one or more performance parameters associated with any of the one or more segments of SQL identified within the target operation.

43. The system of claim 24, wherein at least one of the plurality of scoring algorithms parses a structured query language (SQL) statement into a plurality of segments, and determines whether the plurality of segments satisfy one or more criteria.

44. The system of claim 24, wherein assembling the plurality of packets into one or more messages comprises:

synchronizing the plurality of packets;

sorting each of the plurality of packets into one of two host queues according to the transmission direction of the packet;

processing the two host queues into a single push queue by alternately processing the packets in one of the two host queues until a packet is encountered which cannot be disposed of or the host queue is empty and then processing the packets in the other one of the two host queues until a packet is encountered that cannot be disposed of or the host queue is empty;

if loss of a packet is detected, generating a synthetic gap packet to stand in for the lost packet; and

bundling packets in the single push queue into the one or more messages, wherein each of the one or more messages is a request message or a response message.

45. The system of claim 44, wherein the synthetic gap packet comprises an indication that it is a stand-in for a lost packet.

46. The system of claim 24, wherein the one or more executable modules further prevent one or more identified potentially threatening operations from being performed on a database that is accessible to one of the two or more hosts.

49. The non-transitory computer-readable medium of claim 42, wherein each of the one or more language-independent representations of one or more operations or events identify one or more of a session, a user, a database server, a type of operation or event, a lexical structure of one or more messages associated with the operation or event, a parse structure of the one or more messages associated with the operation or event, a semantic structure of the one or more messages associated with the operation or event, and timing data related to the operation or event.

47. A non-transitory computer-readable medium having one or more instructions stored thereon for detecting threats on a network, wherein the one or more instructions, when executed by a processor, cause the processor to:

capture target network traffic being transmitted between two or more hosts, wherein the target network traffic comprises a plurality of packets;

assemble the plurality of packets into one or more messages;

parse the assembled one or more messages to generate a semantic model of the target network traffic, wherein the semantic model comprises one or more representations of one or more operations or events represented by the one or more messages;

generate one or more scores for the one or more operations or events using a plurality of scoring algorithms; and

identify one or more potentially threatening ones of the one or more operations or events based on the one or more scores.

48. The non-transitory computer-readable medium of claim 47, wherein generating the semantic model of the target network traffic comprises generating one or more language-independent representations of one or more operations or events represented by the one or more messages.

50. The non-transitory computer-readable medium of claim 48, wherein parsing the one or more messages to generate a semantic model of the target network traffic comprises:

lexically analyzing the assembled one or more messages into a plurality of dialect-independent tokens;

parsing one or more sequences of the plurality of tokens into one or more parse trees comprising a plurality of parse nodes; and

semantically analyzing the one or more parse trees to generate one or more dialect-independent semantic representations of the one or more operations or events.

51. The non-transitory computer-readable medium of claim 50, wherein generating one or more scores for the one or more operations or events using a plurality of scoring algorithms comprises:

traversing the one or more parse trees to identify one or more operations or events;

generating a first score for at least one of the one or more operations or events using a first one of the plurality of scoring algorithms;

generating a second score for the at least one operation or event using a second one of the plurality of scoring algorithms, wherein the second algorithm is different than the first algorithm; and

computing a total score for the at least one operation or event based, at least in part, on the first score and the second score.

52. The non-transitory computer-readable medium of claim 47, wherein the one or more instructions, when executed by the processor, further cause the processor to:

receive one or more representations of acceptable network traffic; and

train each of one or more of the plurality of scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic.

53. The non-transitory computer-readable medium of claim 52, wherein the one or more representations of acceptable network traffic comprise a plurality of representations of acceptable operations or events, and wherein training at least one of the one or more scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic comprises:

parsing the plurality of representations of acceptable operations or events into a plurality of parse trees; and

generating a pattern-matching tree that is an isomorphism between two or more of the plurality of parse trees and represents a unification of the two or more parse trees.

54. The non-transitory computer-readable of claim 53, wherein generating one or more scores for the one or more operations or events using a plurality of scoring algorithms comprises generating a score for a target operation or event using the at least one scoring algorithm by:

parsing a representation of the target operation or event into a target parse tree;

computing a tree-edit distance comprising a minimum number of edits necessary to unify the target parse tree with the pattern-matching tree; and

,

based on the tree-edit distance, generating a scalar value indicating a probability that the target operation or event represents a malicious attack or nominal application variability.

55. The non-transitory computer-readable of claim 52, wherein training at least one of the one or more scoring algorithms to score target operations or events using the one or more representations of acceptable network traffic comprises generating one or more profiles of normal network traffic, wherein the one or more profiles of normal network traffic comprise one or more of a normal number of rows returned by an operation, a normal execution time of an operation, one or more normal parameter values for an operation, one or more normal types of content returned by an operation, a normal execution time of an operation for a certain time period, a normal frequency of an operation for a certain time period, an identifier of an application, and a model of normal execution semantics for an operation.

56. The non-transitory computer-readable medium of claim 52, wherein training the one or more scoring algorithms comprises, for each of the one or more scoring algorithms, generating a model, for scoring operations or events, using the one or more representations of acceptable network traffic.

57. The non-transitory computer-readable medium of claim 52, wherein at least one of the trained one or more scoring algorithms determines whether a structural signature of a target operation within the target network traffic matches the structural signature of an acceptable operation, learned during training of the at least one scoring algorithm, to generate a score for the target operation.

58. The non-transitory computer-readable medium of claim 57, wherein the at least one trained scoring algorithm determines a minimum edit distance between a structure of the target operation and a structure of the acceptable operation, and wherein the minimum edit distance represents a minimum number of insertions required to create the structure of the target operation from the structure of the acceptable operation.

59. The non-transitory computer-readable medium of claim 57, wherein the target operation comprises a structured query language (SQL) statement.

60. The non-transitory computer-readable medium of claim 59, wherein the at least one trained scoring algorithm maintains a set of one or more templates of acceptable SQL statements.

61. The non-transitory computer-readable medium of claim 52, wherein the at least one scoring algorithm comprises a first scoring algorithm, and wherein a second one of the plurality of scoring algorithms:

determines a background frequency of lexical errors within one or more acceptable operations learned during training of the first scoring algorithm;

identifies one or more lexical errors within a target operation within the target network traffic; and

computes a probability that the one or more lexical errors within the target operation are in accordance with the background frequency of lexical errors within the one or more acceptable operations learned during the training of the first scoring algorithm.

62. The non-transitory computer-readable medium of claim 47, wherein at least one of the plurality of scoring algorithms searches a target operation within the target network traffic for one or more segments of structured query language (SQL) that potentially indicate an attack.

63. The non-transitory computer-readable medium of claim 62, wherein the one or more segments of SQL represent potentially one or more SQL injections.

64. The non-transitory computer-readable medium of claim 62, wherein the one or more segments of SQL represent potentially one or more time-consuming SQL clauses.

65. The non-transitory computer-readable medium of claim 64, wherein each of the one or more segments of SQL is associated with one or more performance parameters, and wherein the at least one scoring algorithm calculates an estimated performance metric for the target operation based on the one or more performance parameters associated with any of the one or more segments of SQL identified within the target operation.

66. The non-transitory computer-readable medium of claim 47, wherein at least one of the plurality of scoring algorithms parses a structured query language (SQL) statement into a plurality of segments, and determines whether the plurality of segments satisfy one or more criteria.

67. The non-transitory computer-readable medium of claim 47, wherein assembling the plurality of packets into one or more messages comprises:

synchronizing the plurality of packets;

sorting each of the plurality of packets into one of two host queues according to the transmission direction of the packet;

processing the two host queues into a single push queue by alternately processing the packets in one of the two host queues until a packet is encountered which cannot be disposed of or the host queue is empty and then processing the packets in the other one of the two host queues until a packet is encountered that cannot be disposed of or the host queue is empty;

if loss of a packet is detected, generating a synthetic gap packet to stand in for the lost packet; and

bundling packets in the single push queue into the one or more messages, wherein each of the one or more messages is a request message or a response message.

68. The non-transitory computer-readable medium of claim 67, wherein the synthetic gap packet comprises an indication that it is a stand-in for a lost packet.

69. The non-transitory computer-readable medium of claim 47, wherein the one or more instructions, when executed by the processor, further cause the processor to prevent one or more identified potentially threatening operations from being performed on a database that is accessible to one of the two or more hosts.