Method of non-intrusive analysis of secure and non-secure web application traffic in real-time

• US 20040243349A1
• Filed: 05/30/2003
• Published: 12/02/2004
• Est. Priority Date: 05/30/2003
• Status: Active Grant

• Current Assignee: Borland Software Corporation (Open Text Corporation)
• Original Assignee: Segue Software, Inc. (Open Text Corporation)
• Inventors: Greifeneder, Bernd, Schwarzbauer, Gunter, Spiegl, Helmut, Reichl, Bernhard
• Granted Patent: US 7,543,051 B2
• Time in Patent Office: Days
• US Class Current: 702/183
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: G06F 2221/2145 Inheriting rights or properties, e.g., propagation of permissions or restrictions within a hierarchy
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/166 at the transport layer

Abstract

Provided is a method and system for monitoring and analysis of networked systems, that is non-intrusive and real time. Both secure and non-secure traffic may be analyzed. The provided method involves non-intrusively copying data from a communication medium, reconstructing this data to a higher level of communication, such as the application level, grouping the data into sets, each set representing a session, and organizing the data for chosen sessions in hierarchical fashion which corresponds to the hierarchy of the communicated information. If monitored communications are encrypted, they are non-intrusively decrypted in real time. Hierarchically reconstructed session data is used by one or more plug-in applications, such as alarms, archival applications, visualization applications, script generation applications, abandonment monitoring applications, error detection applications, performance monitoring applications, and others.

Claims

1. A method for non-intrusive real-time analysis of secure communications between a first application and a second application, wherein the first and second applications communicate through a communication channel, comprising the steps of:

non-intrusively and securely capturing a plurality of secure communications between the first application and the second application substantially in real-time;

processing the plurality of communications to form a first plurality of information units, each information unit comprising application level information substantially in real-time;

analyzing a second plurality of information units up to an application layer, the second plurality comprising one or more of the first plurality of information units, to determine a plurality of dependencies among the second plurality of information units substantially in real-time;

organizing the second plurality of information units into a hierarchical data structure according to the plurality of dependencies among the information units.

2. The method of claim 1, wherein the capturing step comprises the steps of:

non-intrusively copying a plurality of communications from the communication channel;

selecting the plurality of communications between the first application and the second application from the plurality of communications from the communication channel.

3. The method of claim 1, wherein each information unit also comprises identification data, and monitoring data.

4. The method of claim 3, wherein the information unit comprises a transaction.

5. The method of claim 1 further including the steps of:

storing the information units in memory;

selecting a information unit of interest; and

retrieving the second plurality of information units from the memory;

wherein each information unit of the second plurality is related to the information unit of interest.

6. The method of claim 5, wherein the step of selecting a information unit includes the step of monitoring the information units as they are being formed by the processing step.

7. The method of claim 1, wherein the analyzing step includes the step of parsing the application level information of at least one of the second plurality of information units.

8. The method of claim 1, wherein the processing step includes the step of decrypting one or more of the plurality of communications for accessing the application level information.

9. The method of claim 8, wherein the decrypting step further includes the step of generating a session key from a previously provided pre-master secret.

10. The method of claim 1, further comprising the step of using the hierarchical data structure with a plug-in monitoring application selected from the group consisting of:

a session visualization application, a statistics module, a monitoring and alerting module, test script generation module, a service level validation module, a click path analysis module, and a customer support assistance application.

11. A system for non-intrusive real-time analysis of secure communications between a first application and a second application, the first and second applications using a communication channel, the system comprising:

a non-intrusive and secure communications capture device, connected to the communications channel;

a network module, connected to the communications capture device and configured to process communications from the physical layer to the network layer substantially in real-time; and

a session reconstruction unit, connected to the network module and configured to process communications to the application layer in real-time, to group communications into transactions and to arrange transactions in a hierarchical data structure according to dependencies within the information contained in the transactions.

12. The system of claim 11, further including a transaction analyzer module, connected to the session reconstruction unit and configured to analyze transactions as they are being created by the session reconstruction unit.

13. The system of claim 11, wherein the session reconstruction unit comprises:

a stream creation unit, connected to the network module and configured to receive a plurality of communications and group them into a plurality of streams, and to add connection meta information to each stream, wherein each stream represents a single network connection.

a message decoder, connected to the stream creation unit and configured to create a plurality of transactions from the communications included in the plurality of streams;

a transaction storage, connected to the message decoder, configured to store the plurality of transactions; and

a session reconstruction module, connected to the transaction storage, and configured to receive a transaction of interest, and to retrieve a set of transactions from the transaction storage, the set of transactions being such that each transaction belonging to the set of transactions has a predefined relationship with the transaction of interest, and to group the set of transactions in the hierarchical data structure according to dependencies within the information contained in the transactions.

14. The system of claim 13, wherein each transaction comprises:

transaction meta information;

a request header; and

a response header.

15. The system of claim 14, wherein at least one of the transactions further comprises a request body.

16. The system of claim 14, wherein at least one of the transactions further comprises a response body.

17. The system of claim 13, wherein the message decoder comprises:

a transport protocol decoder, configured to receive a second plurality of streams, derived from the first plurality of streams, and derive a plurality of request messages and a plurality of response messages from the second plurality of streams; and

a transaction assembler, connected to the transport protocol decoder and configured to receive the plurality of request messages and the plurality of response messages from the transaction protocol decoder and to combine the plurality of request messages and plurality of response messages into the plurality of transactions, wherein each transaction is created by combining a request message and a response message.

18. The system of claim 17, wherein the message decoder further comprises:

a secure traffic decryption unit, connected to the stream creation unit and to the transport protocol decoder and configured to receive the plurality of streams, wherein at least one stream within the plurality of streams is encrypted, and to decrypt the at least one encrypted stream forming a plurality of decrypted streams,

wherein the second plurality of streams is derived from the plurality of streams by replacing the at least one encrypted stream with the plurality of decrypted streams.

19. The system of claim 15, wherein the session reconstruction module further comprises:

a meta information based transaction pre-selection unit, connected to the transaction storage, and configured to receive the transaction of interest and to retrieve a part of the set of transactions from the transaction storage, based on the transaction meta information;

a transaction pre-selection buffer, connected to the meta information based transaction pre-selection unit, and configured to store the part of the set of transactions;

a header based transaction correlation unit, connected to the transaction pre-selection buffer and configured to find dependencies between transactions of the part of the set of transactions based on the request headers and response headers of transactions; and

a body based transaction correlation and hierarchical linking unit, connected to the header based transaction correlation unit and the transaction storage and configured to retrieve from the transaction storage additional transactions of the set of transactions based on the request bodies and response bodies of the part of the set of transactions already retrieved and to find dependencies between transactions of the set of transactions based on the request bodies and response bodies of the transactions of the set of transactions, and to group the set of transactions in a hierarchical data structure according to the found dependencies.

20. The system of claim 16, wherein the session reconstruction module further comprises:

a meta information based transaction pre-selection unit, connected to the transaction storage, and configured to receive the transaction of interest and to retrieve a part of the set of transactions from the transaction storage, based on the transaction meta information;

a transaction pre-selection buffer, connected to the meta information based transaction pre-selection unit, and configured to store the part of the set of transactions;

a header based transaction correlation unit, connected to the transaction pre-selection buffer and configured to find dependencies between transactions of the part of the set of transactions based on the request headers and response headers of transactions; and

a body based transaction correlation and hierarchical linking unit, connected to the header based transaction correlation unit and the transaction storage and configured to retrieve from the transaction storage additional transactions of the set of transactions based on the request bodies and response bodies of the part of the set of transactions already retrieved and to find dependencies between transactions of the set of transactions based on the request bodies and response bodies of the transactions of the set of transactions, and to group the set of transactions in a hierarchical data structure according to the found dependencies.

21. The system of claims 11 further comprising a plug-in monitoring application selected from the group consisting of:

a session visualization application, a statistics module, a monitoring and alerting module, a test script generation module, a service level validation module, a click path analysis module, and a customer support assistance application, the plug-in monitoring application being connected to the session reconstruction module and being configured to receive the hierarchical data structure.

22. A method for non-intrusive analysis of secure communications between two or more applications, communicating through a communication channel, comprising the steps of:

non-intrusively and securely capturing the communications passing through the communications channel substantially in real-time;

processing one or more of the communications to the application layer substantially in real-time;

grouping one or more of the processed communications into transactions substantially in real-time;

parsing one or more of the transactions in order to determine dependencies among them substantially in real-time; and

grouping one or more of the transactions into a hierarchical structure, according to the dependencies among the transactions.

23. The method of claim 22, wherein the processing step further comprises the step of decrypting one or more of the communications.

24. The method of claim 22, wherein the dependency between a first and a second transaction signifies an HTTP reference included in the first transaction and referring to an addressable object at least partially included in the second transaction.

25. The method of claim 24, wherein the dependency between a first and a second transaction signifies an HTML reference included in the first transaction and referring to an addressable object at least partially included in the second transaction.

26. The method of claim 24, wherein a first transaction has missing data, and a second transaction includes the missing data and further comprising the step of retrieving the missing data from the second transaction and copying the missing data into the first transaction.

27. The method of claim 25, wherein a first transaction has missing data, and a second transaction includes the missing data and further comprising the step of retrieving the missing data from the second transaction and copying the missing data into the first transaction.

28. The method of claim 22, further comprising the step of using the hierarchical data structure with a plug-in monitoring application selected from the group consisting of:

a session visualization application, a statistics module, a monitoring and alerting module, a test script generation module, a service level validation module, a click path analysis module, and a customer support assistance application.

29. The method of claim 23, further comprising the step of using the hierarchical data structure with a plug-in monitoring application selected from the group consisting of:

a session visualization application, a statistics module, a monitoring and alerting module, a test script generation module, a service level validation module, a click path analysis module, and a customer support assistance application.

30. A method for non-intrusive real-time analysis of secure communications between a first application and a second application, the first and second applications using a communication channel, the system comprising the steps of:

non-intrusively copying a plurality of secure communications from the communication channel substantially in real-time;

processing the plurality of communications to the transport layer substantially in real-time;

grouping the processed plurality of communications into a plurality of transactions substantially in real-time; and

arranging one or more of the plurality of transactions into a hierarchical data structure according to dependencies within the information contained in the plurality of transactions.

31. The method of claim 30 further including the step of:

analyzing the plurality of transactions, in order to select a transaction of interest.

32. The method of claim 30, wherein the grouping step further comprises the steps of:

separating the plurality of communications into a first plurality of streams, each stream representing a single network connection;

adding meta information to the at least one stream;

creating a plurality of transactions, using the communications of the at least one stream, wherein each transaction comprises one or more communications of a single stream;

33. The method of claim 32 further comprising the steps storing the plurality of transactions;

and

receiving a transaction of interest;

and wherein the arranging step further comprises the step of;

retrieving a set of transactions from the stored plurality of transactions, wherein each transaction of the set of transactions has a predefined relationship with the transaction of interest.

34. The method of claim 33, wherein each transaction comprises:

transaction meta information;

a request header; and

a response header.

35. The method of claim 34, wherein at least one of the transactions further comprises a request body.

36. The method of claim 34, wherein at least one of the transactions further comprises a response body.

37. The method of claim 33, wherein the creating of a plurality of transactions step further comprises the steps of:

receiving a second plurality of streams the second plurality being derived from the first plurality of streams;

deriving a plurality of request messages and a plurality of response messages from the second plurality of streams; and

combining the plurality of request messages and the plurality of response messages into a plurality of transactions each transaction comprising a request message and a response message.

38. The method of claim 33, wherein at least one of the streams of the first plurality of streams is encrypted and the creating of a plurality of transactions step further comprises the steps of:

decrypting the at least one encrypted stream to create at least one decrypted stream; and

replacing the at least one encrypted stream with the at least one decrypted stream to form the second plurality of streams.

39. The method of claim 35, wherein the retrieving of a set of transactions step further comprises the steps of:

retrieving a first part of the set of transactions, the first part comprising at least one transaction based on the meta information of the transaction of interest; and

retrieving a second part of the set of transactions, the second part comprising at least one transaction based on the request and response bodies of the transactions of the first part of the set of transactions.

40. The method of claim 39, wherein the arranging step further comprises the steps of:

finding a first set of dependencies among the set of transactions based on the request and response headers of the transactions in the set of transactions;

parsing at least one of the request and response bodies of the set of transactions; and

finding a second set of dependencies among the set of transactions based on the parsed request and response bodies of the set of transactions;

wherein the arranging step is performed according to the first and second sets of dependencies.

41. The method of claim 30, further comprising the step of using the hierarchical data structure with a plug-in monitoring application selected from the group consisting of:

a session visualization application, a statistics module, a monitoring and alerting module, a test script generation module, a service level validation module, a click path analysis module, and a customer support assistance application.