Method of non-intrusive analysis of secure and non-secure web application traffic in real-time

• US 7,543,051 B2
• Filed: 05/30/2003
• Issued: 06/02/2009
• Est. Priority Date: 05/30/2003
• Status: Active Grant

• Current Assignee: Borland Software Corporation (Open Text Corporation)
• Original Assignee: Borland Software Corporation (Open Text Corporation)
• Inventors: Reichl, Bernhard, Greifeneder, Bernd, Schwarzbauer, Gunter, Spiegl, Helmut
• Primary Examiner(s): Meky; Moustafa M
• Application Number:
  US10/455,798
• Publication Number:
  US 20040243349A1
• Time in Patent Office: 2,195 Days
• Field of Search: 709200-203, 709217-227
• US Class Current: 709/224
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: G06F 2221/2145 Inheriting rights or properties, e.g., propagation of permissions or restrictions within a hierarchy
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/166 at the transport layer

Abstract

Provided is a method and system for monitoring and analysis of networked systems, that is non-intrusive and real time. Both secure and non-secure traffic may be analyzed. The provided method involves non-intrusively copying data from a communication medium, reconstructing this data to a higher level of communication, such as the application level, grouping the data into sets, each set representing a session, and organizing the data for chosen sessions in hierarchical fashion which corresponds to the hierarchy of the communicated information. If monitored communications are encrypted, they are non-intrusively decrypted in real time. Hierarchically reconstructed session data is used by one or more plug-in applications, such as alarms, archival applications, visualization applications, script generation applications, abandonment monitoring applications, error detection applications, performance monitoring applications, and others.

Claims

1. A system for non-intrusive real-time analysis of secure communications between a first application running on a first computer and a second application running on a second computer, the first and second applications using a communication channel, the system comprising:

a non-intrusive and secure communications capture device, connected to the communications channel;

a network module, connected to the communications capture device and configured to process communications from a physical layer to a network layer substantially in real-time; and

a session reconstruction unit, connected to the network module and configured to process communications to an application layer in real-time, to group communications into transactions and to arrange transactions in a hierarchical data structure according to dependencies within the information contained in the transactions, the session reconstruction unit further comprising;

a stream creation unit, connected to the network module and configured to receive a plurality of communications and group them into a plurality of streams, and to add connection meta information to each stream, wherein each stream represents a single network connection;

a message decoder, connected to the stream creation unit and configured to create a plurality of transactions from the communications included in the plurality of streams;

a transaction storage, connected to the message decoder, configured to store the plurality of transactions; and

a session reconstruction module, connected to the transaction storage, and configured to receive a transaction of interest, and to retrieve a set of transactions from the transaction storage, the set of transactions being such that each transaction belonging to the set of transactions has a predefined relationship with the transaction of interest, and to group the set of transactions in the hierarchical data structure according to dependencies within the information contained in the transactions.

2. The system of claim 1, further including a transaction analyzer module, connected to the session reconstruction unit and configured to analyze transactions as they are being created by the session reconstruction unit.

3. The system of claim 1, wherein each transaction comprises;

transaction meta information;

a request header; and

a response header.

4. The system of claim 3, wherein at least one of the transactions further comprises a request body.

5. The system of claim 4, wherein the session reconstruction module further comprises:

a meta information based transaction pre-selection unit, connected to the transaction storage, and configured to receive the transaction of interest and to retrieve a part of the set of transactions from the transaction storage, based on the transaction meta information;

a transaction pre-selection buffer, connected to the meta information based transaction pre-selection unit, and configured to store the part of the set of transactions;

a header based transaction correlation unit, connected to the transaction pre-selection buffer and configured to find dependencies between transactions of the part of the set of transactions based on the request headers and response headers of transactions; and

a body based transaction correlation and hierarchical linking unit, connected to the header based transaction correlation unit and the transaction storage and configured to retrieve from the transaction storage additional transactions of the set of transactions based on the request bodies and response bodies of the part of the set of transactions already retrieved and to find dependencies between transaction of the set of transactions based on the request bodies and response bodies of the transactions of the set of transactions, and to group the set of transactions in a hierarchical data structure according to the found dependencies.

6. The system of claim 3, wherein at least one of the transactions further comprises a response body.

7. The system of claim 6, wherein the session reconstruction module further comprises:

a meta information based transaction pre-selection unit, connected to the transaction storage, and configured to receive the transaction of interest and to retrieve a part of the set of transactions from the transaction storage, based on the transaction meta information;

a transaction pre-selection buffer, connected to the meta information based transaction pre-selection unit, and configured to store the part of the set of transactions;

a header based transaction correlation unit, connected to the transaction pre-selection buffer and configured to find dependencies between transactions of the part of the set of transactions based on the request headers and response headers of transactions; and

a body based transaction correlation and hierarchical linking unit, connected to the header based transaction correlation unit and the transaction storage and configured to retrieve from the transaction storage additional transactions of the set of transactions based on the request bodies and response bodies of the part of the set of transactions already retrieved and to find dependencies between transactions of the set of transactions based on the request bodies and response bodies of the transactions of the set of transactions, and to group the set of transactions in a hierarchical data structure according to the found dependencies.

8. The system of claim 1, wherein the message decoder comprises:

a transport protocol decoder, configured to receive a second plurality of streams, derived from the first plurality of streams, and derive a plurality of request messages and a plurality of response messages from the second plurality of streams; and

a transaction assembler, connected to the transport protocol decoder and configured to receive the plurality of request messages and the plurality of response messages from the transaction protocol decoder and to combine the plurality of request messages and plurality of response messages into the plurality of transactions, wherein each transaction is created by combining a request message and a response message.

9. The system of claim 8, wherein the message decoder further comprises:

a secure traffic decryption unit, connected to the stream creation unit and to the transport protocol decoder and configured to receive the plurality of streams, wherein at least one stream within the plurality of streams is encrypted, and to decrypt the at least one encrypted stream forming a plurality of decrypted streams,

wherein the second plurality of streams is derived from the plurality of streams by replacing the at least one encrypted stream with the plurality of decrypted streams.

10. The system of claim 1 further comprising a plug-in monitoring application selected from the group consisting of:

a session visualization application, a statistics module, a monitoring and alerting module, a test script generation module, a service level validation module, a click path analysis module, and a customer support assistance application, the plug-in monitoring application being connected to the session reconstruction module and being configured to receive the hierarchical data structure.