Systems and methods for extracting structured application data from a communications link

• US 9,100,291 B2
• Filed: 01/25/2013
• Issued: 08/04/2015
• Est. Priority Date: 01/31/2012
• Status: Active Grant

• Current Assignee: QuantumXchange, Inc.
• Original Assignee: DB Networks Inc.
• Inventors: Ruddick, Timothy W., Varsanyi, Eric, Paterson, Charles A., Rosenberg, David A.
• Primary Examiner(s): Golabbakhsh, Ebrahim
• Application Number:
  US13/750,579
• Publication Number:
  US 20130194949A1
• Time in Patent Office: 921 Days
• US Class Current: 1/1
• CPC Class Codes: H04L 43/028 by filtering
• CPC Class Codes: H04L 43/065 related to network devices
• CPC Class Codes: H04L 43/12 Network monitoring probes

Abstract

Systems and methods for generating a semantic description of operations between network agents. In an embodiment, packet-level traffic between two or more network agents is captured. The packet-level traffic is bundled into one or more messages, wherein each message comprises one or more elements. For each of the messages, the elements of the message are matched to one or more attributes, and the message is decoded into message data based on the matched attributes. The message data is then used to generate a semantic description of operations between the network agents.

Claims

1. A method for generating a semantic description of operations between network agents, the method comprising, using at least one hardware processor:

capturing packet-level traffic between a first network agent and a second network agent;

bundling the packet-level traffic into one or more messages, wherein each of the one or more messages comprises a plurality of elements;

for each of the one or more messages,

matching one or more of the one or more elements of the message to one or more attributes, wherein matching one or more of the one or more elements of the message to the one or more attributes comprises, for each template in a set of one or more templates,

selecting the template, wherein the template comprises a plurality of attributes,

comparing one or more of the one or more attributes of the template to one or more of the one or more elements of the message,

determining whether the template matches the message based on the comparison, and,

if it is determined that the template matches the message, locating one or more unmatched ones of the plurality of elements of the message using one or more inferred ones of the plurality of attributes of the template, and

decoding the message into message data based on the matched one or more attributes; and

generating a semantic description of operations between the first network agent and the second network agent based on the message data.

2. The method of claim 1, wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a switch positioned on a communicative path between the first network agent and the second network agent.

3. The method of claim 2, wherein the switch is a virtual switch.

4. The method of claim 1, wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a network tap positioned on a communicative path between the first network agent and the second network agent.

5. The method of claim 4, wherein the network tap is a virtual network tap.

6. The method of claim 1, wherein bundling the packet-level traffic into one or more messages comprises, iteratively, for each of one more sessions:

receiving a packet transmitted between the first network agent and the second network agent;

determining whether a boundary condition has occurred, wherein the boundary condition indicates an end of a message;

if it is determined that the boundary condition has not occurred, queuing the received packet in memory; and

,

if it is determined that the boundary condition has occurred, bundling one or more packets queued in the memory into a message.

7. The method of claim 6, wherein determining whether a boundary condition has occurred comprises one or more of:

detecting control information indicating an end of a message;

detecting a change in direction of communication between the first network agent and the second network agent based on the received packet; and

detecting that a timeout has occurred since reception of a packet.

8. The method of claim 1, further comprising selecting the set of one or more templates from a plurality of templates.

9. The method of claim 8, wherein the set of one or more templates is selected based on startup information for a connection between the first network agent and the second network agent.

10. The method of claim 8, wherein the set of one or more templates is selected based on a stored set of one or more templates used for a prior connection between the first network agent and the second network agent.

11. The method of claim 1, further comprising, for each of the one or more messages:

selecting one or more templates that match the message from the set of one or more templates; and

decoding the message based on the one or more attributes of the selected one or more templates.

12. The method of claim 1, wherein the set of one or more templates comprises one or more templates corresponding to one or more of Transmission Control Protocol (TCP), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Sequenced Packet Exchange protocol (SPX), Address Resolution Protocol (ARP), Transparent Network Substrate protocol (TNS), Tabular Data Stream protocol (TDS), Symmetric Multiprocessing protocol (SMP), Two Task Common protocol (TTC), Network File System protocol (NFS), Apple Filing Protocol (AFP), Server Message Block protocol (SMB), Domain Name System protocol (DNS), Simple Mail Transfer Protocol (SMTP), Internet Message Access Protocol (IMAP), and Post Office Protocol (POP).

13. The method of claim 1, further comprising discarding at least a portion of one or both of the packet-level traffic and message data, based on one or more filters, prior to generating the semantic description of operations.

14. The method of claim 1, wherein the one or more messages comprise one or more of a request message for a remote procedure call (RPC) and a response message for an RPC, and wherein decoding each of the one or more messages into message data comprises extracting one or more of verbs, parameters, and result payloads for an RPC.

15. The method of claim 1, wherein the capturing is performed by a capture module, wherein the bundling, matching, decoding, and generating is performed by an analysis module, and wherein the method further comprises passing the packet-level traffic from the capture module to the analysis module using at least one application programming interface (API).

16. The method of claim 15, wherein the capture module and analysis module are executed on separate machines.

17. The method of claim 15, wherein the capture module and analysis module are executed on one machine.

20. The system of claim 10, wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a switch positioned on a communicative path between the first network agent and the second network agent.

21. The system of claim 20, wherein the switch is a virtual switch.

18. A method for generating a semantic description of operations between network agents, the method comprising, using at least one hardware processor:

capturing packet-level traffic between a first network agent and a second network agent;

bundling the packet-level traffic into one or more messages, wherein each of the one or more messages comprises one or more elements;

for each of the one or more messages,

matching one or more of the one or more elements of the message to one or more attributes, wherein matching one or more of the one or more elements of the message to one or more attributes comprises, for each template in a set of one or more templates,

selecting the template, wherein the template comprises one or more attributes,

comparing one or more of the one or more attributes of the template to one or more of the one or more elements of the message,

determining whether the template matches the message based on the comparison, and,

if it is determined that the template matches the message, determining whether the template comprises one or more references to additional templates, and, if it is determined that the template comprises one or more references to additional templates, adding one or more of the additional templates to the set of one or more templates, and

decoding the message into message data based on the matched one or more attributes; and

generating a semantic description of operations between the first network agent and the second network agent based on the message data.

19. A system for generating a semantic description of operations between network agents, the system comprising:

at least one hardware processor; and

at least one executable software module that, when executed by the at least one hardware processor,

captures packet-level traffic between a first network agent and a second network agent,

bundles the packet-level traffic into one or more messages, wherein each of the one or more messages comprises a plurality of elements,

for each of the one or more messages,

matches one or more of the one or more elements of the message to one or more attributes, wherein matching one or more of the one or more elements of the message to the one or more attributes comprises, for each template in a set of one or more templates,

selecting the template, wherein the template comprises a plurality of attributes,

comparing one or more of the one or more attributes of the template to one or more of the one or more elements of the message,

determining whether the template matches the message based on the comparison, and,

if it is determined that the template matches the message, locating one or more unmatched ones of the plurality of elements of the message using one or more inferred ones of the plurality of attributes of the template, and

decodes the message into message data based on the matched one or more attributes, and

generates a semantic description of operations between the first network agent and the second network agent based on the message data.

22. The system of claim 19, wherein capturing packet-level traffic between a first network agent and a second network agent comprises receiving copies of packets from a network tap positioned on a communicative path between the first network agent and the second network agent.

23. The system of claim 22, wherein the network tap is a virtual network tap.

24. The system of claim 19, wherein bundling the packet-level traffic into one or more messages comprises, iteratively, for each of one more sessions:

receiving a packet transmitted between the first network agent and the second network agent;

determining whether a boundary condition has occurred, wherein the boundary condition indicates an end of a message;

if it is determined that the boundary condition has not occurred, queuing the received packet in memory; and

,

if it is determined that the boundary condition has occurred, bundling one or more packets queued in the memory into a message.

25. The system of claim 24, wherein determining whether a boundary condition has occurred comprises one or more of:

detecting control information indicating an end of a message;

detecting a change in direction of communication between the first network agent and the second network agent based on the received packet; and

detecting that a timeout has occurred since reception of a packet.

26. The system of claim 19, wherein the at least one executable software module selects the set of one or more templates from a plurality of templates.

27. The system of claim 26, wherein the set of one or more templates is selected based on startup information for a connection between the first network agent and the second network agent.

28. The system of claim 26, wherein the set of one or more templates is selected based on a stored set of one or more templates used for a prior connection between the first network agent and the second network agent.

29. The system of claim 19, wherein the at least one executable software module, for each of the one or more messages:

selects one or more templates that match the message from the set of one or more templates; and

decodes the message based on the one or more attributes of the selected one or more templates.

30. The system of claim 19, wherein the set of one or more templates comprises one or more templates corresponding to one or more of Transmission Control Protocol (TCP), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Sequenced Packet Exchange protocol (SPX), Address Resolution Protocol (ARP), Transparent Network Substrate protocol (TNS), Tabular Data Stream protocol (TDS), Symmetric Multiprocessing protocol (SMP), Two Task Common protocol (TTC), Network File System protocol (NFS), Apple Filing Protocol (AFP), Server Message Block protocol (SMB), Domain Name System protocol (DNS), Simple Mail Transfer Protocol (SMTP), Internet Message Access Protocol (IMAP), and Post Office Protocol (POP).

31. The system of claim 19, wherein the at least one executable software module discards at least a portion of one or both of the packet-level traffic and message data, based on one or more filters, prior to generating the semantic description of operations.

32. The system of claim 19, wherein the one or more messages comprise one or more of a request message for a remote procedure call (RPC) and a response message for an RPC, and wherein decoding each of the one or more messages into message data comprises extracting one or more of verbs, parameters, and result payloads for an RPC.

33. The system of claim 19, wherein the capturing is performed by a capture module, wherein the bundling, matching, decoding, and generating is performed by an analysis module, and wherein the capture module passes the packet-level traffic to the analysis module using at least one application programming interface (API).

34. The system of claim 33, wherein the system further comprises a first machine and a second machine, and wherein the capture module is hosted on the first machine and the analysis module is hosted on the second machine.

35. The system of claim 33, wherein the system further comprises a single machine that comprises the at least one hardware processor, the capture module, and the analysis module.

36. A system for generating a semantic description of operations between network agents, the system comprising:

at least one hardware processor; and

at least one executable software module that, when executed by the at least one hardware processor,

captures packet-level traffic between a first network agent and a second network agent,

bundles the packet-level traffic into one or more messages, wherein each of the one or more messages comprises one or more elements,

for each of the one or more messages,

matches one or more of the one or more elements of the message to one or more attributes, wherein matching one or more of the one or more elements of the message to one or more attributes comprises, for each template in a set of one or more templates,

selecting the template, wherein the template comprises one or more attributes,

comparing one or more of the one or more attributes of the template to one or more of the one or more elements of the message,

determining whether the template matches the message based on the comparison, and,

if it is determined that the template matches the message, determining whether the template comprises one or more references to additional templates, and, if it is determined that the template comprises one or more references to additional templates, adding one or more of the additional templates to the set of one or more templates, and

decodes the message into message data based on the matched one or more attributes, and

generates a semantic description of operations between the first network agent and the second network agent based on the message data.