Systems, methods, and apparatus to debug a network application

• US 8,245,083 B2
• Filed: 12/24/2009
• Issued: 08/14/2012
• Est. Priority Date: 12/24/2009
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Van Der Merwe, Jacobus, Caesar, Matthew Chapman, Lin, Chia-Chi
• Primary Examiner(s): SCHELL, JOSEPH O
• Application Number:
  US12/647,140
• Publication Number:
  US 20110161730A1
• Time in Patent Office: 964 Days
• Field of Search: 714/33, 714/43, 714/15, 714/16, 714/17
• US Class Current: 714/43
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: H04L 41/0661 by reconfiguring faulty entities
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/0863 by rolling back to previous configuration versions

Abstract

Methods and apparatus to debug a network application are described. A described example network includes a live control network to collect control messages to create a history of network states, the history of network states reflecting an order in which control messages are processed, the live control network to roll back from a current state to a past state upon detection of an improper sequence of messages and to process the messages in a corrected sequence, the corrected sequence to be stored in the history. The described example network further includes a virtualized network corresponding to the live control network, the virtualized network responsive to a command from an operator to step through the history to facilitate debugging.

Claims

1. A system comprising:

a debugging coordinator communicatively coupled to a plurality of virtual service nodes via a plurality of network nodes of a communication network, wherein the debugging coordinator comprises;

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to perform a method comprising;

instantiating a cloned network that includes a second set of virtual service nodes that includes at least one cloned virtual service node that is a clone of a corresponding virtual service node in a first set of virtual service nodes of the plurality of virtual services nodes, wherein the at least one cloned virtual service node has access to a history of events that occurred at the corresponding virtual service node in the first set of virtual service nodes; and

initiating an interactive debugging session that includes step by step processing by the cloned network of the events of the history of events, wherein the step by step processing of the events starts at a first event of the history of events, wherein the first event is identified by a checkpoint, wherein the step by step processing of the events includes sending a transmit command to each virtual service node in the second set of virtual service nodes, wherein the transmit command identifies a transmit phase, and wherein, during the transmit phase, each virtual service node in the second set of virtual service nodes is configured to transmit one or more packets transmitted during the first event by the corresponding virtual service node in the first set of virtual service nodes and to receive one or more packets received during the first event at the corresponding virtual service node in the first set of virtual service nodes.

2. The system of claim 1, wherein the first set of virtual service nodes is configured to route data messages via the communication network.

3. The system of claim 1, wherein a particular event in the history of events includes information related to a state of the corresponding virtual service node before the particular event, a state of the corresponding virtual service node after the particular event, an output of the particular event, one or more packets received during the particular event, and one or more packets transmitted during the particular event.

4. The system of claim 3, wherein each event in the history of events is ordered based on an order hierarchy that approximates an order in which one or more packets were received via the communication network.

5. The system of claim 1, wherein the step by step processing of the events includes sending a processing command to each of the virtual service node in the second set of virtual service nodes upon completion of the transmit phase, wherein the processing command identifies a processing phase, wherein, during the processing phase, each virtual service node in the second set of virtual service nodes is configured to process the one or more received packets received during the first event.

6. The system of claim 5, wherein the method further comprises:

determining whether a breakpoint that is associated with a condition of the communication network has been reached in response to completion of the processing phase; and

in response to a determination that the breakpoint has not been reached, sending a second transmit command and a second processing command to the second set of virtual service nodes, wherein the second transmit command and the second processing command are associated with a next event in the history of events.

7. The system of claim 1, wherein the method further comprises:

modifying a configuration of the cloned network in response to determining that a breakpoint has been reached; and

initiating a second interactive debugging session that includes step by step processing of the events of the history of events via the cloned network.

8. The system of claim 7, wherein modifying the configuration of the cloned network includes modifying a configuration of at least one cloned virtual service node in the cloned network.

9. The system of claim 8, wherein the method further comprises merging the modified configuration into at least one virtual service node of the first set of virtual service nodes.

10. A method comprising:

identifying a first set of virtual service coordinators of a plurality of virtual service coordinators, wherein each of the plurality of virtual service coordinators corresponds to a particular network node of a plurality of network nodes of a communication network, and wherein each of the plurality of virtual service coordinators is associated with at least one virtual service node;

instantiating a cloned network that includes a second set of virtual service coordinators that includes at least one cloned virtual service node that is a clone of a corresponding virtual service node in a first set of virtual service nodes of the plurality of virtual services nodes, wherein the at least one cloned virtual service node has access to a history of network events that occurred at the corresponding virtual service node in the first set of virtual service nodes;

sending an instruction to the second set of virtual service coordinators to load the history of network events;

sending breakpoint information to the second set of virtual service coordinators that identifies a condition of the communication network;

sending a step through instruction to initiate a step through mode to process the network events via the cloned network in a step by step execution mode;

determining whether the condition identified by the breakpoint information is satisfied based on state information collected from at least one virtual service coordinator of the cloned network; and

in response to determining that the condition has been satisfied, modifying a configuration of the cloned network.

11. The method of claim 10, wherein the first set of virtual service coordinators is configured to route data messages via the communication network.

12. The method of claim 10, wherein a first virtual service coordinator of the second set of virtual service coordinators is also included in the first set of virtual service coordinators.

13. The method of claim 10, further comprising sending a second step through instruction to initiate a second step through mode to process the network events via the cloned network after the configuration of the cloned network has been modified.

14. A method, comprising:

receiving a plurality of messages at a first virtual service coordinator of a plurality of virtual service coordinators of a communication network, wherein the first virtual service coordinator controls a first virtual service node;

storing the plurality of messages according to an order hierarchy to create a history of events, wherein the order hierarchy indicates an order in which each of the plurality of messages is received;

in response to determining that at least one message of the plurality of received messages does not conform to the order hierarchy, initiating a rollback operation that includes reordering the messages to conform to the order hierarchy to produce a modified history of events;

sending a rollback message to one or more additional virtual service coordinators of the plurality of virtual service coordinators to identify the at least one message that does not conform to the order hierarchy;

processing the plurality of messages via the first virtual service node based on the modified history of events;

storing information associated with the processing of the plurality of messages via the first virtual service node;

instantiating a second virtual service node at the first virtual service coordinator, wherein the second virtual service node is a clone of the first virtual service node; and

initiating an interactive debug operation that includes step by step processing of events in the modified history of events at the second virtual service node starting at an event identified by a checkpoint.

15. The method of claim 14, wherein the rollback operation includes modifying a software configuration of the first virtual service node based on a state of the first virtual node before receiving the at least one message, wherein the state is identified based on the history of events.

16. The method of claim 14, wherein each event in the history of events is associated with a particular event that occurred at the virtual service coordinator, wherein each event includes information associated with a state of the first virtual service coordinator before the particular event, a state of the first virtual service coordinator after the particular event, a final output of the particular event, one or more packets received during the particular event, and one or more packets transmitted during the particular event.

17. The method of claim 16, wherein the step by step processing includes:

transmitting one or more packets;

receiving one or more packets;

processing the one or more received packets; and

sending information to a debug coordinator via the communication network indicating a status of transmitting the one or more packets, receiving the one or more packets, processing the one or more packets, or any combination thereof.