SYSTEMS, METHODS, AND APPARATUS TO DEBUG A NETWORK APPLICATION

• US 20110161730A1
• Filed: 12/24/2009
• Published: 06/30/2011
• Est. Priority Date: 12/24/2009
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Caesar, Matthew Chapman, Lin, Chia-Chi, Van Der Merwe, Jacobus
• Granted Patent: US 8,245,083 B2
• Time in Patent Office: Days
• US Class Current: 714/15
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: H04L 41/0661 by reconfiguring faulty entities
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/0863 by rolling back to previous configuration versions

Abstract

Methods and apparatus to debug a network application are described. A described example network includes a live control network to collect control messages to create a history of network states, the history of network states reflecting an order in which control messages are processed, the live control network to roll back from a current state to a past state upon detection of an improper sequence of messages and to process the messages in a corrected sequence, the corrected sequence to be stored in the history. The described example network further includes a virtualized network corresponding to the live control network, the virtualized network responsive to a command from an operator to step through the history to facilitate debugging.

Claims

1. A network comprising:

a live control network to collect control messages to create a history of network states, the history of network states reflecting an order in which control messages are processed, the live control network to roll back from a current state to a past state upon detection of an improper sequence of messages and to process the messages in a corrected sequence, the corrected sequence to be stored in the history; and

a virtualized network corresponding to the live control network, the virtualized network responsive to a command from an operator to step through the history to facilitate debugging.

2. A network as defined in claim 1, wherein the live control network is implemented using first virtual service nodes running on first virtual service coordinators, the first virtual service nodes controlled by first runtime managers.

3. A network as defined in claim 2, wherein the virtualized control network is implemented using second virtual service nodes running on second virtual service coordinators, the second virtual service coordinators at least partially overlapping the first virtual service coordinators.

4. A network as defined in claim 1, wherein the command comprises a processing phase command or a transmitting phase command.

5. A network as defined in claim 1, wherein rolling hack the live control network comprises cleaning up an output message corresponding to the messages.

6. A network as defined in claim 1, wherein the order comprises a temporal order in which messages are normally received.

7. A network as defined in claim 1, wherein a change to the virtualized network is merged into the live control network to fix an error.

8. A method, comprising:

collecting control messages in a live control network to create a history of network states, the history of network states reflecting an order in which control messages are processed;

rolling back from a current state to a past state upon detection of an improper sequence of messages;

processing the messages in a corrected sequence;

storing the corrected sequence in the history;

instantiating a virtualized network corresponding to the live control network;

stepping through the history to facilitate debugging by an operator; and

repairing an error determined by the operator.

9. A method as defined in claim 8, wherein the order comprises a temporal order in which messages are normally received.

10. A method as defined in claim 8, wherein the live control network is implemented using first virtual service nodes running on first virtual service coordinators, the first virtual service nodes controlled by first runtime managers.

11. A method as defined in claim 10, wherein the virtualized control network is implemented using second virtual service nodes running on second virtual service coordinators, the second virtual service coordinators at least partially overlapping the first virtual service coordinators.

12. A method as defined in claim 8, wherein the command comprises a processing phase command or a transmitting phase command.

13. A method as defined in claim 8, wherein rolling back the live control network comprises cleaning up an output message corresponding to the messages.

14. An article of manufacture comprising machine readable instructions which, when executed, cause a machine to:

collect control messages in a live control network to create a history of network states, the history of network states reflecting an order in which control messages are processed;

roll back from a current state to a past state upon detection of an improper sequence of messages;

process the messages in a corrected sequence;

store the corrected sequence in the history;

instantiate a virtualized network corresponding to the live control network;

step through the history to facilitate debugging by an operator; and

repair an error determined by the operator.

15. An article of manufacture as defined in claim 14, wherein the order comprises a temporal order in which messages are normally received.

16. An article of manufacture as defined in claim 14, wherein the live control network is implemented using first virtual service nodes running on first virtual service coordinators, the first virtual service nodes controlled by first runtime managers.

17. An article of manufacture as defined in claim 16, wherein the virtualized control network is implemented using second virtual service nodes running on second virtual service coordinators, the second virtual service coordinators at least partially overlapping the first virtual service coordinators.

18. An article of manufacture as defined in claim 14, wherein the command comprises a processing phase command or a transmitting phase command.

19. An article of manufacture as defined in claim 14, wherein rolling back the live control network comprises cleaning up an output message corresponding to the messages.

20. An article of manufacture as defined in claim 14, wherein the order comprises a temporal order in which messages are normally received.

21. A method to debug a network application, comprising:

operating a network comprising a first plurality of virtual service nodes to provide network service to a customer;

cloning at least a portion of the first plurality of virtual service nodes to instantiate a second plurality of virtual service nodes;

identifying a network error using the second plurality of virtual service nodes; and

repairing the network error.

22. A method as defined in claim 21, further comprising processing a plurality of control packets according to a packet ordering using the first plurality of virtual service nodes.

23. A method as defined in claim 22, wherein identifying the network error comprises processing one or more of the control packets according to the packet ordering using the second plurality of virtual service nodes.

24. A method as defined in claim 22, wherein operating the network comprises rolling back the first plurality of virtual service nodes to a previous state when the packet ordering is violated.

25. A method as defined in claim 21, wherein identifying the network error comprises processing one or more received messages at the second plurality of service nodes in response to an operator command.

26. A method as defined in claim 21, wherein identifying the network error comprises transmitting one or more messages between the second plurality of service nodes in response to an operator command.

27. A method as defined in claim 21, wherein identifying the network error comprises causing the second plurality of virtual service nodes to transmit and process messages in lockstep and to pause transmission and processing when a breakpoint is reached.

28. A method as defined in claim 21, wherein identifying the network error comprises transmitting and processing messages according to an event history associated with the first plurality of virtual service nodes.

29. A method as defined in claim 21, wherein repairing the network error comprises merging a change to the second plurality of virtual service nodes into the first plurality of virtual service nodes.

30. A system to debug a network application, comprising:

a plurality of virtual service coordinators to provide network service to a customer via a first plurality of virtual service nodes, to clone at least a portion of the first plurality of virtual service nodes to instantiate a second plurality of virtual service nodes; and

a debugging coordinator to identify a network error using the second plurality of virtual service nodes and to repair the network error.

31. A system as defined in claim 30, wherein at least one of the first plurality of virtual service nodes and at least one of the second plurality of virtual service nodes is instantiated on a first one of the virtual service coordinators.

32. A system as defined in claim 30, wherein the first plurality of virtual service nodes process a plurality of control packets according to a packet ordering.

33. A system as defined in claim 32, wherein the debugging coordinator identifies the network error by causing the second plurality of virtual service nodes to process one or more of the control packets according to the packet ordering.

34. A system as defined in claim 32, wherein the virtual service coordinators roll back the first plurality of virtual service nodes to a previous state when the packet ordering is violated.

35. A system as defined in claim 30, wherein the debugging coordinator repairs the network error by merging a change to the second plurality of virtual service nodes into the first plurality of virtual service nodes.

36. An article of manufacture comprising machine readable instructions which, when executed, cause a machine to:

operate a network comprising a first plurality of virtual service nodes to provide network service to a customer;

clone at least a portion of the first plurality of virtual service nodes to instantiate a second plurality of virtual service nodes;

identify a network error using the second plurality of virtual service nodes; and

repair the network error.

37. An article of manufacture as defined in claim 36, wherein the instructions further cause the machine to process a plurality of control packets according to a packet ordering using the first plurality of virtual service nodes.

38. An article of manufacture as defined in claim 37, wherein identifying the network error comprises processing one or more of the control packets according to the packet ordering using the second plurality of virtual service nodes.

39. An article of manufacture as defined in claim 37, wherein operating the network comprises rolling back the first plurality of virtual service nodes to a previous state when the packet ordering is violated.

40. An article of manufacture as defined in claim 36, wherein repairing the network error comprises merging a change to the second plurality of virtual service nodes into the first plurality of virtual service nodes.

41. A method to debug a network application, comprising:

detecting an error in a network application running on a network;

instructing a plurality of nodes in the network to instantiate cloned virtual nodes representative of corresponding ones of the production nodes;

instructing the plurality of cloned virtual nodes to transmit messages and process messages;

instructing at least one of the virtual nodes to implement a repair for the detected error; and

merging the virtual nodes into the network.

42. A method as defined in claim 41, wherein rolling back the production nodes comprises reversing one or more effects of the exchanged messages and re-ordering the exchanged messages.

43. A method as defined in claim 42, wherein rolling back a first one of the production nodes comprises restoring a state of the first production node to a time prior to the one or more messages processed contrary to one or more ordering hierarchies.

44. A method as defined in claim 42, wherein reversing effects of the exchanged messages comprises sending a cleanup message to at least one of the production nodes.

45. A method as defined in claim 41, further comprising pausing processing of one or more messages on the virtual nodes in response to detecting an error.

46. A method as defined in claim 41, wherein instructing to transmit messages and process messages comprises instructing the plurality of cloned virtual nodes to transmit messages in lockstep and to process messages in lockstep.

47. A method as defined in claim 46, wherein operating the cloned virtual nodes in lockstep comprises instructing the cloned virtual nodes to transmit one or more messages to others of the cloned virtual nodes.

48. A method as defined in claim 46, wherein operating the cloned virtual nodes in lockstep comprises instructing the cloned virtual nodes to process messages received from others of the cloned virtual nodes.

49. A method as defined in claim 41, wherein the production nodes are implemented via virtual service nodes on corresponding virtual service coordinators.

50. A method as defined in claim 41, wherein the ordering hierarchy is based on a likely order of receiving messages at a production node.

51. A method as defined in claim 41, further comprising:

operating a network comprising a plurality of production nodes to exchange a plurality of messages;

operating the plurality of production nodes to process exchanged messages according to one or more ordering hierarchies;

rolling back the network in response to the production nodes processing one or more exchanged messages contrary to one or more ordering hierarchies;

52. A virtual service coordinator, comprising:

a manager queue to receive at least one of an event or a command from a debugging coordinator and to queue one or more message for delivery to the debugging coordinator;

a network queue to receive one or more messages from a network and to receive one or more messages to be delivered to the network;

a first runtime manager to control the execution of a first virtual service node by controlling access to a computing resource; and

a process coordinator in communication with the manager queue, the network queue, and the runtime manager to retrieve one or more messages from at least one of the manager queue or the network queue, to deliver one or more messages to at least one of the manager queue or the network queue, and to instantiate a second runtime manager and a second virtual service node in response to a command to clone a network node.

53. A virtual service coordinator as defined in claim 52, wherein the manager queue receives a command comprising at least one of a processing phase command, a transmission phase command, a breakpoint command, and/or a processing speed change command.

54. A virtual service coordinator as defined in claim 53, wherein the process coordinator causes the runtime manager to process one or more messages from the network queue in response to the manager queue receiving a processing phase command.

55. A virtual service coordinator as defined in claim 53, wherein the process coordinator causes the network queue to transmit one or more messages to the network in response to the manager queue receiving a transmission phase command.

56. A virtual service coordinator as defined in claim 53, wherein the network queue receives one or more messages from the runtime manager for transmission to the network in response to the manager queue receiving a processing phase command.

57. A virtual service coordinator as defined in claim 52, wherein the process coordinator determines whether a first message received in the network queue is received according to an ordering hierarchy.

58. A virtual service coordinator as defined in claim 57, wherein the process coordinator causes the first runtime manager to roll back the first virtual service node when the first message is not received according to the ordering hierarchy.

59. A virtual service coordinator as defined in claim 58, wherein the process coordinator transmits a cleanup message to the network to clean up one or more packets corresponding to the first message when the first message is not received according to the ordering hierarchy.

60. An article of manufacture comprising machine readable instructions which, when executed, cause a machine to:

detect an error in a network application running on a network;

instruct a plurality of nodes in the network to instantiate cloned virtual nodes representative of corresponding ones of the production nodes;

instruct the plurality of cloned virtual nodes to transmit messages and process messages;

instruct at least one of the virtual nodes to implement a repair for the detected error; and

merge the virtual nodes into the network.

61. An article of manufacture as defined in claim 60, wherein rolling back the production nodes comprises reversing one or more effects of the exchanged messages and re-ordering the exchanged messages.

62. An article of manufacture as defined in claim 61, wherein rolling back a first one of the production nodes comprises restoring a state of the first production node to a time prior to the one or more messages processed contrary to one or more ordering hierarchies.

63. An article of manufacture as defined in claim 60, wherein operating the cloned virtual nodes in lockstep comprises:

instructing the cloned virtual nodes to transmit one or more messages to others of the cloned virtual nodes during a first period; and

instructing the cloned virtual nodes to process received messages during a second period.

64. An article of manufacture as defined in claim 60, wherein the instructions further cause the machine to:

operating a network comprising a plurality of production nodes to exchange a plurality of messages;

operating the plurality of production nodes to process exchanged messages according to one or more ordering hierarchies;

rolling back the network in response to the production nodes processing one or more exchanged messages contrary to one or more ordering hierarchies;