Dynamic update of a distributed message processing system

• US 9,367,301 B1
• Filed: 03/06/2013
• Issued: 06/14/2016
• Est. Priority Date: 03/06/2013
• Status: Active Grant

• Current Assignee: ServiceNow Incorporated
• Original Assignee: Attivio, Incorporated
• Inventors: Serrano, Martin Anthony, Johnson, William K. III
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): Smith, Cheneca
• Application Number:
  US13/787,047
• Time in Patent Office: 1,196 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1433 during software upgrading
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: G06F 8/656 while running
• CPC Class Codes: G06F 8/658 Incremental updates; Differential updates

Abstract

A distributed system with multiple nodes, each of which processes input data streams according to a set of independent, asynchronous processing components connected in a manner represented by a directed graph, is dynamically updated. Such a dynamic update occurs while the system is processing input data by pausing the input data stream, making changes to the system configuration, and then restarting the input data stream. If the update fails for any reason, the system configuration is rolled back to its prior state.

Claims

1. In a distributed message processing system, having a plurality of nodes implemented on one or more computers, each node independently processing input streams of data, according to a set of independent, asynchronous processing components connected in a manner represented by a directed graph, a process for dynamically updating, on each node, a runtime version of computer program defining the set of processing components during execution of the runtime version, comprising:

receiving a new runtime version of the computer program into memory;

determining differences between the new runtime version of the computer program and a current runtime version of the computer program;

preparing a change listing defining an ordered set of commands for each node to perform to modify the current runtime version of the computer program into the new runtime version of the computer program, such commands being invertible such that when an inverse of such commands are performed in reverse order, the current runtime version of the computer program is restored;

pausing the input streams of data from being provided to the nodes;

instructing each node to update the runtime version of the computer program from the current version to the new version;

monitoring progress of updates to the nodes;

after failure of the update by one or more nodes, instructing the nodes to rollback the update to the current runtime version of the computer program; and

after rollback or successful update by the nodes, resuming the input streams of data to be provided to the nodes.

2. The process of claim 1, wherein monitoring progress of updates to the nodes comprises:

maintaining a queue wherein each entry of the queue includes a command from the change list and a status for each of the nodes for completing the command.

3. The process of claim 2, further comprising receiving, from each of the nodes, for each command, a status for the node for the command.

4. The process of claim 3, wherein, for each command in the queue, each node waits for the other nodes to complete processing the command before proceeding to the next entry in the queue.

5. The process of claim 1, further comprising adding a new node during the update, and starting the new node with the configuration of the nodes after the update completes on the nodes.

6. The process of claim 5, further comprising restarting one of the nodes, and treating the restarted node as a new node.

7. The process of claim 1, wherein the computer program is defined by a tree-based data structure.

8. The process of claim 7, wherein determining the differences between the new runtime version of the computer program and the current runtime version of the computer program comprises applying a tree-based difference algorithm.

9. A distributed message processing system comprising:

a plurality of nodes implemented on one or more computers, each node independently processing input streams of data, according to a set of independent, asynchronous processing components connected in a manner represented by a directed graph;

a computer operating an update manager configured to perform a process for dynamically updating, on each node, a runtime version of computer program defining the set of processing components during execution of the runtime version, the update manager configured to;

receive a new runtime version of the computer program into memory;

determine differences between the new runtime version of the computer program and a current runtime version of the computer program;

generate a change listing defining an ordered set of commands for each node of the plurality of nodes to perform to modify the current runtime version of the computer program into the new runtime version of the computer program, such commands being invertible such that when an inverse of such commands are performed in reverse order, the current runtime version of the computer program is restored;

pause the input streams of data from being provided to the plurality of nodes;

instruct each node of the plurality of nodes to update the runtime version of the computer program from the current version to the new version;

monitor progress of updates to the plurality of nodes;

after failure of the update by one or more nodes of the plurality of nodes, instruct all of the nodes to rollback the update to the current runtime version of the computer program; and

after rollback or successful update by the plurality of nodes, resume the input streams of data to be provided to the plurality of nodes.

10. The distributed message processing system of claim 9, wherein to monitor progress of updates to the nodes, the update manager is further configured to:

maintain a queue wherein each entry of the queue includes a command from the change list and a status for each of the nodes for completing the command.

11. The distributed message processing system of claim 10, wherein to maintain the queue, the update manager is further configured to receive, from each of the nodes, for each command, a status for the node for the command.

12. The distributed message processing system of claim 11, wherein, for each command in the queue, each node waits for the other nodes to complete processing the command before proceeding to the next entry in the queue.

13. The distributed message processing system of claim 9, wherein the update manager is further configured to add a new node during the update, wherein the new node is started with the configuration of the nodes after the update completes on the nodes.

14. The distributed message processing system of claim 13, wherein when one of the nodes is restarted, and the update manager is configured to treat the restarted node as a new node.

15. The distributed message processing system of claim 9, wherein the computer program is defined by a tree-based data structure.

16. The distributed message processing system of claim 15, wherein to determine the differences between the new runtime version of the computer program and the current runtime version of the computer program, the update manager is configured to apply a tree-based difference algorithm.

17. An article of manufacture comprising a non-transitory computer storage medium storing instructions that when executed by a computer configure the computer to provide an update manager for a distributed message processing system having a plurality of nodes implemented on one or more computers, each node independently processing input streams of data, according to a set of independent, asynchronous processing components connected in a manner represented by a directed graph, the update manager configured to perform a process for dynamically updating, on each node, a runtime version of a computer program defining the set of processing components during execution of the runtime version, the update manager configured to:

receive a new runtime version of the computer program into memory;

determine differences between the new runtime version of the computer program and a current runtime version of the computer program;

generate a change listing defining an ordered set of commands for each node of the plurality of nodes to perform to modify the current runtime version of the computer program into the new runtime version of the computer program, such commands being invertible such that when an inverse of such commands are performed in reverse order, the current runtime version of the computer program is restored;

pause the input streams of data from being provided to the plurality of nodes;

instruct each node of the plurality of nodes to update the runtime version of the computer program from the current version to the new version;

monitor progress of updates to the plurality of nodes;

after failure of the update by one or more nodes of the plurality of nodes, instruct the plurality of nodes to rollback the update to the current runtime version of the computer program; and

after rollback or successful update by the plurality of nodes, resume the input streams of data to be provided to the plurality of nodes.

18. The article of manufacture of claim 17, wherein to monitor progress of updates to the nodes, the update manager is further configured to:

maintain a queue wherein each entry of the queue includes a command from the change list and a status for each of the nodes for completing the command.

19. The article of manufacture of claim 18, wherein to maintain the queue, the update manager is further configured to receive, from each of the nodes, for each command, a status for the node for the command.

20. The article of manufacture of claim 19, wherein, for each command in the queue, each node waits for the other nodes to complete processing the command before proceeding to the next entry in the queue.