Automated upgrade system for a service-based distributed computer system

• US 10,042,628 B2
• Filed: 06/30/2016
• Issued: 08/07/2018
• Est. Priority Date: 06/29/2016
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: OlmstedThompson, Jeremy
• Primary Examiner(s): Nguyen, Phillip H
• Application Number:
  US15/199,801
• Publication Number:
  US 20180004503A1
• Time in Patent Office: 768 Days
• Field of Search: 717168, 717170, 717171
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: H04L 41/0273 using web services for network management, e.g. simple object access protocol [SOAP]
• CPC Class Codes: H04L 41/082 the condition being updates or upgrades of network functionality
• CPC Class Codes: H04L 41/0886 Fully automatic configuration
• CPC Class Codes: H04L 41/0895 Configuration of virtualised networks or elements, e.g. virtualised network function or OpenFlow elements
• CPC Class Codes: H04L 41/0897 by horizontal or vertical scaling of resources, or by migrating entities, e.g. virtual resources or entities
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

The current document is directed to an automated upgrade subsystem within a distributed computer system that executes a distributed, service-based application comprising multiple local instances of the a distributed, service-based application, each instance including one or more control programs executing on a physical computer-system component of the distributed computer system. In one implementation, the automated upgrade subsystem provides a user interface that allows a system administrator or other user to launch a distributed-system-wide upgrade of a distributed application with a single input to a single input feature on a displayed user interface. The automated upgrade subsystem carries out a hierarchical, rolling upgrade, automatically generates snapshots for each physical computer system prior to launching local upgrade logic on the physical computer systems, detects upgrade problems and, in certain cases, automatically conducts a distributed-computer-system-wide rollback for problems that cannot be automatically or manually resolved.

Claims

1. A distributed computer system, comprising:

a processor; and

an automated upgrade subsystem within a distributed service-based application, multiple instances of which execute on multiple physical computer systems within the distributed computer system, the automated upgrade subsystem comprising;

an upgrade controller that displays an upgrade interface on a display device and that launches an automated upgrade in response to user input to an input feature of the upgrade interface by transmitting an initiate-upgrade request, for upgrading at least a portion of the distributed service-based application, to a selected upgrade service; and

upgrade services, within components of the distributed service-based application, that upgrade themselves to newer versions of upgrade services and request component upgrades from any subordinate upgrade services within lower-level components of the distributed service-based application, each upgrade service;

determining whether or not an upgrade of a component is possible, in response to receiving the initiate-upgrade request,

when the component upgrade is possible, preparing and storing a snapshot having information sufficient to restore the component to a pre-upgrade version,

when a snapshot has been successfully prepared, persistently storing the component upgrade, upgrading the component, and restarting the upgraded component, and

when the component is successfully restarted, verifying functionality of the upgraded component.

2. The automated upgrade subsystem of claim 1 wherein, after determining whether or not an upgrade is possible, when there are additional upgrade services subordinate to the upgrade service, the upgrade service transmits an initiate-upgrade request to the additional upgrade services to direct them to begin an upgrade.

3. The automated upgrade subsystem of claim 1 wherein, after preparing and storing a snapshot, when there are additional upgrade services subordinate to the upgrade service, the upgrade service transmits an initiate-upgrade request to the additional upgrade services to direct them to prepare and store a snapshot.

4. The automated upgrade subsystem of claim 1 wherein, after restarting the component, when there are additional upgrade services subordinate to the upgrade service, the upgrade service transmits a finish-upgrade request to additional upgrade services to direct them to persistently store a target version and restart the component.

5. The automated upgrade subsystem of claim 1 wherein, after verifying the functionality of the upgraded component, when there are additional upgrade services subordinate to the upgrade service, the upgrade service transmits a verify request to additional upgrade services to direct them to verify the target-version functionality that they previously stored and that is now executing following a restart.

6. The automated upgrade subsystem of claim 1 wherein an upgrade service receives an upgrade package from either a higher-level upgrade service or the upgrade controller, the upgrade package including:

information that allows the upgrade service to store new executable and data files in a local computational environment, such as a physical server, in order to create a persistent instance of an upgraded entity; and

one or more upgrade scripts and executables.

7. The automated upgrade subsystem of claim 1

wherein the distributed service-based application executes within a virtual machine on one or more physical processors of a physical computer system;

and

wherein the distributed service-based application comprises

one or more service processes that interact with remote computational entities through a service interface and a communications protocol selected from the REST protocol and the SOAP protocol, and

one or more internal processes that communicate with the service processes through an interface and a communications protocol selected from the REST protocol, the SOAP protocol, and a local-network protocol.

8. The automated upgrade subsystem of claim 1 wherein upgrade of the distributed service-based application proceeds sequentially, component-by-component or component-group-by-component-group.

9. The automated upgrade subsystem of claim 8 wherein, during upgrade of the distributed service-based application, those components of the distributed service-based application that are not currently upgrading in response to receiving initiate-upgrade requests or that have completed upgrading following successful verification continue to function as they function when the distributed service-based application is not being upgraded.

10. The automated upgrade subsystem of claim 8 wherein the displayed upgrade interface displays the current versions of distributed service-based application components and available target versions to which the distributed service-based application can be upgraded.

11. The automated upgrade subsystem of claim 1 wherein, when an upgrade service fails to restart the component, the upgrade service carries out a rollback operation, using the stored snapshot, to return the component being upgraded by the upgrade service to a pre-upgrade state.

12. The automated upgrade subsystem of claim 11 wherein, after returning the component being upgraded by the upgrade service to a pre-upgrade state, the upgrade service transmits a rollback request to the additional upgrade services to direct them to carry out a rollback operation.

13. The automated upgrade subsystem of claim 1 wherein, when an upgrade service fails to verify the functionality of the upgraded component, the upgrade service carries out a rollback operation, using the stored snapshot, to return the component being upgraded by the upgrade service to a pre-upgrade state.

14. The automated upgrade subsystem of claim 13 wherein, after returning the component being upgraded by the upgrade service to a pre-upgrade state, the upgrade service transmits a rollback request to the additional upgrade services to direct them to carry out a rollback operation.

15. The automated upgrade subsystem of claim 1 wherein, during upgrade of the distributed service-based application, when a network failure or other problem arises within a component of the distributed service-based application, the upgrade service controlling upgrade of the component may raise an exception or alert, pausing upgrade of the distributed service-based application.

16. The automated upgrade subsystem of claim 15 wherein, in response to the raised exception or alert, a higher-level upgrade service or the upgrade controller attempts to automatically address the network failure or other problem and, when the network failure or other problem cannot be automatically addressed, an alert is displayed on the upgrade interface to initiate manual intervention.

17. The automated upgrade subsystem of claim 16 wherein, when the network failure or other problem is addressed, the upgrade service that raised the exception or alert resumes upgrading the component.

18. The automated upgrade subsystem of claim 16 wherein, when the network failure or other problem is addressed, the upgrade service that raised the exception or alert carries out a rollback operation.