Orchestration of software applications upgrade using automatic hang detection

• US 9,740,474 B2
• Filed: 10/28/2015
• Issued: 08/22/2017
• Est. Priority Date: 10/29/2014
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Hu, Fang, Vong, Chi Kin, Thiagarajan, Bhuvaneswari, Richardson, Tim, Lu, Jinghuan Jay
• Primary Examiner(s): Vo, Ted T
• Application Number:
  US14/925,354
• Publication Number:
  US 20160124741A1
• Time in Patent Office: 664 Days
• Field of Search: 717168-178, 717127-128
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

In an upgrade infrastructure performing an overall upgrade operation comprising multiple upgrade processes being executed, possibly concurrently, on multiple hosts for upgrading one or more software applications hosted by hosts, automated hang detection mechanisms are disclosed for quickly, efficiently, and automatically detecting when one or more of the upgrade process are in a hang state. Different hang detection techniques are described including a metadata-driven hang detection mechanism and a code-driven hang detection mechanism.

Claims

1. A method comprising:

monitoring execution of a first upgrade process on a first host machine, the first upgrade process upgrading a first software application on the first host machine;

accessing, by a computing system, a runtime execution time for the first upgrade process, the runtime execution time captured for the first upgrade process during the monitoring of the execution of the first upgrade process;

determining a first reference time defined for the first upgrade process for the first host machine, wherein the first reference time is different than a second reference time defined for the first upgrade process for a second host machine;

determining a latency tolerance time for the first upgrade process;

determining, based on the runtime execution time, that the first upgrade process continues execution after a total of the first reference time and the latency tolerance time has passed;

determining, by the computing system that the first upgrade process executing on the first host machine is to be indicated as being in a hang state; and

generating, by the computing system, an alert message indicating the first upgrade process executing on the first host machine is in the hang state.

2. The method of claim 1, wherein accessing the runtime execution time includes:

receiving the runtime execution time from the first upgrade process;

or

retrieving the runtime execution time from the first upgrade process;

or

retrieving the runtime execution time from a storage, wherein the first upgrade process writes the runtime execution time to the storage.

3. The method of claim 1, further comprising:

monitoring execution of a second upgrade process on the first host machine, the second upgrade process upgrading a second software application on the first host machine;

accessing, by a computing system, runtime execution timing data for the second upgrade process, the runtime execution timing data for the second upgrade process captured for the second upgrade process during the monitoring of the execution of the second upgrade process;

comparing, by the computing system, the runtime execution timing data associated with the second upgrade process to a first reference timing data specified for the second upgrade process for the first host machine, wherein the first reference timing data specified for the second upgrade process for the first host machine is different from a second reference timing data specified for the second upgrade process for the second host machine;

determining, by the computing system, based upon the comparing, that the second upgrade process executing on the first host machine is to be indicated as being in a hang state; and

generating, by the computing system, an alert message indicating the second upgrade process executing on the first host machine is in the hang state.

4. The method of claim 1, wherein the first reference time indicates a time length or a time of completion for execution of the first upgrade process on the first host machine.

5. The method of claim 1, wherein the first reference time is pre-determined prior to the execution of the first upgrade process on the first host machine or the first reference time is determined during execution of the first upgrade process on the first host machine.

6. A non-transitory computer-readable storage memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:

instructions that cause at least one processor from the one or more processors to monitor execution of a first upgrade process on a first host machine, the first upgrade process upgrading a first software application on the first host machine;

instructions that cause at least one processor from the one or more processors to access runtime execution time for the first upgrade process, the runtime execution time captured for the first upgrade process during the monitoring of the execution of the first upgrade process;

instructions that cause at least one processor from the one or more processors to determine a first reference time defined for the first upgrade process for the first host machine, wherein the first reference time is different than a second reference time defined for the first upgrade process for a second host machine;

instructions that cause at least one processor from the one or more processors to determine a latency tolerance time for the first upgrade process;

instructions that cause at least one processor from the one or more processors to determine, based on the runtime execution time, that the first upgrade process continues execution after a total of the first reference time and the latency tolerance time has passed;

instructions that cause at least one processor from the one or more processors to determine that the first upgrade process executing on the first host machine is to be indicated as being in a hang state; and

instructions that cause at least one processor from the one or more processors to generate an alert message indicating the first upgrade process executing on the first host machine is in the hang state.

7. The non-transitory computer-readable storage memory of claim 6, wherein the instructions that cause at least one processor from the one or more processors to access runtime execution time associated with the first upgrade process based on monitoring further comprise:

instructions that cause at least one processor from the one or more processors to receive the runtime execution time from the first upgrade process;

or

instructions that cause at least one processor from the one or more processors to retrieve the runtime execution time from the first upgrade process;

or

instructions that cause at least one processor from the one or more processors to retrieve the runtime execution time from a storage, wherein the first upgrade process writes the runtime execution time to the storage.

8. The non-transitory computer-readable storage memory of claim 6, wherein the plurality of instructions further comprises:

instructions that cause at least one processor from the one or more processors to monitor monitoring execution of a second upgrade process on the first host machine, the second upgrade process upgrading a second software application on the first host machine;

instructions that cause at least one processor from the one or more processors to access runtime execution timing data for the second upgrade process, the runtime execution timing data captured for the second upgrade process during the monitoring of the execution of the second upgrade process;

instructions that cause at least one processor from the one or more processors to compare the runtime execution timing data associated with the second upgrade process to a first reference timing data specified for the second upgrade process for the first host machine, wherein the first reference timing data specified for the second upgrade process for the first host machine is different from a second reference timing data specified for the second upgrade process for the second host machine;

instructions that cause at least one processor from the one or more processors to determine, based upon the comparing, that the second upgrade process executing on the first host machine is to be indicated as being in a hang state; and

instructions that cause at least one processor from the one or more processors to generate an alert message indicating the second upgrade process executing on the first host machine is in the hang state.

9. The non-transitory computer-readable storage memory of claim 6, wherein the first reference time indicates a time length or a time of completion for execution of the first upgrade process on the first host machine.

10. The non-transitory computer-readable storage memory of claim 6, wherein the first reference time is pre-determined prior to the execution of the first upgrade process on the first host machine or the first reference time is determined during execution of the first upgrade process on the first host machine.

11. A system comprising:

one or more processors; and

a memory coupled with and readable by the one or more processors, the memory configured to store a set of instructions which, when executed by the one or more processors, causes at least one processor from the one or more processors to;

monitor execution of a first upgrade process on a first host machine, the first upgrade process upgrading a first software application on the first host machine;

access runtime execution time for the first upgrade process, the runtime execution time captured for the first upgrade process during the monitoring of the execution of the first upgrade process;

determine a first reference time defined for the first upgrade process for the first host machine, wherein the first reference time is different than a second reference time defined for the first upgrade process for a second host machine;

determine a latency tolerance time for the first upgrade process;

determine, based on the runtime execution time, that the first upgrade process continues execution after a total of the first reference time and the latency tolerance time has passed;

determine that the first upgrade process executing on the first host machine is to be indicated as being in a hang state; and

generate an alert message indicating the first upgrade process executing on the first host machine is in the hang state.

12. The system of claim 11, wherein accessing the runtime execution timing data includes:

receiving the runtime execution time from the first upgrade process;

or

retrieving the runtime execution time from the first upgrade process;

or

retrieving the runtime execution time from a storage, wherein the first upgrade process writes the runtime execution time to the storage.

13. The system of claim 11, wherein the set of instructions further comprises instructions, which when executed by the one or more processors, cause the one or more processors to:

monitor execution of a second upgrade process on the first host machine, the second upgrade process upgrading a second software application on the first host machine;

access runtime execution timing data for the second upgrade process, the runtime execution timing data for the second upgrade process captured for the second upgrade process during the monitoring of the execution of the second upgrade process;

compare the runtime execution timing data associated with the second upgrade process to a first reference timing data specified for the second upgrade process for the first host machine, wherein the first reference timing data specified for the second upgrade process for the first host machine is different from a second reference timing data specified for the second upgrade process for the second host machine;

determine, based upon the comparing, that the second upgrade process executing on the first host machine is to be indicated as being in a hang state; and

generate an alert message indicating the second upgrade process executing on the first host machine is in the hang state.