Cluster-wide service agents

• US 10,255,146 B2
• Filed: 12/04/2016
• Issued: 04/09/2019
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Buzzard, Gregory, Cady, Justin Travis, Snyder, Thomas Gilbert, Venneti, Satya R., Yucel, Sakir
• Primary Examiner(s): Ko, Chae M
• Application Number:
  US15/368,609
• Publication Number:
  US 20170139790A1
• Time in Patent Office: 856 Days
• Field of Search: 714 63
• CPC Class Codes: G06F 11/2017 where memory access, memory control or I/O control functionality is redundant redundant communication control functionality G06F11/2005; redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: H04L 61/5007 Internet protocol [IP] addresses
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

One or more techniques and/or systems are provided for cluster configuration information replication, managing cluster-wide service agents, and/or for cluster-wide outage detection. In an example of cluster configuration information replication, a replication workflow corresponding to a storage operation implemented for a storage object (e.g., renaming of a volume) of a first cluster may be transferred to a second storage cluster for selectively implementation. In an example of managing cluster-wide service agents, cluster-wide service agents are deployed to nodes of a cluster storage environment, where a master agent actively processes cluster service calls and standby agents passively wait for reassignment as a failover master in the event the master agent fails. In an example of cluster-wide outage detection, a cluster-wide outage may be determined for a cluster storage environment based upon a number of inaccessible nodes satisfying a cluster outage detection metric.

Claims

1. A computing device, comprising:

a memory containing having stored thereon instructions for performing a method; and

a processor coupled with the memory, the processor configured to execute the instructions to cause the processor to;

assign a first agent, deployed to a first node, as a master agent configured to actively process cluster service API calls and a second agent, deployed to a second node, as a standby agent, wherein the first agent and the second agent comprise an acquire master duties interface used to be assigned as the master agent and a release master duties interface used to relinquish assignment from being the master agent;

reassign the second agent as the master agent, wherein the first agent executes the release master duties interface to relinquish assignment from being the master agent and the second agent subsequently executes the acquire master duties interface to being assigned as the master agent for processing the cluster service API calls to a set of storage cluster services; and

instruct the second agent, as the master agent, to perform a clean-up of prior cluster service API call processing that was in progress by the first agent before the reassignment.

2. The computing device of claim 1, wherein the instructions cause the processor to:

automatically reassign the second agent as the master agent in real-time based upon a service interruption tolerance metric to mitigate service interruption of cluster service API calls.

3. The computing device of claim 1, the first agent comprising a first physical process configured to actively process API calls for the set of storage cluster services while having a master agent assignment.

4. The computing device of claim 3, the second agent comprising a second physical process, as a second instance of the first physical process, configured to actively process API calls for the set of storage cluster services while having the master agent assignment.

5. The computing device of claim 1, wherein the instructions cause the processor to:

reassign the first agent as the standby agent.

6. The computing device of claim 1, wherein the instructions cause the processor to:

expose an initialization interface and a shutdown interface to the first agent and the second agent for the set of storage cluster services.

7. The computing device of claim 6, the initialization interface and the shutdown interface exposed according to an asynchronous and idempotent implementation.

8. The computing device of claim 1, wherein the instructions cause the processor to:

maintain a storage cluster service data cache for use by the master agent, wherein a first IP address of the first agent, as the master agent, is assigned to the storage cluster service data cache.

9. The computing device of claim 8, wherein the instructions cause the processor to:

purge the storage cluster service data cache to replace the first IP address with a second IP address of the second agent for the storage cluster service data cache based upon the first agent failing.

10. The computing device of claim 9, wherein the instructions cause the processor to:

retarget an inter process remote procedure call to the master agent based upon the assignment of the second IP address to the storage cluster service data cache.

11. The computing device of claim 9, wherein the instructions cause the processor to:

retarget an inter process remote procedure call according to an original network configuration of a cluster storage environment comprising the first node and the second node before the reassignment of the master agent.

12. The computing device of claim 1, wherein the instructions cause the processor to:

define a sync point for the master agent.

13. The computing device of claim 1, wherein the instructions cause the processor to:

enforce a single master policy for the first agent and the second agent.

14. The computing device of claim 1, wherein the instructions cause the processor to:

evaluate a first set of operational statistics for the first agent against a second set of operational statistics for the second agent to award the first agent as the master agent.

15. The computing device of claim 1, wherein the instructions cause the processor to:

identify a failure of the first agent based upon a first set of operational statistics for the first agent indicating that the first agent is not satisfying a performance criteria.

16. A method, comprising:

assigning a first agent, deployed to a first node, as a master agent configured to actively process cluster service API calls and a second agent, deployed to a second node, as a standby agent, wherein the first agent and the second agent comprise an acquire master duties interface used to be assigned as the master agent and a release master duties interface used to relinquish assignment from being the master agent;

reassigning the second agent as the master agent, wherein the first agent executes the release master duties interface to relinquish assignment from being the master agent and the second agent subsequently executes the acquire master duties interface to being assigned as the master agent for processing the cluster service API calls to a set of storage cluster services; and

instructing the second agent, as the master agent, to perform a clean-up of prior cluster service API call processing that was in progress by the first agent before the reassignment.

17. The method of claim 16, the reassigning comprising:

automatically reassigning the second agent as the master agent in real-time based upon a service interruption tolerance metric to mitigate service interruption of cluster service API calls.

18. The method of claim 16, comprising:

evaluating a first set of operational statistics for the first agent against a second set of operational statistics for the second agent to award the first agent as the master agent.

19. The method of claim 16, comprising:

maintaining a storage cluster service data cache for use by the master agent.

20. A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to:

assign a first agent, deployed to a first node, as a master agent configured to actively process cluster service API calls and a second agent, deployed to a second node, as a standby agent, wherein the first agent and the second agent comprise an acquire master duties interface used to be assigned as the master agent and a release master duties interface used to relinquish assignment from being the master agent;

reassign the second agent as the master agent, wherein the first agent executes the release master duties interface to relinquish assignment from being the master agent and the second agent subsequently executes the acquire master duties interface to being assigned as the master agent for processing the cluster service API calls to a set of storage cluster services; and

instruct the second agent, as the master agent, to perform a clean-up of prior cluster service API call processing that was in progress by the first agent before the reassignment.