Extensible and elastic data management services engine external to a storage domain

• US 10,769,212 B2
• Filed: 07/31/2015
• Issued: 09/08/2020
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Kenchammana-Hosekote, Deepak Raghu, Chaitanya, Shiva, Ghanbari, Saeed, Benight, Barry Patrick, Slik, David Anthony
• Primary Examiner(s): Hershley, Mark E
• Application Number:
  US14/815,470
• Publication Number:
  US 20170031955A1
• Time in Patent Office: 1,866 Days
• Field of Search: 707694
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/142 Managing session states for stateless protocols; Signalling session states; State transitions; Keeping-state mechanisms
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context

Abstract

A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Claims

1. A method comprising:

determining, by a data management services engine, that an input request is to be processed by a service based upon an evaluation of the input request using a service rule;

querying, by the data management services engine, a service map to identify a location of executable program code of the service to execute for deploying the service and service attributes of the service comprising a name of the service, a service identifier derived from the executable program code, and whether the service is stateful or stateless;

requesting, by the data management services engine, an allocation of resources for hosting the service based upon a determination that the allocation satisfies a consumption rule, wherein the allocation is based upon a capacity for executing the service determined using the service attributes and provisioned resources;

deploying, by the data management services engine, the service utilizing the allocated resources and the executable program code, wherein the service is deployed utilizing the service attributes to be accessible using the name and service identifier of the service, wherein the deploying comprises;

in response to the service attributes indicating that the service is stateful, configuring a services architecture to maintain state data for the service; and

in response to the service attributes indicating that the service is stateless, configuring the service to include a storage request within a service instance output for the storage request; and

updating service state information, within the data management services engine that deployed the service, to track at least one of the service identifier of the service, an available capacity of the service to process additional requests, a status of a time of a latest heartbeat of the service, or a deployed location of the programming code.

2. The method of claim 1, wherein the service map specifies an attribute of the service, and the method comprising:

executing the service based upon the attribute.

3. The method of claim 2, wherein the attribute is a compression ratio and the service is a compression service.

4. The method of claim 1, comprising:

monitoring the service state information to determine that the service failed to update the status within a predefined time window.

5. The method of claim 4, comprising:

transmitting a message to the service to determine whether the status is stale and a reason that the status is stale.

6. The method of claim 1, wherein the requesting comprises:

determining an amount of resources to request based upon the amount being less than a defined amount by the consumption rule.

7. The method of claim 1, comprising:

tracking an amount of resources consumed by services and which resources are available to support additional services.

8. The method of claim 1, comprising:

deploying a counterpart service as part of processing the input request based upon an analysis of data and metadata of the service indicating that the counterpart service is a counterpart of the service.

9. The method of claim 8, comprising:

prioritizing resource allocation for the counterpart service over other services.

10. A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to:

determine, by a data management services engine, that an input request is to be processed by a service based upon an evaluation of the input request using a service rule;

query, by the data management services engine, a service map to identify a location of executable program code of the service to execute for deploying the service and service attributes of the service comprising a name of the service, a service identifier derived from the executable program code, and whether the service is stateful or stateless;

request, by the data management services engine, an allocation of resources for hosting the service based upon a determination that the allocation satisfies a consumption rule, wherein the allocation is based upon a capacity for executing the service determined using the service attributes and provisioned resources;

deploy, by the data management services engine, the service utilizing the allocated resources and the executable program code, wherein the service is deployed utilizing the service attributes to be accessible using the name and service identifier of the service, comprising;

in response to the service attributes indicating that the service is stateful, configure a services architecture to maintain state data for the service; and

in response to the service attributes indicating that the service is stateless, configure the service to include a storage request within a service instance output for the storage request; and

update service state information, within the data management services engine that deployed the service, to track at least one of the service identifier of the service, an available capacity of the service to process additional requests, a status of a time of a latest heartbeat of the service, or a deployed location of the programming code.

11. The non-transitory machine readable medium of claim 10, wherein the instructions cause the machine to:

queue incoming input requests until a sufficient amount of resources are allocated for hosting the service.

12. The non-transitory machine readable medium of claim 10, wherein the instructions cause the machine to:

execute a command to release allocated resources based upon a reduction in workload of a storage domain external to a compute domain hosting the service.

13. The non-transitory machine readable medium of claim 10, wherein the instructions cause the machine to:

maintain a list of message types of input requests that are to be redirected to services, wherein the list is used to redirect the input request to the service based upon the input request having a message type mapped to the service, and wherein a second input request is redirected a second service based upon the second input request having a second message type mapped to the second service.

14. The non-transitory machine readable medium of claim 10, wherein the instructions cause the machine to:

assign a service request identifier to an I/O operation based upon a determination that the I/O operation is mapped to the service.

15. The non-transitory machine readable medium of claim 14, wherein the instructions cause the machine to:

create the service request identifier by concatenating a client address of the I/O operation and a target of the I/O operation.

16. A computing device comprising:

a memory comprising machine executable code for performing a method; and

a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to;

determine, by a data management services engine, that an input request is to be processed by a service based upon an evaluation of the input request using a service rule;

query, by the data management services engine, a service map to identify a location of executable program code of the service to execute for deploying the service and service attributes of the service comprising a name of the service, a service identifier derived from the executable program code, and whether the service is stateful or stateless;

request, by the data management services engine, an allocation of resources for hosting the service based upon a determination that the allocation satisfies a consumption rule, wherein the allocation is based upon a capacity for executing the service determined using the service attributes and provisioned resources;

deploy, by the data management services engine, the service utilizing the allocated resources and the executable program code, wherein the service is deployed utilizing the service attributes to be accessible using the name and service identifier of the service, comprising;

in response to the service attributes indicating that the service is stateful, configure a services architecture to maintain state data for the service; and

in response to the service attributes indicating that the service is stateless, configure the service to include a storage request within a service instance output for the storage request; and

update service state information, within the data management services engine that deployed the service, to track at least one of the service identifier of the service, an available capacity of the service to process additional requests, a status of a time of a latest heartbeat of the service, or a deployed location of the programming code.

17. The computing device of claim 16, wherein the machine executable code causes the processor to:

select an instance of the service from a set of instances of the service for processing I/O operations based upon configured parameters specified within metadata of the I/O operations.

18. The computing device of claim 17, wherein a configured parameter corresponds to a largest amount of available resources for satisfying a quality of service metric.

19. The computing device of claim 16, wherein the machine executable code causes the processor to:

process the input request for the service based upon whether the service is a guaranteed service.

20. The computing device of claim 16, wherein the machine executable code causes the processor to:

process the input request for the service based upon whether resources provisioned for the service are reserved.