OPTIMIZED RESOURCE METERING IN A MULTI TENANTED DISTRIBUTED FILE SYSTEM

• US 20180052714A1
• Filed: 08/16/2016
• Published: 02/22/2018
• Est. Priority Date: 08/16/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Badjatia, Sudeep, Bhose, Rajarshi, Deb, Bijoy, Nandi, Ripon
• Granted Patent: US 10,120,724 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: H04L 12/14 Charging , metering or billing arrangements for data wireline or wireless communications
• CPC Class Codes: H04M 15/31 Distributed metering or calculation of charges
• CPC Class Codes: H04M 15/66 Policy and charging system
• CPC Class Codes: H04W 4/24 Accounting or billing

Abstract

A method and system for automatically metering a distributed file system node is provided. The method includes receiving data associated with jobs for execution via a distributed file system. Characteristics of the jobs are uploaded and policy metrics data associated with hardware usage metering is retrieved. Resource requests associated with hardware resource usage are retrieved and attributes associated with the resource requests are uploaded. The policy metrics data is analyzed and a recommendation circuit is queried with respect to the resource requests. A set of metrics of the policy metrics data associated with the resource requests is determined and a machine learning circuit is updated. Utilized hardware resources are determined with respect to the hardware usage metering and said resource requests.

Claims

1. A distributed file system node fine grained metering method comprising:

receiving from a user, by a processor of a hardware device, job data associated with jobs for execution via a distributed file system;

uploading, by said processor to a memory device of said hardware device, characteristics of said jobs;

retrieving, by a processor enabling a policy engine circuit of said hardware device, policy and cost metrics data associated with hardware usage metering, wherein said policy and cost metrics data comprises policies implemented as pluggable components defined in advance, by;

uploading xml files, via descriptor files or j son files, or via a command line, for said jobs;

retrieving, by said processor enabling a hardware device cluster, resource requests associated with hardware resource usage;

uploading, by said processor enabling a job descriptor engine of said hardware device, attributes associated with said resource requests;

analyzing, by said processor enabling said job descriptor engine of said hardware device, said policy and cost metrics data;

querying, by said processor enabling said job descriptor engine of said hardware device, a recommendation circuit with respect to said resource requests;

determining, by said processor enabling a machine learning circuit with respect to results of said querying, a set of metrics of said policy and cost metrics data associated with said resource requests;

updating, by said processor based on said set of metrics, said machine learning circuit; and

determining, by said processor enabling a metrics circuit with respect to said set of metrics, utilized hardware resources with respect to said hardware usage metering and said resource requests.

2. The method of claim 1, wherein said distributed file system comprises a multi tenanted distributed file system.

3. The method of claim 1, further comprising:

generating, by said processor executing a resource descriptor circuit, resource infrastructure associated with hardware resources of said resource requests, wherein said generating is executed based on said policy engine circuit providing various policies and cost metrics for calculating said hardware usage metering, and wherein said policy engine circuit comprises pluggable components.

4. The method of claim 3, wherein said job descriptor engine comprises circuitry for defining said characteristics of said jobs for execution on:

a plurality of nodes, hardware resources, processors, memory devices, and distributed system networks.

5. The method of claim 4, wherein said job descriptor engine receives inputs associated with said characteristics of said jobs during submission of said jobs submission via a descriptive syntax, wherein said characteristics of said jobs comprise a data volume, a minimum and maximum memory required, a limit associated with a data transfer rate, a speculative execution to be followed, or a type of scheduler to be used.

6. The method of claim 5, wherein said job descriptor engine executes said machine learning circuit for dynamically updating itself based on a history of job types, of said jobs, that have been executed in a specified sequence or have been submitted based on similar groups of users.

7. The method of claim 5, wherein said job descriptor engine dynamically updates said the characteristics of the jobs based on network delays, sudden change of data volumes, or hardware failure, wherein said job descriptor engine executes said machine learning circuit for estimating job characteristics and adapting to changing workload requirements, and wherein said job descriptor engine monitors said plurality of nodes, said hardware resources, said processors, a cluster health, and cluster optimum processing capability.

8. The method of claim 5, further comprising:

segmenting, by said processor, said jobs for allocation of resources and estimation of capacity requirements.

9. The method of claim 8, further comprising:

generating, by said processor, dynamic values for constants associated with a cost function.

10. The method of claim 3, further comprising:

monitoring, by said processor executing said job descriptor engine, a plurality of nodes, hardware resources, processors, memory devices, and distributed system networks; and

determining, by said processor, a configuration of said hardware device cluster based on an analysis of types of workloads and job characterizations.

11. The method of claim 10, further comprising:

determining, by said processor, workloads associated with said plurality of nodes, said hardware resources, said processors, said memory devices, and said distributed system networks; and

determining, by said processor based on said workloads, optimized recommendations associated with a group of said jobs.

12. The method of claim 11, further comprising:

determining, by said processor based on said optimized recommendations additional recommendation for improving said configuration of said hardware device cluster.

13. The method of claim 12, further comprising:

calculating, by said computer processor, a cost for improving said configuration of said hardware device cluster, wherein said

Cost=(Confidence Factor)*Σ

₀ⁿ((aΣ

_0%^100%Memory)+(bΣ

_0%^100%CPU Utilization)+(cΣ

_n=1^n=∞

Network Data Transfer)+(d∫

_t=0^t=nScheduler Time)+(eΣ

₀ⁿPreemption)+(Σ

₀ⁿSlots Used)),

wherein (a) comprises a weightage factor associated with memory, wherein said weightage factor associated with said memory comprises a variable per unit cost associated with said memory, wherein (b) comprises a weightage factor for a CPU, wherein said weightage factor for said CPU comprises a variable per unit cost associated with said CPU, wherein (c) comprises a weightage factor for network data transfer, wherein said weightage factor for said network data transfer comprises a variable per unit cost associated with said network data transfer, wherein (d) comprises a weightage factor associated with preemptive processes utilized, wherein said weightage factor associated with said preemptive processes comprise a variable per unit cost associated with said preemptive processes, wherein (e) comprises a weightage factor for a total number of preemptions, wherein said weightage factor for said total number of preemptions comprises a variable per unit cost associated with said total number of preemptions, and wherein said (a), (b), (c), (d), and (e) remain constant for a same type or category of said jobs if all the additional parameters of a distributed environment remain constant.

14. The method of claim 3, wherein said resource infrastructure enables resource infrastructure improvements comprising replacing hardware components of said hardware resources.

15. The method of claim 3, wherein said resource infrastructure enables resource infrastructure improvements comprising adding new hardware resources to said resource infrastructure comprising said hardware resources.

16. The method of claim 3, wherein said resource infrastructure enables resource infrastructure improvements comprising optimizing functionality of said hardware resources, and wherein said resource infrastructure improvements comprise dynamically recommending allocation of separate queue or fixed size allocation of said hardware resources to said jobs such that an overall resource utilization of a cluster enables additional jobs being executed on said cluster.

17. The method of claim 1, further comprising:

providing at least one support service for at least one of creating, integrating, hosting, maintaining, and deploying computer-readable code in the hardware device, said code being executed by the computer processor to implement;

said receiving, said uploading said characteristics, said retrieving said policy and cost metrics data, said retrieving said resource requests, said uploading said attributes, said analyzing, said querying, said determining said set of metrics, said updating, and said determining said utilized hardware resources.

18. A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, said computer readable program code comprising an algorithm that when executed by a processor of a hardware device implements a distributed file system node fine grained metering method, said method comprising:

receiving from a user, by said processor, job data associated with jobs for execution via a distributed file system;

uploading, by said processor to a memory device of said hardware device, characteristics of said jobs;

retrieving, by a processor enabling a policy engine circuit of a hardware device, policy and cost metrics data associated with hardware usage metering, wherein said policy and cost metrics data comprises policies implemented as pluggable components defined in advance, by;

uploading xml files, via descriptor files or j son files, or via a command line, for said jobs;

retrieving, by said processor enabling a hardware device cluster, resource requests associated with hardware resource usage;

uploading, by said processor enabling a job descriptor engine of said hardware device, attributes associated with said resource requests;

analyzing, by said processor enabling said job descriptor engine of said hardware device, said policy and cost metrics data;

querying, by said processor enabling said job descriptor engine of said hardware device, a recommendation circuit with respect to said resource requests;

determining, by said processor enabling a machine learning circuit with respect to results of said querying, a set of metrics of said policy and cost metrics data associated with said resource requests;

updating, by said processor based on said set of metrics, said machine learning circuit; and

determining, by said processor enabling a metrics circuit with respect to said set of metrics, utilized hardware resources with respect to said hardware usage metering and said resource requests.

19. A hardware device comprising a processor coupled to a computer-readable memory unit, said memory unit comprising instructions that when executed by the processor executes a distributed file system node fine grained metering method comprising:

receiving from a user, by said processor, job data associated with jobs for execution via a distributed file system;

uploading, by said processor to a memory device of said hardware device, characteristics of said jobs;

retrieving, by a processor enabling a policy engine circuit of a hardware device, policy and cost metrics data associated with hardware usage metering, wherein said policy and cost metrics data comprises policies implemented as pluggable components defined in advance, by;

uploading xml files, via descriptor files or j son files, or via a command line, for said jobs;

retrieving, by said processor enabling a hardware device cluster, resource requests associated with hardware resource usage;

uploading, by said processor enabling a job descriptor engine of said hardware device, attributes associated with said resource requests;

analyzing, by said processor enabling said job descriptor engine of said hardware device, said policy and cost metrics data;

querying, by said processor enabling said job descriptor engine of said hardware device, a recommendation circuit with respect to said resource requests;

determining, by said processor enabling a machine learning circuit with respect to results of said querying, a set of metrics of said policy and cost metrics data associated with said resource requests;

updating, by said processor based on said set of metrics, said machine learning circuit; and

determining, by said processor enabling a metrics circuit with respect to said set of metrics, utilized hardware resources with respect to said hardware usage metering and said resource requests.

20. A distributed file system resource utilization metering method comprising:

analyzing, by a processor enabling an analysis circuit, resource requests associated with resource usage of hardware devices in a multi-tenancy system;

monitoring, by said processor, said resource usage of said hardware devices with respect to functions executed by said hardware devices in response to said resource requests;

determining, by said processor, that said functions executed by said hardware devices have been completed;

determining, by said processor, billing amounts for said resource usage of said hardware devices;

analyzing, by said processor, said billing amounts; and

modifying components of said hardware devices based on said analyzing said billing amounts.

21. The method of claim 20, wherein said monitoring said resource usage of said hardware devices comprises monitoring an amount of memory space being used by said components of said hardware devices.

22. The method of claim 20, wherein said monitoring said resource usage of said hardware devices comprises monitoring a CPU utilization rate and percentage associated with execution of said components of said hardware devices.

23. The method of claim 20, wherein said monitoring said resource usage of said hardware devices comprises monitoring network device usage of said components of said hardware devices.

24. The method of claim 20, wherein said monitoring said resource usage of said hardware devices is performed via hardware sensors of said hardware devices.

25. A hardware device comprising a processor coupled to a computer-readable memory unit, said memory unit comprising instructions that when executed by the processor executes a distributed file system resource utilization metering method comprising:

analyzing, by said processor enabling an analysis circuit, resource requests associated with resource usage of hardware devices in a multi-tenancy system;

monitoring, by said processor, said resource usage of said hardware devices with respect to functions executed by said hardware devices in response to said resource requests;

determining, by said processor, that said functions executed by said hardware devices have been completed;

determining, by said processor, billing amounts for said resource usage of said hardware devices;

analyzing, by said processor, said billing amounts; and

modifying components of said hardware devices based on said analyzing said billing amounts.