Grid computing accounting and statistics management system

• US 8,713,179 B2
• Filed: 10/04/2005
• Issued: 04/29/2014
• Est. Priority Date: 10/04/2005
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dawson, Christopher J., Hamilton, Rick A. II, Joseph, Joshy, Seaman, James W.
• Primary Examiner(s): Winder, Patrice
• Assistant Examiner(s): Chang, Julian
• Application Number:
  US11/243,658
• Publication Number:
  US 20070078960A1
• Time in Patent Office: 3,129 Days
• Field of Search: 709/205, 709223-229, 718/102, 718/104, 718/105
• US Class Current: 709/226
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: H04L 41/147 for predicting network behaviour
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 41/5032 Generating service level reports
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Performance data is captured periodically from resources and groups of resources in a grid computing environment and stored in a content-addressable data repository from which it can be accessed in response to an arbitrarily complex query in regard to specifics of particular jobs or job portions, particular resources utilized, grid architecture, application environment, concurrent jobs or job portions and the like. The data repository may be distributed or divided in regard to grid environment architecture, security domains or the like and each portion or division may be implemented in a modular fashion including an accounting and statistics management module and additional modules or computing engines for performing particular desired analyses or functions. Results of such analyses or functions may be communicated to a grid workload agent (and associated modules) to improve grid management on a fine-grained basis.

Claims

1. A method of managing a grid computing environment comprising a plurality of resources, said method including steps of:

determining currently executing and scheduled job portions having particular characteristics for said plurality of resources;

performing job portions of a data processing request using respective heterogeneous and distributed resources of said grid computing environment, said resources including a plurality of platforms having different implementations, semantic behaviors and application programming interfaces and which may exhibit different performance for a given portion of a given job;

periodically capturing and storing as data records, in correspondence with a plurality of characteristics of a respective job portion, a snapshot of current instantaneous operations of individual application environments in said grid computing environment from said respective resources performing said job portions of said data processing request in content-addressable storage, said data records corresponding to performance of respective ones of said respective resources while processing respective ones of said job portions, said performance being monitored in correspondence with said characteristics of respective ones of said job portions wherein said plurality of characteristics include nature of processing being performed during said instantaneous operation, said platform on which said processing is performed, said resource or resources on which said processing is performed and concurrent processing in said grid environment, such that any performance data or other parameter of said data records for respective ones of said resources may be retrieved based on any other performance data or parameter of said data records or logical combination thereof with arbitrarily fine granularity to closely match characteristics of job portions of any other job to assess impact of said other job on said grid environment, determine allocation of said resources to said portions of said other job or predict performance of said other job as performed on available resources;

storing actual performance and predicted performance of said other job as performed by ones of said resources allocated to said other job;

selecting data records in accordance with one or more of said plurality of characteristics stored in said data records as selected data records;

retrieving said selected data records; and

processing data retrieved in said retrieving step to produce processed performance data as historical performance statistics corresponding to said characteristics of said job portions and said resources.

2. A method as recited in claim 1 including a further steps of managing job portion allocation based on said processed performance data.

3. A method as recited in claim 1 including a further step of estimating job run time on the grid based on historical performance analysis.

4. A method as recited in claim 1 including a further step of processing a request for proposal based on historical performance analysis.

5. A method as recited in claim 3 including a further step of determining a likelihood that a job run time can be performed within an associated historical performance statistic.

6. A method as recited in claim 1 wherein said step of periodically storing data records includes storing data records in a data repository.

7. A method as recited in claim 6 including the further step of dividing said data repository into a plurality of distributed data repositories.

8. A method as recited in claim 7 wherein said step of dividing said data repository is based on how said grid computing environment is dispersed over multiple security domains.

9. A method as recited in claim 1, wherein said step of performing portions of a data processing request include a step of determining if portions of said data processing request are application specific.

10. A method as recited in claim 9, including the further step of interrogating the grid or interrogating an application environment of the grid based on a result of said step of determining if portions of said data processing request are application specific.

11. A system for managing a grid computing environment comprising:

a plurality of heterogenous and distributed data processing resources, wherein respective ones of said data processing resources include a plurality of platforms having different implementations, semantic behaviors and application programming interfaces and which may exhibit different performance for a given portion of a given job;

a grid management system including a grid workload agent to monitor respective ones of said data processing resources of said grid computing environment processing job portions of data processing requests, wherein said job portions have particular characteristics;

content-addressable storage means for periodically storing data records forming a snapshot of current instantaneous operations of individual application environments in said grid computing environment which include, in correspondence with a plurality of characteristics of a respective job portion, performance data from respective resources monitored by said grid workload agent, wherein said plurality of characteristics include nature of processing being performed, said platform on which said processing is performed during said instantaneous operation, said resource or resources on which said processing is performed and concurrent processing in said grid environment, said performance data including data identifying a corresponding monitored respective resource while that resource is processing a respective one of said job portions of data processing requests in correspondence with said characteristic of a respective one of said job portions such that any performance data or other parameter of said data records for respective ones of said resources may be retrieved based on any other performance data or parameter of said data records or logical combination thereof with arbitrarily fine granularity to closely match characteristics of job portions of any other job to assess impact of said other job on said grid environment, determine allocation of said data processing resources to said portions of said other job or predict performance of said other job as performed on available ones of said data processing resources;

a memory for storing actual performance and predicted performance of said other job as performed by ones of said data processing resources allocated to said other job;

a content-addressable memory for retrieving said actual performance data on the basis of a said characteristic or combination of said characteristics, and processing data stored by said content-addressable memory to form processed performance data; and

at least one module cooperating with said grid workload agent to manage said grid computing environment based on said processed performance data.

12. A system as recited in claim 11, wherein said means for managing said grid computing environment includes a plurality of modules for performing respective grid environment management functions including at least one of cost estimation, determination of hardware/software requirements, RFP processing, determination of intragrid processes, analysis of grid element financial data and historical trend analysis.

13. A system as recited in claim 11, wherein said means for managing said grid computing environment includes a module for performing a grid environment management function.

14. A system as recited in claim 11 wherein said means for retrieving and processing data includes an accounting and statistics management module.

15. A system as recited in claim 14 further including a grid workload agent wherein said accounting and statistics management module provides said processed performance data to said grid workload agent.

16. A system as recited in claim 15 further including functional modules associated with said grid workload agent to perform functions associated with said grid workload agent and wherein said accounting and statistics management module provides said processed performance data to said functional modules.