Hierarchical management of the dynamic allocation of resources in a multi-node system

• US 7,516,221 B2
• Filed: 08/12/2004
• Issued: 04/07/2009
• Est. Priority Date: 08/14/2003
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Kantarjiev, Christopher A., Souder, Benny, Brower, David, Stamos, James W., Chatterjee, Debashish, Chidambaran, Lakshminarayanan, Colrain, Carol, Semler, Daniel
• Primary Examiner(s): Maung; Zarni
• Application Number:
  US10/917,873
• Publication Number:
  US 20050038834A1
• Time in Patent Office: 1,699 Days
• Field of Search: 709/203, 709226-229, 718/105
• US Class Current: 709/226
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/87 Monitoring of transactions
• CPC Class Codes: G06F 2209/501 Performance criteria
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer

Abstract

Approaches are used for efficiently and effectively managing the dynamic allocation of resources of multi-node database systems between services provided by the multi-node database server. A service is a category of work that is hosted on the database server. The approaches manage allocation of resources at different levels. For services that use a particular database, the performance realized by the services is monitored. Resources assigned to the database are allocated between these services to ensure performance goals for each are met. Resources assigned to a cluster of nodes are allocated between the databases to ensure that performance goals for all the services that use the databases are met. Resources assigned to a farm of clusters are assigned amongst clusters based on service level agreements and back-end policies. The approach uses a hierarchy of directors to manage resources at the different levels.

Claims

1. A method for dynamically allocating computer resources of a multi-node computer system, the method comprising computer implemented steps of:

monitoring performance realized by a plurality of services running on the multi-node computer system, wherein said plurality of services includes a first service and a second service;

based on said monitoring the performance of a plurality of services, generating performance metrics that indicate performance realized by each service of said plurality of services;

based on the performance metrics, said multi-node computer system detecting a violation of service-level agreements for said first service;

in response to detecting said violation of said service-level agreements, said multi-node computer system adjusting allocation of computer resources of said multi-node system between said first service and said second service;

said computer resources containing pools of resources;

the step of adjusting allocation of computer resources includes attempting to resolve said performance violation by adjusting, for said first service, allocation of a first pool of resources that is lower in a hierarchy before attempting to adjust allocation of a second pool of resources that is higher in said hierarchy.

2. The method of claim 1, wherein the step of adjusting allocation of resources includes allocating another node of said multi-node system to host said first service.

3. The method of claim 1, wherein a first server runs on a first node of said multi-node system, wherein the step of adjusting allocation of resources includes expanding said first service to said first server.

4. The method of claim 3, wherein adjusting allocation of resources includes provisioning said first server to said first node to perform the step of expanding said first service to said first server.

5. The method of claim 1, wherein:

said multi-node system includes a first node and a second node;

a first group of sessions are established for said first service on said first node;

a second group of sessions are established for said first service on said second node; and

the step of allocating resources includes migrating at least one session from said second group to said first node.

6. The method of claim 1, wherein:

the pools of resources include a first pool of servers as said first pool of resources and a second pool of nodes as said second pool of resources; and

the step of attempting to resolve said performance violation includes attempting to allocate another server from said first pool to host said first service before attempting to add another node from said second pool to host said first service.

7. The method of claim 6, wherein the steps further include, in response to adding said another node, provisioning another server to said another node.

8. The method of claim 1, wherein:

the first multi-node server includes a first server, a second server, and a third server;

the first server hosts a plurality of sessions for said first service; and

the step of attempting to resolve said performance violation includes attempting to migrate at least one of said plurality of sessions to said second server before allocating said third server to host said first service.

9. The method of claim 1, wherein:

each service of said plurality of services is a category of work performed by said multi-node system; and

the step of adjusting allocation of computer resources of said multi-node system includes adjusting allocation of computer resources to each category of said plurality of category of work.

10. The method of claim 9, wherein:

a plurality of units of software execute on a set of client computers interconnected to said multi-node system; and

each category of work corresponds to a particular unit of software from said plurality of units.

11. The method of claim 10, wherein each category of work corresponds to work performed in response to a request generated by a certain particular unit of software from said plurality of units.

21. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 1.

22. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 2.

23. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 3.

24. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 4.

25. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 5.

26. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 6.

27. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 7.

28. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 8.

29. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 9.

30. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 10.

31. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 11.

12. A method for dynamically allocating computer resources of a multi-node computer system that includes a first set of nodes and a second set of nodes, the method comprising the steps of:

monitoring performance of a plurality of services hosted on said multi-node system to generate performance metrics;

based on said monitoring the performance of a plurality of services, generating performance metrics that indicate performance realized by each service of said plurality of services;

wherein said multi-node system includes a first set of nodes and a second set of nodes;

running a first multi-node server and a second multi-node server on said first set of nodes;

wherein said plurality of services includes a first service and a second service hosted by said first multi-node server;

a first system component running on said first set of nodes adjusting an allocation of computer resources of the first multi-node server between said first service and said second service based on said performance metrics; and

a second system component running on said first set of nodes adjusting an allocation of computer resources of the first set of nodes between said first multi-node server and said second multi-node server based on said performance metrics.

said computer resources containing pools of resources;

the step of adjusting allocation of computer resources includes attempting to resolve said performance violation by adjusting, for said first service, allocation of a first pool of resources that is lower in a hierarchy before attempting to adjust allocation of a second pool of resources that is higher in said hierarchy.

13. The method of claim 12, wherein the steps further include:

the first system component detecting a service-level violation for said first service; and

in response to detecting the service-level violation for said first service, said first system component transmitting a first request to said second system component to adjust the allocation of resources of the first set of nodes between said first multi-node server and said second multi-node server.

14. The method of claim 12, wherein said second system component adjusting an allocation of resources of the first set of nodes between said first multi-node server and said second multi-node server includes said second system component allocating nodes between said first multi-node server and said second multi-node server.

15. The method of claim 12, the steps further include:

the first system component detecting a service-level violation for said first service; and

in response to detecting the service-level violation for said first service, the first system component adjusting allocation of resources of the first multi-node server between said first service and said second service.

16. The method of claim 15, wherein:

the first multi-node server includes a first server instance and a second server instance; and

wherein the step of adjusting allocation of resources of the first multi-node server includes causing said second server to host said first service.

17. The method of claim 15, wherein:

the first multi-node server includes a first server instance and a second server instance;

the first server instance hosts at least one session for said first service;

the second server instance hosts at least one session for said first service; and

the step of adjusting allocation of resources of the first multi-node server includes adjusting the balance of sessions for said first service between said first server instance and said second server instance.

18. The method of claim 13, wherein in response to receiving said first request, said second system component provisioning another server instance of the first multi-node server to another node.

19. The method of claim 18, wherein:

the first set of nodes is a first cluster of nodes that share access to at least one persistent storage device;

the second system component transmits a second request to a third system component running on the multi-node system to request another node; and

the third system component allocates nodes between the first cluster and another cluster within the multi-node system based on said performance metrics.

20. The method of claim 18, wherein provisioning another server instance causes said second system component to transmit a message to said first system component to indicate said another server instance is available.

32. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 12.

33. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 13.

34. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 14.

35. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 15.

36. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 16.

37. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 17.

38. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 18.

39. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 19.

40. A computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 20.