Managing Hardware Resources by Sending Messages Amongst Servers in a Data Center

• US 20110179134A1
• Filed: 01/29/2010
• Published: 07/21/2011
• Est. Priority Date: 01/15/2010
• Status: Active Grant

• Current Assignee: Joyent, Inc. (Samsung Electronics Co. Ltd.)
• Original Assignee: Joyent, Inc. (Samsung Electronics Co. Ltd.)
• Inventors: Hoffman, Jason A., Young, David P., Mayo, Mark G., Candel, Pedro Palazón, Duncan, James
• Granted Patent: US 8,346,935 B2
• Time in Patent Office: Days
• US Class Current: 709/214
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

Systems and methods to manage workloads and hardware resources in a data center or cloud. In one embodiment, a method includes a data center having a plurality of servers in a network. The data center provides a virtual machine for each of a plurality of users, each virtual machine to use a portion of hardware resources of the data center. The hardware resources include storage and processing resources distributed onto each of the plurality of servers. The method further includes sending messages amongst the servers, some of the messages being sent from a server including status information regarding a hardware resource utilization status of that server. The method further includes detecting a request from the virtual machine to handle a workload requiring increased use of the hardware resources, and provisioning the servers to temporarily allocate additional resources to the virtual machine, wherein the provisioning is based on status information provided by one or more of the messages.

Claims

1. A method for managing resources in a data center, the data center comprising a plurality of servers in a network, the method comprising:

providing, via the data center, a virtual machine for each of a plurality of users, each respective virtual machine to use a portion of hardware resources of the data center, and wherein the hardware resources include storage and processing resources distributed onto each of the plurality of servers;

sending a plurality of messages amongst the plurality of servers, at least a portion of the messages being sent from any respective server comprising status information regarding a hardware resource utilization status of that respective server;

detecting a request from a first of the respective virtual machines to handle a workload requiring increased use of the hardware resources; and

provisioning the plurality of servers to temporarily allocate additional resources from the hardware resources to the first respective virtual machine, the provisioning based at least in part on status information provided by one or more of the messages.

2. The method of claim 1, further comprising:

providing an application program interface (API) to each of the plurality of users for operation of its respective virtual machine in the data center;

wherein each respective user is able to customize the provisioning to allocate the additional resources for its respective virtual machine.

3. The method of claim 1, further comprising:

associating an economic value to the storage and processing resources;

wherein provisioning to allocate the additional resources is further based at least in part on the economic value associated with the storage or processing resources that are available for use by the first respective virtual machine.

4. The method of claim 1, wherein at least a portion of the messages further comprise presence information for a server or commands to partition hardware resources on a server.

5. The method of claim 1, wherein the messages are sent using at least one of an Extensible Messaging and Presence Protocol (XMPP) or an Advance Message Queuing Protocol (AMQP).

6. The method of claim 1, wherein:

each respective server of the plurality of servers comprises a plurality of logical solid state drives (SSDs), each respective solid state drive being individually addressable by a virtual machine supported by the respective server; and

provisioning to allocate the additional resources comprises allocating at least one of the logical SSDs to the first respective virtual machine.

7. The method of claim 1, wherein each of the plurality of messages is associated with a topic, each message being routed in the network based on its respective topic, the method further comprising sorting the messages into one or more queues based on the respective topic of each message.

8. The method of claim 7, further comprising executing a plurality of listener processes in the network, each of the plurality of listener processes being associated with a respective one of the queues, wherein a first listener process of the plurality of listener processes listens for presence information, the presence information to include an indication that a new server has joined the network.

9. The method of claim 1, further comprising sending a new message from a new server that has joined the network, the new message comprising presence information regarding one or more agents executing on the new server.

10. The method of claim 9, further comprising initiating a first new agent on the new server, the initiating comprising:

discovering, by the first new agent, services in the network available to the first new agent;

self-configuring, by the first new agent, in response to the services discovered;

causing a second new agent to log in to a first messaging system to advertise presence of the new server; and

causing a third new agent to log in to a second messaging system, the third new agent to manage a listener queue on the new server, to publish messages to the second messaging system, and to listen for new messages in the listener queue.

11. The method of claim 10, further comprising launching, by the second new agent after logging in to the first messaging system, a command processor executing on the new server.

12. The method of claim 10, wherein discovering the services is performed using domain name system-based service discovery (DNS-SD).

13. The method of claim 10, wherein each of the plurality of servers is configured to send and receive messages using the first messaging system and the second messaging system.

14. The method of claim 10, wherein:

the first messaging system implements an Extensible Messaging and Presence Protocol (XMPP); and

the second messaging system implements an Advance Message Queuing Protocol (AMQP).

15. The method of claim 1, further comprising executing an events agent, a metrics agent, and a provisioning agent on each of the plurality of servers, each respective events agent, metrics agent, and provisioning agent to provide a portion of the plurality of messages, wherein the respective metrics agent provides a portion of the status information, and the respective provisioning agent implements a temporary allocation of additional resources from its respective server to a virtual machine of a user.

16. A method for managing resources in a data center, the data center comprising a plurality of servers in a network, the plurality of servers comprising hardware resources for use by the data center, each respective server running an operating system, the method comprising:

monitoring, via an agent executing on each of the plurality of servers, a plurality of workloads being handled by each respective server, the agent using system calls to the operating system of the respective server;

sending a plurality of messages amongst all of the plurality of servers, at least a portion of the messages being sent from any respective server comprising status information regarding a hardware resource utilization status of that respective server as determined at least in part based on information provided in reply to the system calls made to the operating system of that respective server;

identifying, using one or more of the messages, a first workload requiring increased use of the hardware resources; and

provisioning the plurality of servers to temporarily allocate additional resources from the hardware resources to the first workload.

17. The method of claim 16, wherein each of the plurality of messages is associated with a topic, each message being routed in the network based on the respective topic.

18. The method of claim 17, further comprising sorting the messages into one or more queues based on the respective topic of each message, each of the one or more queues corresponding to a respective predefined topic.

19. The method of claim 18, wherein:

each of the one or more queues is stored on a different one of the plurality of servers; and

the predefined topics for the queues are stored as a predefined topic hierarchy on two or more of the plurality of servers.

20. The method of claim 18, further comprising associating, via the respective agent executing on the respective server sending the respective message, the respective predefined topic with the respective message.

21. The method of claim 18, wherein the plurality of messages are routed in the network using a messaging system, the messaging system comprising a plurality of topic routers to route messages into one of the queues based on a respective topic of the message.

22. The method of claim 21, wherein the messaging system is operated according to an Advance Message Queuing Protocol (AMQP).

23. The method of claim 19, further comprising:

executing a dedicated agent on each of the plurality of servers, the agent to listen for a dedicated one or more topics from the topic hierarchy; and

sending a stream of messages from the dedicated agent to update the network regarding the operation of the respective server from which the stream of messages is being sent.

24. The method of claim 23, wherein each of the stream of messages comprises information for the respective server regarding its processor usage, its system memory usage, and a data input/output rate utilization of a storage device on the respective server.

25. The method of claim 23, wherein each of the stream of messages is sent periodically.

26. The method of claim 18, further comprising executing a plurality of listener processes, each of the plurality of listener processes being associated with a respective one of the queues.

27. The method of claim 26, wherein each of the plurality of listener processes is associated with respective one of a plurality of decision-making engines, each decision-making engine configured to generate a new message of a message type selected from a set of predefined message types based on an event reported to the decision-making engine.

28. The method of claim 27, wherein the decision-making engine generates the new message further based on a user economic model and on server resource data, the server resource data comprising information previously provided in one or more of the plurality of messages.

29. The method of claim 26, wherein a first listener process of the plurality of listener processes listens for presence information, the presence information to include an indication of initiation of a new agent when a new server joins the network.

30. The method of claim 26 wherein a first listener process of the plurality of listener processes listens for an event reported in one of the plurality of messages sent from one of the plurality of servers, the method further comprising:

in response to the first listener process detecting the event, generating a new message using decision-making logic, wherein the new message includes a provisioning command directed to a first server of the plurality of servers; and

sending the new message to the first server to implement the provisioning command.

31. The method of claim 30, wherein the provisioning command includes specifications to change two or more of system memory space usage, processor space usage, and storage device space usage of the first server.

32. The method of claim 30, wherein generating the new message comprises selecting a type of message to generate from a predefined set of message types.

33. The method of claim 32, wherein the set of message types includes the following types:

Extensible Messaging and Presence Protocol (XMPP), Simple Mail Transfer Protocol (SMTP), and Simple Network Management Protocol (SNMP).

34. The method of claim 16, wherein each of the plurality of workloads corresponds to one of a plurality of users of the data center, the method further comprising:

storing user data for each respective user, the user data comprising data regarding behavior of prior workloads handled by the data center for the respective user;

wherein provisioning the plurality of servers to temporarily allocate the additional resources comprises correlating the system calls for a respective server to user data for a respective user having a workload being handled by the respective server.

35. The method of claim 34, wherein the prior workload behavior comprises usage of network bandwidth within the data center or a data input/output rate utilization of a storage device of the respective server.

36. The method of claim 16, wherein monitoring the plurality of workloads comprises monitoring for each respective workload its processor usage, system memory usage, and storage device usage within the network.

37. The method of claim 36, wherein a user may specify processor usage, system memory usage, and storage device usage for a virtual machine of the user using an application programming interface (API) coupled to the network.

38. The method of claim 36, wherein monitoring the plurality of workloads further comprises monitoring for each respective workload its system memory input/output, network input/output, and storage device input/output within the network.

39. A machine readable media embodying instructions, the instructions causing a computing device to perform a method for managing resources in a data center, the data center comprising a plurality of servers in a network, the method comprising:

providing, via the data center, a virtual machine for each of a plurality of users, each respective virtual machine to use a portion of hardware resources of the data center, and wherein the hardware resources include storage and processing resources distributed onto each of the plurality of servers;

sending messages amongst all of the plurality of servers, at least a portion of the messages being sent from any respective server comprising status information regarding a hardware resource utilization status of that respective server;

detecting a request from a first of the respective virtual machines to handle a workload requiring increased use of the hardware resources; and

provisioning the plurality of servers to temporarily allocate additional resources from the hardware resources to the first respective virtual machine, the provisioning based at least in part on status information provided by one or more of the messages.