AUTONOMOUS DISTRIBUTED WORKLOAD AND INFRASTRUCTURE SCHEDULING

• US 20170264493A1
• Filed: 05/31/2017
• Published: 09/14/2017
• Est. Priority Date: 03/09/2015
• Status: Active Grant

• Current Assignee: Vapor Corp.
• Original Assignee: Vapor Corp.
• Inventors: Cencini, Andrew, Crawford, Cole Malone, Daniszewski, Erick
• Granted Patent: US 10,833,940 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 1/189 Power distribution
• CPC Class Codes: G06F 1/206 comprising thermal management
• CPC Class Codes: G06F 1/26 Power supply means, e.g. regulation thereof for memories G11C
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 41/044 comprising hierarchical management structures
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/0894 Policy-based network configuration management
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1012 based on compliance of requirements or conditions with available server resources
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Provided is a process of autonomous distributed workload and infrastructure scheduling based on physical telemetry data of a plurality of different data centers executing a plurality of different workload distributed applications on behalf of a plurality of different tenants.

Claims

1. A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising:

allocating, with one or more processors, resources in, or added to, a compute-cluster with a compute-cluster manager, the resources including usage of a plurality of computing nodes in, or added to, the compute-cluster to execute one or more distributed workload applications, the distributed workload applications being configured to be executed on a plurality of different computing nodes of the compute-cluster;

obtaining, with one or more processors, physical telemetry data of each of a plurality of at least some of the computing nodes, the physical telemetry data indicating attributes of a physical environment in which the respective computing node executes and being distinct from logical telemetry data indicative of logical attributes of computing nodes accessible via a respective operating system within which at least some of the computing nodes execute;

accessing, with one or more processors, a policy that indicates how to allocate compute-cluster resources based on physical telemetry data, logical telemetry data, and workload; and

allocating, with one or more processors, additional resources of, or added to, the compute-cluster to at least one of the distributed workload applications with the compute-cluster manager based on both the policy and the physical telemetry data, wherein the computer-cluster manager is configured to allocate resources to cause workloads to be scheduled based on amounts of computing resources needed to execute workloads, logical telemetry data of computing nodes, and physical telemetry data of computing nodes in accordance with one or more policies.

2. The medium of claim 1, wherein:

the compute-cluster manager is configured to allocate resources a multi-tenant cluster distributed across a plurality of different data centers in different geographic locations based on a plurality of different policies;

the compute-cluster manager is a distributed application executing on a plurality of computing devices and implementing a fault-tolerant protocol to select a master computing node of the compute-cluster manager;

each computing node has a different network/network address/port from other computing nodes in the compute-cluster;

the compute-cluster manager is configured to allocate resources of a heterogeneous mix of computing devices to a heterogenous mix of distributed workload applications executed concurrently;

the resources include durations of time reserved by the on computing nodes to run tasks;

the computing nodes include instances of micro-kernels, containers, virtual machines, or computing devices of the plurality of different data centers; and

the logical attributes include central processing unit utilization, memory utilization, and storage utilization.

3. The medium of claim 1, wherein the physical telemetry data includes at least one of the following:

current temperature of a computing component;

current temperature of ambient air in a data center;

current temperature of ambient air outside of a data center;

current humidity inside or outside of a data center;

predicted temperature of a computing component;

predicted temperature of ambient air in a data center;

predicted temperature of ambient air outside of a data center;

predicted humidity inside or outside of a data center;

geolocation of a data center;

location of a rack in a data center;

location of a computing device on the rack in the data center;

current thermal load of a computing device;

current thermal load of a rack;

current thermal load of a data center;

predicted thermal load of a computing device;

predicted thermal load of a rack;

or

predicted thermal load of a data center.

4. The medium of claim 1, wherein the physical telemetry data includes at least four of the following:

current temperature of a computing component;

current temperature of ambient air in a data center;

current temperature of ambient air outside of a data center;

current humidity inside or outside of a data center;

predicted temperature of a computing component;

predicted temperature of ambient air in a data center;

predicted temperature of ambient air outside of a data center;

predicted humidity inside or outside of a data center;

geolocation of a data center;

location of a rack in a data center;

location of a computing device on the rack in the data center;

current thermal load of a computing device;

current thermal load of a rack;

current thermal load of a data center;

predicted thermal load of a computing device;

predicted thermal load of a rack;

or

predicted thermal load of a data center.

5. The medium of claim 1, wherein the physical telemetry data includes each of the following:

current temperature of a computing component;

current temperature of ambient air in a data center;

current temperature of ambient air outside of a data center;

current humidity inside or outside of a data center;

predicted temperature of a computing component;

predicted temperature of ambient air in a data center;

predicted temperature of ambient air outside of a data center;

predicted humidity inside or outside of a data center;

geolocation of a data center;

location of a rack in a data center;

location of a computing device on the rack in the data center;

current thermal load of a computing device;

current thermal load of a rack;

current thermal load of a data center;

predicted thermal load of a computing device;

predicted thermal load of a rack; and

predicted thermal load of a data center.

6. The medium of claim 1, the operations comprising:

obtaining cost data including at least one of the following;

a current cost of electricity;

a predicted cost of electricity;

a current cost of bandwidth;

a predicted cost of bandwidth;

a current cost of computing resources;

or

a predicted cost of computing resources,

obtaining performance data including at least one of the following;

current network latency between two locations;

predicted network latency between two locations;

current processing performance;

or

predicted processing performance;

wherein;

the policy indicates how to balance between cost, performance, and physical telemetry based on the physical telemetry data, the cost data, and the performance data; and

allocating additional resources is based on the policy, the physical telemetry data, the cost data, and the performance data.

7. The medium of claim 1, wherein:

the policy is one of a plurality of policies each associated with different tenant accounts in a data repository;

the policy comprises a plurality of rules specifying resource allocation actions and criteria upon which determinations to perform the actions are based.

8. The medium of claim 1, wherein:

the policy comprises weights by which performance metrics and physical telemetry data are combined in a plurality of weighted scores, each weighted score being associated with a different candidate resource allocation action;

allocating additional resources comprises selecting a resource allocation action from the candidate resource allocation actions based on the scores.

9. The medium of claim 1, wherein:

the policy is operative to exclude computing resources from a pool of available resources used by a scheduler based on the physical telemetry data.

10. The medium of claim 1, wherein:

the policy indicates how to select among a plurality of data centers based on the physical telemetry data to provision additional computing nodes for the compute-cluster;

the operations comprise;

selecting a data center among the plurality of data centers based on the policy and the physical telemetry data; and

causing additional computing nodes to be provisioned in the selected data center.

11. The medium of claim 1, wherein:

the policy comprises;

a plurality of thermal models of a plurality of data centers, each including a model of heat-rejection of the data center and a model of heat-generation within the data center based on workload;

a plurality of cost models of the plurality of data centers, each cost model being indicative of aggregate cost of operating at least part of the respective data center, including cost associated with heat rejection and cost arising from powering execution of a workload;

allocating additional resources comprises;

inputting the physical telemetry data into at least some of the thermal models;

and allocating additional resources based on computational performance per watt per unit of currency determined based on the thermal models and the cost models.

12. The medium of claim 1, wherein resources are allocated based on type of workload.

13. The medium of claim 1, wherein the operations comprise operations to control a plurality of rack-mounted computing devices to provision resources or obtain the physical telemetry data, comprising:

receiving, with a rack-controller, via a first network, an application program interface (API) request, wherein;

the rack-controller is configured to control a plurality of rack-mounted computing devices mounted in a plurality of different rack units of one or more racks;

the rack-controller is configured to control the rack-mounted computing devices via a second network, different from the first network;

the rack-controller includes a gateway between the first network and the second network;

the second network is an out-of-band network distinct from an in-band network with which data is conveyed between rack-mounted computing devices or between rack-mounted computing devices and the internet; and

the API request is encoded in a first protocol;

based on the API request, selecting, with the rack-controller, one of a plurality of routines to effectuate control via the second network of at least some of the plurality of rack-mounted computing devices, the plurality of routines including;

a first routine that reads a sensor via the second network on one of the rack-mounted computing devices;

a second routine that reads a sensor via the second network on the rack but not on one of the rack-mounted computing devices;

a third routine that scans computing devices on the second network and produces an inventory of the scanned computing devices on the second network;

a fourth routine by which a configuration of an extensible firmware interface (EFI) of a given one of the rack-mounted computing device is adjusted; and

executing, with the rack-controller, the selected routine and, as a result, sending one or more commands via the second network encoded in a second protocol different from the first protocol to effectuate an action indicated by the API request.

14. The medium of claim 13, wherein:

the second network connects a control plane of a rack over which a plurality of servers mounted in the rack are controlled; and

the first network is an out-of-band network over which a data center is managed based on workload and environmental data gathered from a plurality of racks via a plurality of respective instances of the first network.

15. The medium of claim 13, wherein:

the operations comprise;

gathering, with the rack-controller, via the second network, agentless monitoring metrics from each of the rack-mounted computing devices, the metrics including processor utilization of the respective rack-mounted computing devices;

at least some of the agentless monitoring metrics are not reported by an operating system or a process running within the operating system of the rack-mounted computing devices, and wherein the metrics include memory utilization and an indication of processor temperature; and

at least two of the routines are not effectuated via a baseboard management controller of the rack-mounted computing devices.

16. The medium of claim 1, wherein:

the physical telemetry data comprises a physical rack computing device location of a computing device in a rack; and

the operations comprises obtaining the physical location by operations comprising;

receiving a request for a rack computing device location, the rack computing device being mounted in the rack;

wirelessly sensing a value indicative of the location based on physical proximity between a transmitter and a receiver, the proximity being caused by the rack computing device being mounted in one of a plurality of receptacles in the rack, each of the receptacles being associated with a different value indicative of a different location; and

generating, based on the wirelessly sensed value, output signals conveying location information related to the location of the rack-mounted computing device.

17. The medium of claim 16, wherein:

the rack holds a plurality of computing devices including the rack computing device, each computing device being in spaced relation with a different respective rack-mounted wireless transmitter configured to emit a wireless signal encoding a different identifier;

the rack comprises a direct current (DC) power bus configured to power operation of the rack computing device;

the rack comprises a rack controller having a processor, a memory, and a powerline modem operative to send commands to, and receive data from the rack computing device via the DC power bus; and

the rack computing device comprises a power adapter connecting the rack computing device to the DC power bus, the power adapter comprising;

a powerline modem;

a microcontroller operative to execute commands sent by the rack controller, the commands including requests for location of the rack computing device; and

a tag reader configured to obtain location information of the rack computing device from one of the wireless transmitters and provide the location information to the microcontroller, wherein a range of the wireless transmitters is less than 20 centimeters

18. The medium of claim 1, wherein:

allocating additional resources comprises steps for allocating resources based on physical telemetry data.

19. The medium of claim 1, the operations comprising:

executing the workload application.

20. A method, comprising:

allocating, with one or more processors, resources in, or added to, a compute-cluster with a compute-cluster manager, the resources including usage of a plurality of computing nodes in, or added to, the compute-cluster to execute one or more distributed workload applications, the workload applications being configured to be executed on a plurality of different computing nodes of the compute-cluster;

obtaining, with one or more processors, physical telemetry data of each of the plurality of computing nodes, the physical telemetry data indicating attributes of a physical environment in which the respective computing node executes and being distinct from logical telemetry data indicative of logical attributes of computing nodes accessible via a respective operating system within which at least some of the computing nodes execute;

accessing, with one or more processors, a policy that indicates how to allocate compute-cluster resources based on physical telemetry data, logical telemetry data, and workload; and

allocating, with one or more processors, additional resources of, or added to, the compute-cluster to at least one of the distributed workload applications with the compute-cluster manager based on both the policy and the physical telemetry data, wherein the computer-cluster manager is configured to allocate resources to cause workloads to be scheduled based on amounts of computing resources needed to execute workloads, logical telemetry data of computing nodes, and physical telemetry data of computing nodes in accordance with one or more policies.