Resource over-subscription

• US 9,509,627 B2
• Filed: 09/30/2014
• Issued: 11/29/2016
• Est. Priority Date: 12/12/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: DeCusatis, Casimer, Kashyap, Sujatha, Krishnamurthy, Rajaram B.
• Primary Examiner(s): Swearingen, Jeffrey R
• Assistant Examiner(s): Barker, Todd L
• Application Number:
  US14/501,305
• Publication Number:
  US 20150172209A1
• Time in Patent Office: 791 Days
• Field of Search: 709/224, 709/226, 370/229, 370/235
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0805 by checking availability
• CPC Class Codes: H04L 43/0823 Errors, e.g. transmission errors
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 47/11 Identifying congestion
• CPC Class Codes: H04L 47/30 in combination with information about buffer occupancy at either end or at transit nodes
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation

Abstract

Embodiments include a method, system, and computer program product for managing workloads in a network. A switch receives data associated with a workload. The received data is tagged with an identifier that associates the data with the workload. The received data is compressed based on determining that second data stored in a buffer of the switch exceeds a threshold. The switch stores the compressed data in the buffer. The compressed data is transmitted to a second network based on a determination that the switch is over-subscribed.

Claims

1. A computer implemented method performed by computer system embodied in hardware for managing workloads in a network, wherein the network comprises a data network that implements at least one of a Hadoop network switch or a software-defined network (SDN) enabled switch and an independent management network, the switch performing a method comprising:

receiving, by the switch including a buffer therein, data associated with a workload from a server, wherein the workload is part of a shuffle phase of a map-reduced algorithm;

tagging said data with a unique identifier that associates the data with the workload;

determining that the switch is not over-subscribed when second data stored in the buffer of the switch does not exceed a threshold, and determining that the switch is over-subscribed when the second data stored in the buffer of the switch exceeds the threshold, wherein the threshold is selected based on at least one of;

a throughput requirement associated with said network comprises said data network and said independent management network, an anticipated maximum rate of incoming data at said at least one switch, and a capacity of said at least one buffer in said at least one switch;

compressing the received data based on determining that the second data stored in the buffer of the switch exceeds the threshold when the switch is over-subscribed;

storing, by the switch, the compressed data in the buffer; and

transmitting the compressed data and the unique identifier to a controller of said independent management network in response to determining that the switch in the data network is over-subscribed such that spare bandwidth available in said independent management network is leveraged and utilized to transfer the compressed data while the at least one switch is over-subscribed so as to manage the workload between said data network and said independent management network;

receiving, from said controller, said unique identifier and sub-results associated with a processing of said compressed data by said controller, and merging said sub-results with second sub-results identified by said unique identifier to generate overall results;

decompressing said overall results; and

transmitting said overall results to a final destination.

2. The method of claim 1, wherein the management network monitors the network for errors and monitors the network'"'"'s performance.

3. The method of claim 1, further comprising:

processing the compressed data to generate sub-results.

4. The method of claim 3, further comprising:

merging the sub-results with second sub-results identified by the identifier.

5. The method of claim 1, wherein the switch is installed in a HADOOP network that includes a Hadoop Distributed File System (HDFS) serving as a storage portion and a Map Reduce serving as a processing portion.

6. The method of claim 1, wherein the switch is a software-defined network (SDN) enabled switch.

7. A computer program product executable by a computer system embodied in hardware for managing workloads in a network, the computer program product comprising:

a tangible non-transitory computer storage medium readable by a hardware processing circuit and storing instructions for execution by the processing circuit,

wherein the network comprises a data network and an independent management network, the data network including a switch including a buffer, the switch comprising at least one at least one of a Hadoop network switch or a software-defined network (SDN) enabled switch that performs a method comprising;

receiving data associated with a workload from a server, wherein the workload is part of a shuffle phase of a map-reduce algorithm;

tagging said data with a unique identifier that associates the data with the workload;

determining that the switch is not over-subscribed when second data stored in a buffer of the switch does not exceed a threshold, and determining that the switch is over-subscribed when the second data stored in the buffer of the switch exceeds the threshold, the threshold selected based on at least one of;

a throughput requirement associated with said network comprises said data network and said independent management network, an anticipated maximum rate of incoming data at said at least one switch, and a capacity of said at least one buffer in said at least one switch;

compressing the received data based on determining that the second data stored in the buffer exceeds the threshold when the switch is over-subscribed;

storing the compressed data in the buffer; and

transmitting the compressed data and the unique identified to said independent management network in response to determining that the switch in the data network is over-subscribed such that available bandwidth available in said independent management network is leveraged and utilized to transfer the compressed data while the at least one switch is over-subscribed so as to manage the workload between said data network and said independent management network;

receiving, from said controller, said unique identifier and sub-results associated with a processing of said compressed data by said controller, and merging said sub-results with second sub-results identified by said unique identifier to generate overall results;

decompressing said overall results; and

transmitting said overall results to a final destination.

8. The computer program product of claim 7, wherein the management network monitors the network for errors and monitors the network'"'"'s performance.

9. The computer program product of claim 7, wherein the method performed by the processing circuit further comprises:

processing the compressed data to generate sub-results.

10. The computer program product of claim 9, wherein the method performed by the processing circuit further comprises:

merging the sub-results with second sub-results identified by the identifier.

11. The computer program product of claim 7, wherein the processor is associated with a switch installed in a HADOOP network that includes a Hadoop Distributed File System (HDFS) serving as a storage portion and a Map Reduce serving as a processing portion.

12. The computer program product of claim 7, wherein the processor is associated with a software-defined network (SDN) enabled switch.