Resource over-subscription

• US 9,473,418 B2
• Filed: 12/12/2013
• Issued: 10/18/2016
• Est. Priority Date: 12/12/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: DeCusatis, Casimer, Kashyap, Sujatha, Krishnamurthy, Rajaram B.
• Primary Examiner(s): Swearingen, Jeffrey R
• Assistant Examiner(s): Barker, Todd L
• Application Number:
  US14/104,738
• Publication Number:
  US 20150172383A1
• Time in Patent Office: 1,041 Days
• Field of Search: 709/201, 709/202, 709/223, 709/224, 709/225, 709/226, 709/238, 709/239, 709/240, 709/241, 709/245, 370/229, 370/235, 370/252, 370/389, 370/400
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: H04L 43/026 using flow identification
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0805 by checking availability
• CPC Class Codes: H04L 43/0823 Errors, e.g. transmission errors
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 47/11 Identifying congestion
• CPC Class Codes: H04L 47/30 in combination with information about buffer occupancy at either end or at transit nodes
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation

Abstract

Embodiments include a method, system, and computer program product for managing workloads in a network. A switch receives data associated with a workload. The received data is tagged with an identifier that associates the data with the workload. The received data is compressed based on determining that second data stored in a buffer of the switch exceeds a threshold. The switch stores the compressed data in the buffer. The compressed data is transmitted to a second network based on a determination that the switch is over-subscribed.

Claims

1. A computer system embodied in hardware for managing workloads in a network, wherein the network comprises a data network and an independent management network, the system comprising:

at least one switch included in the data network, the at least one switch comprising at least one buffer;

wherein said at least one switch comprises at least one of a Hadoop network switch and a software-defined network (SDN) enabled switch;

wherein said at least one switch performs a method comprising;

receiving data associated with a workload from a server, wherein the workload is part of a shuffle phase of a map-reduce algorithm;

tagging said data with an unique identifier that associates the data with the workload;

determining that the switch is not over-subscribed when second data stored in that at least one buffer of the switch does not exceed a threshold, and determining that the switch is over-subscribed when the second data stored in the at least one buffer of the switch exceeds the threshold;

wherein the threshold is selected based on at least one of;

a throughput requirement associated with said network comprises said data network and said independent management network, an anticipated maximum rate of incoming data at said at least one switch, and a capacity of said at least one buffer in said at least one switch;

compressing the received data based on determining that the second data stored in the at least one buffer exceeds the threshold when the switch is over-subscribed;

storing the compressed data in the at least one buffer; and

transmitting the compressed data and the unique identifier to a controller of said independent management network in response to determining that the at least one switch in the first network is over-subscribed such that spare bandwidth available in said independent management network is leveraged and utilized to transfer the compressed data while the at least one switch is over-subscribed so as to manage the workload between said data network and said independent management network;

receiving, from said controller, said unique identifier and sub-results associated with a processing of said compressed data by said controller;

merging said sub-results with second sub-results identified by said unique identifier to generate overall results;

decompressing said overall results; and

transmitting said overall results to a final destination.