Dynamic streaming data dispatcher

• US 9,367,501 B2
• Filed: 11/10/2011
• Issued: 06/14/2016
• Est. Priority Date: 11/10/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Idziorek, Joseph R., Kumar, Vibhore, Venkatramani, Chitra, Wagle, Rohit
• Primary Examiner(s): Recek, Jason
• Application Number:
  US13/293,822
• Publication Number:
  US 20130124668A1
• Time in Patent Office: 1,678 Days
• Field of Search: 709/214, 709/231
• US Class Current: 1/1
• CPC Class Codes: G06F 13/385 for adaptation of a particular data processing system to different peripheral devices
• CPC Class Codes: H04L 47/125 by balancing the load, e.g. traffic engineering

Abstract

A method includes receiving, by a computing device, a plurality of data streams from plurality of sources, distributing the data streams to a plurality of sinks on multiple hosts, receiving load information indicating a load on at least one of the plurality of sinks and adjusting the distribution of the data stream accordingly and instructing the plurality of sinks to write the data streams to a distributed data store.

Claims

1. A method, comprising:

receiving, by a computing device, a data stream from a source to be stored on a distributed data store;

distributing the data stream to a plurality of sinks on multiple hosts by sending a plurality of data tuples to the plurality of sinks, wherein each of the sinks on the multiple hosts are configured to buffer the data stream and wherein the distributed data store is configured to receive data from each of the multiple hosts, and wherein distributing the data stream to the plurality of sinks on multiple hosts comprises sending a data tuple from the plurality of data tuples to two or more of the plurality of sinks;

receiving load information indicating a load on at least one of the plurality of sinks and responsively adjusting the distribution of the data stream;

sending, by the computing device, to a first sink from the plurality of sinks, a control tuple, the control tuple comprising one of a write request or a discard request, the write request instructing the first sink to write data stored on the first sink to the distributed data store based on a buffer size and a load on the first sink, and the discard request instructing the first sink to discard the data stored on the first sink;

receiving a result tuple from the first sink indicating a success or failure of writing the data stream to the distributed data store; and

sending the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data store by the first sink.

2. The method of claim 1, wherein a data rate of the data stream is variable.

3. The method of claim 1, wherein distributing the data stream to a plurality of sinks on multiple hosts is performed in a balanced manner.

4. The method of claim 1, wherein writing the data stream to the distributed store comprises simultaneously writing a plurality of buffered data tuples on the plurality of sinks to the distributed data store.

5. The method of claim 1, wherein the data stream is distributed to the plurality of sinks based on a total size of data being sent to each sink.

6. A method, comprising:

receiving, by a computing device, a plurality of data streams from a plurality of data sources to be stored on a distributed data store;

distributing the plurality data stream to a plurality of sinks on multiple hosts by sending a plurality of data tuples to the plurality of sinks, wherein each of the sinks on the multiple hosts are configured to buffer the data stream and wherein the distributed data store is configured to receive data from each of the multiple hosts, wherein distributing the data stream to the plurality of sinks on multiple hosts comprises sending a data tuple from the plurality of data tuples to two or more of the plurality of sinks;

receiving load information indicating a load on at least one of the plurality of sinks and responsively adjusting the distribution of the data stream;

sending, by the computing device, to a first sink from the plurality of sinks, a control tuple, the control tuple comprising one of a write request or a discard request, the write request instructing the first sink to write data stored on the first sink to the distributed data store based on a buffer size and a load on the first sink, and the discard request instructing the first sink to discard the data stored on the first sink;

receiving a result tuple from the first sink indicating a success or failure of writing the data stream to the distributed data store in response to the control tuple; and

sending the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data store by the first sink.

7. The method of claim 6, where a data rate of at least one of the plurality of data streams is variable.

8. The method of claim 6, wherein writing the plurality of data streams to the distributed store comprises simultaneously writing a plurality of buffered data tuples on the plurality of sinks to the distributed data store.

9. A computer program product for distributing incoming data streams, the computer program product comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to;

receive, by a computing device a plurality of data streams from a plurality of data sources to be stored on distributed data store;

distribute the plurality data stream to a plurality of sinks on multiple hosts by sending a plurality of data tuples to the plurality of sinks, wherein each of the sinks on the multiple hosts are configured to buffer the data stream and wherein the distributed data store is configured to receive data from each of the multiple hosts, wherein distributing the data stream to the plurality of sinks on multiple hosts comprises sending a data tuple from the plurality of data tuples to two or more of the plurality of sinks;

receive load information indicating a load on at least one of the plurality of sinks and responsively adjust the distribution of the data stream;

send, by the computing device, to a first sink from the plurality of sinks, a control tuple, the control tuple comprising one of a write request or a discard request, the write request indicative for the first sink to write data stored on the first sink to the distributed data store based on a buffer size and a load on the first sink, and the discard request indicative for the first sink to discard the data stored on the first sink;

receive a result tuple from the first sink indicating a success or failure of writing the data stream to the distributed data store in response to the control tuple; and

send the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data store by the first sink.

10. The computer program product of claim 9, wherein distributing the plurality of data streams to a plurality of sinks on multiple hosts is performed in a balanced manner.

11. A system, comprising:

a splitter for receiving a data stream from a data source;

a plurality of sinks in operable communication with the splitter;

a distributed data storage device in operable communication with the plurality of sinks; and

a load manager in operable communication with the plurality of sinks and the splitter, wherein the load manager instructs the splitter on distribution of the data stream to the plurality of sinks and instructs the plurality of sinks to write data stored on the plurality of sinks to the distributed data store based on a buffer size and a load on the plurality of the sinks; and

wherein the splitter is configured to;

split the data stream into a plurality portions;

send a first data tuple from comprising a portion of the data stream to a first sink and a second sink, the first sink and the second sink being from the plurality of sinks;

send a control tuple to the first sink, the control tuple comprising a write request for the first sink to write the portion of the data stream in the data tuple to the distributed data store;

receive a result tuple from the first sink indicating a success or failure of writing the portion of the data stream to the distributed data store; and

send the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data store by the first sink.

12. The system of claim 11, further comprising a second splitter for receiving a second data stream from a second data source, wherein the second splitter is in operable communication with the plurality of sinks and the load manager instructs the splitter and the second splitter on distribution of the data streams to the plurality of sinks.

13. The system of claim 11, wherein the data stream has a variable rate.

14. The system of claim 11, wherein the control tuple comprises a discard request for the first sink to discard the portion of the data stream from the data tuple that the first sink stores in a buffer.