SYSTEM AND METHOD FOR DYNAMIC ACTIVATION OF REAL-TIME STREAMING DATA OVERFLOW PATHS

• US 20180324069A1
• Filed: 05/08/2017
• Published: 11/08/2018
• Est. Priority Date: 05/08/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kleinheksel, Cory J., Koster, David M., Nikolai, Jason A., Sabotta, Christopher R.
• Granted Patent: US 10,834,177 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

A method, computer program product, and computer system for defining, on at least one processor, one or more potential overflow paths for processing at least a portion of a data load through a plurality of stream operators deployed on a plurality of computing devices. Embodiments may include determining, on the at least one processor, a conditional availability for each computing device of the plurality of computing devices. Embodiments may also include identifying, on the at least one processor, a change in one or more of a configuration of the plurality of computing devices and the at least a portion of the data load. Embodiments may further include dynamically activating, on the at least one processor, one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

Claims

1. A computer-implemented method comprising:

defining, on at least one processor, one or more potential overflow paths for processing at least a portion of a data load through a plurality of stream operators deployed on a plurality of computing devices;

determining, on the at least one processor, a conditional availability for each computing device of the plurality of computing devices;

identifying, on the at least one processor, a change in one or more of a configuration of the plurality of computing devices and the at least a portion of the data load; and

dynamically activating, on the at least one processor, one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

2. The computer-implemented method of claim 1, wherein dynamically activating one or more potential overflow paths includes:

streaming a first portion of the data load through the plurality of stream operators deployed on the plurality of computing devices via an original path; and

streaming a second portion of the data load through the plurality of stream operators deployed on the plurality of computing devices via a first overflow path of the one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices, wherein the data load is divided between the original path and the first overflow path.

3. The computer-implemented method of claim 1, further comprising:

sampling one or more system state variables of each computing device of the plurality of computing devices during run time of one or more streaming applications, wherein the one or more sampled system state variables include one or more of CPU capability, memory capability, disk utilization, I/O capability, and virtual memory capability.

4. The computer-implemented method of claim 3, further comprising:

generating one or more best fit models for application patterns based upon, at least in part, the sampled system state variables of each computing device, wherein the one or more best fit models are configured for one or more of duplicating stream operators, creating and removing dynamic connections, and optimizing graph flows.

5. The computer-implemented method of claim 1, wherein identifying a change in one or more of the configuration of the plurality of computing devices and the at least a portion of the data load includes predicting a change in one or more of the configuration of the plurality of computing devices and the at least a portion of the data load based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

6. The computer-implemented method of claim 5, wherein dynamically activating one or more potential overflow paths in response to the identified change includes dynamically activating one or more potential overflow paths in response to the predicted change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

7. The computer-implemented method of claim 1, further comprising:

creating one or more pools of computing devices having one or more common characteristics; and

dynamically interchanging one or more computing devices included in at least one of the potential overflow paths with one or more computing devices from the same pool of the one or more pools of computing devices in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

8. A computer program product comprising a non-transitory computer readable storage medium having a plurality of instructions stored thereon, which, when executed by a processor, cause the processor to perform operations including:

defining one or more potential overflow paths for processing at least a portion of a data load through a plurality of stream operators deployed on a plurality of computing devices;

determining a conditional availability for each computing device of the plurality of computing devices;

identifying a change in one or more of a configuration of the plurality of computing devices and the at least a portion of the data load; and

dynamically activating one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

9. The computer program product of claim 8, further comprising instructions for:

streaming a first portion of the data load through the plurality of stream operators deployed on the plurality of computing devices via an original path; and

streaming a second portion of the data load through the plurality of stream operators deployed on the plurality of computing devices via a first overflow path of the one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices, wherein the data load is divided between the original path and the first overflow path.

10. The computer program product of claim 8, further comprising instructions for:

sampling one or more system state variables of each computing device of the plurality of computing devices during run time of one or more streaming applications, wherein the one or more sampled system state variables include one or more of CPU capability, memory capability, disk utilization, I/O capability, and virtual memory capability.

11. The computer program product of claim 10, further comprising instructions for:

generating one or more best fit models for application patterns based upon, at least in part, the sampled system state variables of each computing device, wherein the one or more best fit models are configured for one or more of duplicating stream operators, creating and removing dynamic connections, and optimizing graph flows.

12. The computer program product of claim 8, wherein identifying a change in one or more of the configuration of the plurality of computing devices and the at least a portion of the data load includes predicting a change in one or more of the configuration of the plurality of computing devices and the at least a portion of the data load based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

13. The computer program product of claim 12, wherein dynamically activating one or more potential overflow paths in response to the identified change includes dynamically activating one or more potential overflow paths in response to the predicted change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

14. The computer program product of claim 13, further comprising instructions for:

creating one or more pools of computing devices having one or more common characteristics; and

dynamically interchanging one or more computing devices included in at least one of the potential overflow paths with one or more computing devices from the same pool of the one or more pools of computing devices in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

15. A computing system comprising:

a processor and a memory module coupled with the processor, the processor being configured for;

defining one or more potential overflow paths for processing at least a portion of a data load through a plurality of stream operators deployed on a plurality of computing devices;

determining a conditional availability for each computing device of the plurality of computing devices;

identifying a change in one or more of a configuration of the plurality of computing devices and the at least a portion of the data load; and

dynamically activating one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

16. The computing system of claim 15, wherein the processor is further configured for:

streaming a first portion of the data load through the plurality of stream operators deployed on the plurality of computing devices via an original path; and

streaming a second portion of the data load through the plurality of stream operators deployed on the plurality of computing devices via a first overflow path of the one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices, wherein the data load is divided between the original path and the first overflow path.

17. The computing system of claim 15, wherein the processor is further configured for:

sampling one or more system state variables of each computing device of the plurality of computing devices during run time of one or more streaming applications, wherein the one or more sampled system state variables include one or more of processing capability, memory capability, disk utilization, I/O capability, and virtual memory capability.

18. The computing system of claim 17, wherein the processor is further configured for:

generating one or more best fit models for application patterns based upon, at least in part, the sampled system state variables of each computing device, wherein the one or more best fit models are configured for one or more of duplicating stream operators, creating and removing dynamic connections, and optimizing graph flows.

19. The computing system of claim 15, wherein identifying a change in one or more of the configuration of the plurality of computing devices and the at least a portion of the data load includes predicting a change in one or more of the configuration of the plurality of computing devices and the at least a portion of the data load based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

20. The computing system of claim 19, wherein dynamically activating one or more potential overflow paths in response to the identified change includes dynamically activating one or more potential overflow paths in response to the predicted change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.