Managing resource allocation in a stream processing framework

• US 10,146,592 B2
• Filed: 01/12/2016
• Issued: 12/04/2018
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Bishop, Elden Gregory, Chao, Jeffrey
• Primary Examiner(s): Lee, Tammy E
• Application Number:
  US14/994,131
• Publication Number:
  US 20170083380A1
• Time in Patent Office: 1,057 Days
• Field of Search: None
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

The technology disclosed relates to managing resource allocation to task sequences in a stream processing framework. In particular, it relates to operating a computing grid that includes machine resources, with heterogeneous containers defined over whole machines and some containers including multiple machines. It also includes initially allocating multiple machines to a first container, initially allocating first set of stateful task sequences to the first container, running the first set of stateful task sequences as multiplexed units of work under control of a container-scheduler, where each unit of work for a first task sequence runs to completion on first machine resources in the first container, unless it overruns a time-out, before a next unit of work for a second task sequence runs multiplexed on the first machine resources. It further includes automatically modifying a number of machine resources and/or a number assigned task sequences to a container.

Claims

1. A method, comprising:

operating, by at least one processor, a computing grid that includes machine resources;

identifying, by the at least one processor, a container configured to include at least one machine configured to use an allocated amount of the machine resources;

initially allocating, by the at least one processor, a set of machines to the container;

initially allocating, by the at least one processor, a set of machine resources to the container, for use by the set of machines;

initially allocating, by the at least one processor, a task sequence to the container;

running, by the at least one processor, the task sequence,

wherein the task sequence is configured to be run as multiplexed units of work in the container under control of a container-scheduler, and

wherein a multiplexed unit of work of the multiplexed units of work is configured to be run to completion in the container, within a predetermined time, using the set of machine resources;

performing, by the at least one processor, data stream analysis, of a continuous data stream for processing by the task sequence;

determining, by the at least one processor, based on the data stream analysis, that a data volume for the task sequence is expected to decrease after a surge in the data volume; and

responsive to the determining, automatically changing, by the at least one processor, the container'"'"'s allocation of task sequences, machines, or machine resources, based on the data stream analysis.

2. The method of claim 1, further including:

detecting, by the at least one processor, that excess machine resources are dedicated to the task sequence;

scheduling, by the container-scheduler, the task sequence to use a first subset of machines in the container and to leave a second subset of machines in the container unused; and

automatically deallocating, by the at least one processor, the second subset of machines from the container.

3. The method of claim 1, further including:

detecting, by the at least one processor, that excess machine resources are dedicated to the task sequence allocated to the container; and

automatically allocating, by the at least one processor, an additional task sequence to the container.

4. The method of claim 1, further including a non-transitory computer readable storage medium impressed with computer program instructions that, when executed on a processor, implement the method of claim 1.

5. The method of claim 1, wherein the data stream analysis comprises matrix calculations on directed acyclic graphs.

6. A method, comprising:

operating by at least one processor, a computing grid that includes machine resources;

identifying, by the at least one processor, a container configured to include at least one machine configured to use an allocated amount of the machine resources;

initially allocating, by the at least one processor, a set of machines to the container;

initially allocating, by the at least one processor, a set of machine resources to the container, for use by the set of machines;

initially allocating, by the at least one processor, a task sequence to the container;

running, by the at least one processor, the task sequence,

wherein the task sequence is configured to be run as multiplexed units of work in the container under control of a container-scheduler, and

wherein a multiplexed unit of work of the multiplexed units of work is configured to be run to completion in the container, within a predetermined time, using the set of machine resources;

performing, by the at least one processor, data stream analysis, of a continuous data stream for processing by the task sequence;

determining, by the at least one processor, based on the data stream analysis, that a data volume for the task sequence is expected to surge and that the set of machines allocated to the container have not yet reached a predetermined maximum; and

responsive to the determining, automatically changing, by the at least one processor, the container'"'"'s allocation of task sequences, machines, or machine resources, based on the data stream analysis.

7. The method of claim 6, further including:

detecting, by the at least one processor, latency exceeding a predetermined threshold in running the task sequence in the container;

automatically allocating, by the at least one processor, an additional machine to the container; and

scheduling, by the at least one processor, the multiplexed units of work for the task sequence over the set of machines and the additional machine allocated to the container.

8. The method of claim 6, further including:

detecting, by the at least one processor, latency exceeding a predetermined threshold while running the task sequence from the container; and

determining, by the at least one processor, that the set of machines allocated to the container have reached a predetermined maximum and that no more machines will be allocated to the task sequence as a result of the predetermined maximum being reached.

9. The method of claim 6, further including:

allocating, by the at least one processor, a set of task sequences to the container, wherein the set of task sequences includes the task sequence as a first task sequence and at least one additional task sequence;

persisting, by the at least one processor, state information of the set of task sequences, during running of the set of task sequences; and

detecting, by the at least one processor, latency of at least the first task sequence exceeding a predetermined threshold while the set of task sequences is running and allocated to the container;

automatically reallocating, by the at least one processor, a first subset of task sequences from the container to an additional container, while leaving a second subset of task sequences allocated to the container;

initializing the first subset of task sequences in the additional container based on the persisted state information; and

scheduling, by the at least one processor, the multiplexed units of work for the second subset of task sequences in the container and the first subset of task sequences in the additional container.

10. The method of claim 6, further including a non-transitory computer readable storage medium impressed with computer program instructions that, when executed on a processor, implement the method of claim 6.

11. The method of claim 6, wherein the data stream analysis comprises matrix calculations on directed acyclic graphs.

12. A method, comprising:

identifying, by at least one processor, a container configured to include at least one machine configured to use an allocated amount of machine resources;

initially allocating, by the at least one processor, a set of machines to the container;

initially allocating, by the at least one processor, a set of machine resources to the container, for use by the set of machines;

initially allocating by the at least one processor, a set of task sequences to the container;

receiving, by the at least one processor, input from a replayable input source;

triggering, by the at least one processor, the set of task sequences to process the input;

running, by the at least one processor, the set of task sequences,

wherein the set of task sequences is configured to be run as multiplexed units of work in the container under control of a container-scheduler,

wherein a multiplexed unit of work of the multiplexed units of work is configured to be run to completion in the container, within a predetermined time, using the set of machine resources;

during the running, persisting, by the at least one processor, state information of the set of task sequences;

detecting, by the at least one processor, runtime of a unit of work in a faulty task sequence, of the set of task sequences, exceeding a predetermined timeout threshold;

restarting, by the at least one processor, the faulty task sequence by automatically reloading persisted state information of the faulty task sequence;

automatically rewinding, by the at least one processor, the input from the replayable input source to the faulty task sequence to a point preceding the detecting and synchronized with the persisted state information for the faulty task sequence; and

rerunning, by the at least one processor, the faulty task sequence to completion of the unit of work without exceeding the predetermined timeout threshold.

13. The method of claim 12, further including:

reallocating, by the at least one processor, the faulty task sequence from the container to an additional container;

restarting, by the at least one processor, the faulty task sequence at the additional container by reloading the persisted state information of the faulty task sequence at the additional container;

rewinding, by the at least one processor, the input from the replayable input source to the faulty task sequence at the additional container to a point preceding the detecting and synchronized with the persisted state information for the faulty task sequence; and

rerunning, by the at least one processor, the faulty task sequence to completion of the unit of work without exceeding the predetermined timeout threshold.

14. The method of claim 12, further including a non-transitory computer readable storage medium impressed with computer program instructions that, when executed on a processor, implement the method of claim 12.

15. A system including at least one processor coupled to memory, the memory loaded with computer instructions that, when executed by the at least one processor, cause the at least one processor to perform actions comprising:

operating a computing grid that includes machine resources;

identifying a container configured to include at least one machine configured to use an allocated amount of the machine resources;

initially allocating a set of machines to the container;

initially allocating a set of machine resources to the container, for use by the set of machines;

initially allocating a task sequence to the container;

running the task sequence,

wherein the task sequence is configured to be run as multiplexed units of work in the container under control of a container-scheduler,

wherein a multiplexed unit of work of the multiplexed units of work is configured to be run to completion in the container, within a predetermined time, using the set of machine resources;

performing data stream analysis, of a continuous data stream for processing by the task sequence;

determining based on the data stream analysis, that a data volume for the task sequence is expected to decrease after a surge in the data volume; and

responsive to the determining, automatically changing the container'"'"'s allocation of task sequences, machines, or machine resources, based on the data stream analysis.

16. The system of claim 15, the actions further comprising:

detecting that excess machine resources are dedicated to the task sequence;

scheduling the task sequence over a first subset of machines in the container to leave a second subset of machines in the container unused; and

automatically deallocating the second subset of machines from the container.

17. The system of claim 15, the actions further comprising:

detecting that excess machine resources are dedicated to the task sequence, and automatically allocating additional stateful task sequences to the container.

18. The system of claim 15, the actions further comprising:

detecting, by the at least one processor, based on the data stream analysis, that a data volume for the task sequence is expected to surge and that the set of machines allocated to the container have not yet reached a predetermined maximum; and

responsive to the detecting, automatically changing, by the at least one processor, the container'"'"'s allocation of task sequences, machines, or machine resources, based on the data stream analysis.

19. The system of claim 15, the actions further comprising:

allocating a set of task sequences to the container;

receiving input from a replayable input source;

triggering the set of task sequences to process the input;

running the set of task sequences,

wherein the set of task sequences is configured to be run as multiplexed units of work in the container under control of a container-scheduler,

wherein a multiplexed unit of work of the multiplexed units of work is configured to be run to completion in the container, within a predetermined time, using the set of machine resources;

during the running, persisting state information of the set of task sequences;

detecting runtime of a unit of work in a faulty task sequence, of the set of task sequences, exceeding a predetermined timeout threshold;

restarting the faulty task sequence by automatically reloading persisted state information of the faulty task sequence;

automatically rewinding the input from the replayable input source to the faulty task sequence to a point preceding the detecting and synchronized with the persisted state information for the faulty task sequence; and

rerunning the faulty task sequence to completion of the unit of work without exceeding the predetermined timeout threshold.