Maintaining throughput of a stream processing framework while increasing processing load

• US 9,965,330 B2
• Filed: 12/31/2015
• Issued: 05/08/2018
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Bishop, Elden Gregory, Chao, Jeffrey
• Primary Examiner(s): KIM, DONG U
• Application Number:
  US14/986,401
• Publication Number:
  US 20170083368A1
• Time in Patent Office: 859 Days
• Field of Search: None
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources

Abstract

The technology disclosed relates to maintaining throughput of a stream processing framework while increasing processing load. In particular, it relates to defining a container over at least one worker node that has a plurality workers, with one worker utilizing a whole core within a worker node, and queuing data from one or more incoming near real-time (NRT) data streams in multiple pipelines that run in the container and have connections to at least one common resource external to the container. It further relates to concurrently executing the pipelines at a number of workers as batches, and limiting simultaneous connections to the common resource to the number of workers by providing a shared connection to a set of batches running on a same worker regardless of the pipelines to which the batches in the set belong.

Claims

1. A method of maintaining throughput of a stream processing framework while increasing processing load, the method including:

defining a container over at least one worker node that has a plurality of workers, with one worker utilizing a whole processor core within a worker node;

queuing data from one or more incoming data streams into multiple pipelines that run in the container and have connections to at least one common resource external to the container;

concurrently executing the pipelines at a number of workers as batches; and

limiting simultaneous connections between the common resource and the workers by providing a shared connection used to process a set of batches running on a same worker regardless of the pipelines to which the batches in the set belong.

2. The method of claim 1, wherein processed batches from the pipelines are committed to a message bus, further including:

limiting simultaneous connections to the message bus to the number of workers concurrently executing the pipelines such that each worker commits to the message bus using a single connection shared between batches processed at each worker.

3. The method of claim 2, wherein the number of workers is 10, further including limiting simultaneous connections to the message bus to 10.

4. The method of claim 1, further including:

limiting concurrent execution of the pipelines to a number of workers and multiplexing the pipelines sequentially over the workers when a number of pipelines exceeds the number of the workers.

5. The method of claim 1, further including:

limiting simultaneous connections to the common resource to the number of workers by providing a shared connection to a set of batches running on a same worker regardless of computation required to process the set.

6. The method of claim 1, wherein the common resource maintains a state, further including:

providing a stateless shared connection to the common resource by externalizing information about the state.

7. The method of claim 6, wherein the information about the state is externalized at a key-value column store.

8. The method of claim 1, further including:

maintaining a stash of connections for each of the workers, wherein the stash includes a configuration component and an instance component;

receiving a request from a first worker executing a first pipeline to connect to at least one common resource external to the container, wherein the request includes an access key;

comparing the access key against a configuration component of a corresponding stash of the first worker;

responsive to a match, establishing a first connection between the first worker and the common resource using an instance component of the corresponding stash; and

limiting simultaneous connections to the common resource by sharing the first connection across a plurality of pipelines executed by the first worker.

9. The method of claim 8, wherein the configuration component includes a login and a password.

10. A system including one or more processors coupled to memory, the memory loaded with computer instructions to maintain throughput of a stream processing framework while increasing processing load, the instructions, when executed on the processors, implement actions comprising:

defining a container over at least one worker node that has a plurality of workers, with one worker utilizing a whole processor core within a worker node;

queuing data from one or more incoming data streams into multiple pipelines that run in the container and have connections to at least one common resource external to the container;

concurrently executing the pipelines at a number of workers as batches; and

limiting simultaneous connections between the common resource and the workers by providing a shared connection used to process a set of batches running on a same worker regardless of the pipelines to which the batches in the set belong.

11. The system of claim 10, wherein processed batches from the pipelines are committed to a message bus, further implementing actions comprising:

limiting simultaneous connections to the message bus to the number of workers concurrently executing the pipelines such that each worker commits to the message bus using a single connection shared between batches processed at each worker.

12. The system of claim 11, wherein the number of workers is 10, further implementing actions comprising limiting simultaneous connections to the message bus to 10.

13. The system of claim 10, further implementing actions comprising:

limiting concurrent execution of the pipelines to a number of workers and multiplexing the pipelines sequentially over the workers when a number of pipelines exceeds the number of the workers.

14. The system of claim 10, further implementing actions comprising:

limiting simultaneous connections to the common resource to the number of workers by providing a shared connection to a set of batches running on a same worker regardless of computation required to process the set.

15. The system of claim 10, further implementing actions comprising:

maintaining a stash of connections for each of the workers, wherein the stash includes a configuration component and an instance component;

receiving a request from a first worker executing a first pipeline to connect to at least one common resource external to the container, wherein the request includes an access key;

comparing the access key against a configuration component of a corresponding stash of the first worker;

responsive to a match, establishing a first connection between the first worker and the common resource using an instance component of the corresponding stash; and

limiting simultaneous connections to the common resource by sharing the first connection across a plurality of pipelines executed by the first worker.

16. The system of claim 15, wherein the configuration component includes a login and a password.

17. A non-transitory computer readable storage medium impressed with computer program instructions to maintain throughput of a stream processing framework while increasing processing load, the instructions, when executed on a processor, implement a method comprising:

defining a container over at least one worker node that has a plurality of workers, with one worker utilizing a whole processor core within a worker node;

queuing data from one or more incoming data streams into multiple pipelines that run in the container and have connections to at least one common resource external to the container;

concurrently executing the pipelines at a number of workers as batches; and

limiting simultaneous connections between the common resource and the workers by providing a shared connection used to process a set of batches running on a same worker regardless of the pipelines to which the batches in the set belong.

18. The non-transitory computer readable storage medium of claim 17, wherein processed batches from the pipelines are committed to a message bus, implementing the method further comprising:

limiting simultaneous connections to the message bus to the number of workers concurrently executing the pipelines such that each worker commits to the message bus using a single connection shared between batches processed at each worker.

19. The non-transitory computer readable storage medium of claim 18, wherein the number of workers is 10, implementing the method further comprising limiting simultaneous connections to the message bus to 10.

20. The non-transitory computer readable storage medium of claim 17, implementing the method further comprising:

maintaining a stash of connections for each of the workers, wherein the stash includes a configuration component and an instance component;

receiving a request from a first worker executing a first pipeline to connect to at least one common resource external to the container, wherein the request includes an access key;

comparing the access key against a configuration component of a corresponding stash of the first worker;

responsive to a match, establishing a first connection between the first worker and the common resource using an instance component of the corresponding stash; and

limiting simultaneous connections to the common resource by sharing the first connection across a plurality of pipelines executed by the first worker.