Generating a Distributed Stream Processing Application

• US 20120259910A1
• Filed: 06/13/2012
• Published: 10/11/2012
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Andrade, Henrique, Gedik, Bugra, Wu, Kun-Lung
• Granted Patent: US 8,407,360 B2
• Time in Patent Office: Days
• US Class Current: 709/201
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

Techniques for generating a distributed stream processing application are provided. The techniques include obtaining a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task, generating one or more containers that encompass a combination of one or more stream processing operators, and generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary.

Claims

1. A method for generating a distributed stream processing application, comprising the steps of:

obtaining a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task;

generating one or more containers that encompass a combination of one or more stream processing operators; and

generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary.

2. The method of claim 1, wherein the declarative description expresses at least one stream processing task using one or more fine granularity operators.

3. The method of claim 1, further comprising using a stream-centric and operator based paradigm for declaring one or more stream processing applications.

4. The method of claim 1, further comprising using one or more type-generic data stream processing operators that are declaratively specified and for which code is automatically generated by a compiler.

5. The method of claim 1, further comprising using one or more user-defined operators to wrap legacy code.

6. The method of claim 1, further comprising using one or more procedural constructs to control a coalescing level of one or more operators that are employed by a stream processing application.

7. The method of claim 1, further comprising the steps of:

generating one or more coarse granularity containers that encompass one or more fine granularity stream processing operators; and

using the one or more coarse granularity containers to generate a distributed stream processing application.

8. The method of claim 1, wherein generating the one or more containers comprises the steps of:

coalescing a combination of one or more operators into one or more containers; and

fusing an outflow of an operator into an inflow of a downstream operator within a same container.

9. The method of claim 8, wherein coalescing the combination of one or more operators into one or more containers comprises the steps of:

using an optimizer to automatically decide which of the one or more operators are to be coalesced into which of the one or more containers; and

using user input to manually group the one or more operators into the one or more containers.

10. The method of claim 8, wherein fusing an outflow of an operator into an inflow of a downstream operator within the same container comprises the steps of:

using one or more function calls to propagate one or more stream data objects from the output of an upstream operator to the input of a downstream operator that is in the same container; and

using one or more queues and one or more execution threads to propagate one or more stream data objects from the output of an upstream operator to the input of a downstream operator that is in the same container.

11. The method of claim 1, wherein generating one or more execution units from the declarative description of one or more data stream processing tasks comprises using a compiler.

12. A computer program product comprising a tangible computer readable recordable storage medium having computer readable program code for generating a distributed stream processing application, said computer program product including:

computer readable program code for obtaining a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task;

computer readable program code for generating one or more containers that encompass a combination of one or more stream processing operators; and

computer readable program code for generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary.

13. The computer program product of claim 12, wherein the declarative description expresses at least one stream processing task using one or more fine granularity operators.

14. The computer program product of claim 12, further comprising computer readable program code for using a stream-centric and operator based paradigm for declaring one or more stream processing applications.

15. The computer program product of claim 12, further comprising computer readable program code for using one or more type-generic data stream processing operators that are declaratively specified and for which code is automatically generated by a compiler.

16. The computer program product of claim 12, further comprising computer readable program code for using one or more user-defined operators to wrap legacy code.

17. The computer program product of claim 12, further comprising computer readable program code for using one or more procedural constructs to control a coalescing level of one or more operators that are employed by a stream processing application.

18. The computer program product of claim 12, wherein the computer readable program code for generating one or more execution units from the declarative description of one or more data stream processing tasks comprises computer readable program code for using a compiler.

19. An apparatus for generating a distributed stream processing application, comprising:

a memory; and

at least one processor coupled to said memory and operative to;

obtain a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task;

generate one or more containers that encompass a combination of one or more stream processing operators; and

generate one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary.

20. The apparatus of claim 19, wherein the declarative description expresses at least one stream processing task using one or more fine granularity operators.

21. The apparatus of claim 19, wherein the at least one processor coupled to said memory is further operative to use a stream-centric and operator based paradigm for declaring one or more stream processing applications.

22. The apparatus of claim 19, wherein the at least one processor coupled to said memory is further operative to use one or more type-generic data stream processing operators that are declaratively specified and for which code is automatically generated by a compiler.

23. The apparatus of claim 19, wherein the at least one processor coupled to said memory is further operative to use one or more user-defined operators to wrap legacy code.

24. The apparatus of claim 19, wherein the at least one processor coupled to said memory is further operative to use one or more procedural constructs to control a coalescing level of one or more operators that are employed by a stream processing application.

25. The apparatus of claim 19, wherein the at least one processor coupled to said memory and operative to generate one or more execution units from the declarative description of one or more data stream processing tasks is further operative to use a compiler to generate one or more execution units.