Method for generating a distributed stream processing application

• US 8,291,006 B2
• Filed: 05/30/2008
• Issued: 10/16/2012
• Est. Priority Date: 05/30/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Andrade, Henrique, Gedik, Bugra, Wu, Kun-Lung
• Primary Examiner(s): JEAN, FRANTZ B
• Application Number:
  US12/130,043
• Publication Number:
  US 20090300615A1
• Time in Patent Office: 1,600 Days
• Field of Search: 709/201, 709/231, 718/100, 717/117, 717/100, 717/114
• US Class Current: 709/201
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

Techniques for generating a distributed stream processing application are provided. The techniques include obtaining a declarative description of one or more data stream processing tasks, wherein the declarative description expresses at least one stream processing task, and generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application.

Claims

1. A method for generating a distributed stream processing application, comprising the steps of:

obtaining a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task;

generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary;

generating one or more coarse granularity containers that encompass one or more fine granularity stream processing operators;

using the one or more coarse granularity containers to generate a distributed stream processing application;

generating one or more containers that encompass a combination of one or more stream processing operators, wherein said generating comprises;

coalescing a combination of one or more operators into one or more containers, wherein said coalescing comprises;

using an optimizer to automatically decide which of the one or more operators are to be coalesced into which of the one or more containers; and

using user input to manually group the one or more operators into the one or more containers; and

fusing an outflow of an operator into an inflow of a downstream operator within a same container.

2. The method of claim 1, wherein the declarative description expresses at least one stream processing task using one or more fine granularity operators.

3. The method of claim 1, further comprising using a stream-centric and operator based paradigm for declaring one or more stream processing applications.

4. The method of claim 1, further comprising using one or more type-generic data stream processing operators that are declaratively specified and for which code is automatically generated by a compiler.

5. The method of claim 1, further comprising using one or more user-defined operators to wrap legacy code.

6. The method of claim 1, further comprising using one or more procedural constructs to control a coalescing level of one or more operators that are employed by a stream processing application.

7. The method of claim 1, wherein fusing an outflow of an operator into an inflow of a downstream operator within the same container comprises the steps of:

using one or more function calls to propagate one or more stream data objects from the output of an upstream operator to the input of a downstream operator that is in the same container; and

using one or more queues and one or more execution threads to propagate one or more stream data objects from the output of an upstream operator to the input of a downstream operator that is in the same container.

8. The method of claim 1, wherein generating one or more execution units from the declarative description of one or more data stream processing tasks comprises using a compiler.

9. A computer program product comprising a tangible computer readable recordable storage medium having computer readable program code for generating a distributed stream processing application, said computer program product including:

computer readable program code for obtaining a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task;

computer readable program code for generating one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary;

computer readable program code for generating one or more coarse granularity containers that encompass one or more fine granularity stream processing operators;

computer readable program code for using the one or more coarse granularity containers to generate a distributed stream processing application;

computer readable program code for generating one or more containers that encompass a combination of one or more stream processing operators, wherein said generating comprises;

coalescing a combination of one or more operators into one or more containers, wherein said coalescing comprises;

using an optimizer to automatically decide which of the one or more operators are to be coalesced into which of the one or more containers; and

using user input to manually group the one or more operators into the one or more containers; and

computer readable program code for fusing an outflow of an operator into an inflow of a downstream operator within a same container.

10. The computer program product of claim 9, wherein the declarative description expresses at least one stream processing task using one or more fine granularity operators.

11. The computer program product of claim 9, further comprising computer readable program code for using a stream-centric and operator based paradigm for declaring one or more stream processing applications.

12. The computer program product of claim 9, further comprising computer readable program code for using one or more type-generic data stream processing operators that are declaratively specified and for which code is automatically generated by a compiler.

13. The computer program product of claim 9, further comprising computer readable program code for using one or more user-defined operators to wrap legacy code.

14. The computer program product of claim 9, further comprising computer readable program code for using one or more procedural constructs to control a coalescing level of one or more operators that are employed by a stream processing application.

15. The computer program product of claim 9, wherein the computer readable program code for generating one or more execution units from the declarative description of one or more data stream processing tasks comprises computer readable program code for using a compiler.

16. An apparatus for generating a distributed stream processing application, comprising:

a memory; and

at least one processor coupled to said memory and operative to;

obtain a declarative description of one or more data stream processing tasks from a graph of operators, wherein the declarative description expresses at least one stream processing task;

generate one or more execution units from the declarative description of one or more data stream processing tasks, wherein the one or more execution units are deployable across one or more distributed computing nodes, and comprise a distributed data stream processing application binary;

generate one or more coarse granularity containers that encompass one or more fine granularity stream processing operators;

use the one or more coarse granularity containers to generate a distributed stream processing application;

generate one or more containers that encompass a combination of one or more stream processing operators, wherein said generating comprises;

coalescing a combination of one or more operators into one or more containers, wherein said coalescing comprises;

using an optimizer to automatically decide which of the one or more operators are to be coalesced into which of the one or more containers; and

using user input to manually group the one or more operators into the one or more containers; and

fuse an outflow of an operator into an inflow of a downstream operator within a same container.

17. The apparatus of claim 16, wherein the declarative description expresses at least one stream processing task using one or more fine granularity operators.

18. The apparatus of claim 16, wherein the at least one processor coupled to said memory is further operative to use a stream-centric and operator based paradigm for declaring one or more stream processing applications.

19. The apparatus of claim 16, wherein the at least one processor coupled to said memory is further operative to use one or more type-generic data stream processing operators that are declaratively specified and for which code is automatically generated by a compiler.

20. The apparatus of claim 16, wherein the at least one processor coupled to said memory is further operative to use one or more user-defined operators to wrap legacy code.

21. The apparatus of claim 16, wherein the at least one processor coupled to said memory is further operative to use one or more procedural constructs to control a coalescing level of one or more operators that are employed by a stream processing application.

22. The apparatus of claim 16, wherein the at least one processor coupled to said memory and operative to generate one or more execution units from the declarative description of one or more data stream processing tasks is further operative to use a compiler to generate one or more execution units.