Multi-granular stream processing

• US 8,892,762 B2
• Filed: 12/15/2009
• Issued: 11/18/2014
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Krishnamurthy, Rajaram B., Parris, Carl J., Boice, Charles, Liu, Lurng-Kuo
• Primary Examiner(s): Whipple, Brian P
• Application Number:
  US12/637,972
• Publication Number:
  US 20110145429A1
• Time in Patent Office: 1,799 Days
• Field of Search: None
• US Class Current: 709/231
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: H04L 65/765 intermediate

Abstract

Stream processing is facilitated by distributing responsibility for processing the stream to multiple components of a computing environment. A programmable unit receives one or more streams and determines the operations to be performed for the one or more streams and which components of the computing environment are to perform those operations. It forwards data relating to the one or more streams to one or more components of the computing environment for processing and/or information purposes.

Claims

1. A computer program product for facilitating stream processing in a computing environment, said computer program product comprising:

a non-transitory computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

obtaining, by a programmable unit, a description of one or more streams to be received by the programmable unit from at least one source of the one or more streams, the description indicating a plurality of operations to be performed on content of the one or more streams to be received by the programmable unit;

receiving, by the programmable unit, from a first component of the computing environment, different from the at least one source of the one or more streams, data to program, based at least in part on the description of the one or more streams, the programmable unit to recognize one set of one or more operations, of the plurality of operations, determined by the first component to be performed by the programmable unit, and to recognize another set of one or more operations, of the plurality of operations, determined by the first component to be performed by a second component of the computing environment;

receiving, by the programmable unit, the one or more streams from the at least one source; and

processing by the programmable unit the one or more streams, wherein the processing includes;

performing the one set of one or more operations for the one or more streams at the programmable unit to obtain at least one result of performing the one set of one or more operations; and

performing at least one of;

forwarding one or more results of the at least one result to the second component for use in performing the another set of one or more operations by the second component;

or

forwarding one or more results of the at least one result to the first component.

2. The computer program product of claim 1, wherein the obtaining comprises receiving a summary record from at least one source of the one or more streams, the summary record including one or more characteristics of the one or more streams.

3. The computer program product of claim 2, wherein the first component comprises a server coupled to the programmable unit, and wherein the method further comprises the programmable unit providing the summary record to the server.

4. The computer program product of claim 3, wherein the method further comprises updating by the server one or more stream data structures, in response to receiving the summary record.

5. The computer program product of claim 1, wherein the obtaining comprises receiving a workload directive from the first component, the workload directive describing one or more characteristics relating to the one or more streams.

6. The computer program product of claim 1, wherein the processing further comprises determining by the programmable unit, based on the received data to program the programmable unit, the one set of operations to be performed by the programmable unit and the another set of operations to be performed by the second component.

7. The computer program product of claim 6, wherein the second component comprises one or more computing units of a stream cluster coupled to the programmable unit.

8. The computer program product of claim 7, wherein the one or more computing units comprise one or more accelerators.

9. The computer program product of claim 1, wherein the method further comprises providing output resulting from processing the one or more streams to the first component, wherein the provided output comprises the one or more results of the at least one result of performing the one set of one or more operations by the programmable unit and at least one result of performing the another set of one or more operations by the second component.

15. The computer program product of claim 1, wherein the programmable unit is programmed by the first component identifying, via the received data, processing logic for the programmable unit, to enable the programmable unit to process the one or more streams.

16. The computer program product of claim 1, wherein the one or more streams comprise the description of the one or more streams, and wherein the programmable unit obtains the description of the one or more streams with receipt of the one or more streams.

17. The computer program product of claim 1, wherein, responsive to a source of the description of the one or more streams comprising the at least one source of the one or more streams, the processing the one or more streams comprises:

providing data relating to the one or more streams to the first component;

performing at the programmable unit the one set of one or more operations, the one set of one or more operations recognized by the programmable unit as being those operations which are to be performed by the programmable unit; and

forwarding one or more results of the at least one result to the second component to facilitate performance of the another set of one or more operations at the second component, the another set of one or more operations recognized by the programmable unit as being those operations which are to be performed by the second component.

18. The computer program product of claim 1, wherein, responsive to a source of the description of the one or more streams being the first component, the first component being other than the at least one source of the one or more streams, the processing the one or more streams comprises:

performing at the programmable unit the one set of one or more operations, the one set of one or more operations recognized by the programmable unit as being those operations which are to be performed by the programmable unit; and

forwarding one or more results of the at least one result to the second component to facilitate performance of the another set of one of more operations at the second component, the another set of one or more operations recognized by the programmable unit as being those operations which are to be performed by the second component.

19. The computer program product of claim 1, wherein determination of the one set of one or more operations to be performed by the programmable unit and determination of the another set of one or more operations to be performed by the second component are based on levels of compute intensity of the respective one set of one or more operations and another set of one or more operations, wherein relatively less computationally-intensive operations are determined to be performed by the programmable unit and relatively more computationally-intensive operations are determined to be performed by the second component.

20. The computer program product of claim 1, wherein the at least one result of performing the one set of one or more operations by the programmable unit comprise multiple results, the multiple results comprising (i) at least one result of relatively coarse data granularity and (ii) at least one result of relative fine data granularity for further processing by the second component as part of the another set of one or more operations to be performed by the second component, and wherein processing the one or more streams by the programmable unit comprises forwarding the at least one result of relatively coarse data granularity to the first component and forwarding the at least one result of relatively fine data granularity to the second component for further processing by the another component in performing the another set of one or more operations.

10. A computer system for facilitating stream processing in a computing environment, the computer system comprising:

a memory; and

a processor in communications with the memory, wherein the computer system is capable of performing a method, said method comprising;

obtaining, by a programmable unit, a description of one or more streams to be received by the programmable unit from at least one source of the one or more streams, the description indicating a plurality of operations to be performed on content of the one or more streams to be received by the programmable unit;

receiving, by the programmable unit, from a first component of the computing environment, different from the at least one source of the one or more streams, data to program, based at least in part on the description of the one or more streams, the programmable unit to recognize one set of one or more operations, of the plurality of operations, determined by the first component to be performed by the programmable unit, and to recognize another set of one or more operations, of the plurality of operations, determined by the first component to be performed by a second component of the computing environment;

receiving, by the programmable unit, the one or more streams from the at least one source; and

processing by the programmable unit the one or more streams, wherein the processing includes;

performing the one set of one or more operations for the one or more streams at the programmable unit to obtain at least one result of performing the one set of one or more operations; and

performing at least one of;

forwarding one or more results of the at least one result to the second component for user in performing the another set of one or more operations by the second component;

or

forwarding one or more results of the at least one result to the first component.

11. The computer system of claim 10, wherein the obtaining comprises at least one of receiving a summary record from at least one source of the one or more streams, the summary record including one or more characteristics of the one or more streams or receiving a workload directive from the first component, the workload directive describing one or more characteristics relating to the one or more streams.

12. The computer system of claim 10, wherein the processing further comprises determining by the programmable unit, based on the received data to program the programmable unit, the one set of operations to be performed by the programmable unit and the another set of operations to be performed by the second component.

13. A method of facilitating stream processing in a computing environment, said method comprising:

obtaining, by a programmable unit, a description of one or more streams to be received by the programmable unit from at least one source of the one or more streams, the description indicating a plurality of operations to be performed on content of the one or more streams to be received by the programmable unit;

receiving, by the programmable unit, from a first component of the computing environment, different from the at least one source of the one or more streams, data to program, based at least in part on the description of the one or more streams, the programmable unit to recognize one set of one or more operations, of the plurality of operations, determined by the first component to be performed by the programmable unit, and to recognize another set of one or more operations, of the plurality of operations, determined by the first component to be performed by a second component of the computing environment;

receiving, by the programmable unit, the one or more streams from the at least one source; and

processing by the programmable unit the one or more streams, wherein the processing includes;

performing the one set of one or more operations for the one or more streams at the programmable unit to obtain at least one result of performing the one or more operations; and

performing at least one of;

forwarding one or more results of the at least one result to the second component for use in performing the another set of one or more operations by the second component;

or

forwarding one or more results of the at least one result to the first component.

14. The method of claim 13, wherein the obtaining comprises at least one of receiving a summary record from at least one source of the one or more streams, the summary record including one or more characteristics of the one or more streams or receiving a workload directive from the first component, the workload directive describing one or more characteristics relating to the one or more streams.