Stream processing with context data affinity

• US 9,300,712 B2
• Filed: 06/12/2015
• Issued: 03/29/2016
• Est. Priority Date: 08/01/2014
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Inventors: Nick, Jeffrey Mark, Brown, David Russell, Manoharan, Vinesh Prasanna
• Primary Examiner(s): Ho, Andy
• Application Number:
  US14/737,746
• Publication Number:
  US 20160036878A1
• Time in Patent Office: 291 Days
• Field of Search: 719/318, 709/201, 709/231
• US Class Current: 1/1
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: H04L 65/60 Network streaming of media packets

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for stream processing with contextual data affinity. One of the methods includes receiving an event at a computing node of a stream processing system that includes one or more computing nodes and data representing a topology of functional modules. At a particular functional module of the topology, an operation that requests access to context data related to the event is determined based on a data attribute of the event. The context data is partitioned among the one or more computing nodes. Context data related to the event is obtained from the computing node. The operation of the particular functional module is processed using the context data, and a modified event is generated. The modified event is provided to a subsequent functional module according to the topology of functional modules.

Claims

1. A computer-implemented method comprising:

receiving an event at a first computing node of a stream processing system, the stream processing system comprising a plurality of computing nodes, wherein each computing node of the stream processing system has installed a same stream processing topology of functional modules, wherein context data related to events received by the stream processing system is partitioned among the plurality of computing nodes, and wherein each partition of context data is stored in operational memory of each respective computing node;

obtaining, by the first computing node of the plurality of computing nodes, context data related to the event from operational memory of the first computing node;

processing, by the first computing node, an operation of a functional module of the stream processing topology using the context data obtained from the operational memory of the first computing node; and

providing, to a subsequent functional module of the stream processing topology, a modified event according to the operation of the functional module of the stream processing topology.

2. The method of claim 1, wherein each functional module of the stream processing topology installed on each computing node executes within a same operating system process of the computing node.

3. The method of claim 1, wherein processing the operation of the functional module using the context data comprises modifying the context data in operational memory of the first computing node.

4. The method of claim 3, further comprising:

receiving, by the first computing node, a different second event;

obtaining, by the first computing node, the modified context data from operational memory; and

processing, by the first computing node, the operation of the functional module of the stream processing topology using the modified context data obtained from the operational memory of the first computing node.

5. The method of claim 1, further comprising:

receiving, by the first computing node, a different second event;

determining that an operation of the stream processing topology requests access to context data related to the second event based on a particular data attribute of the different event;

determining that the requested context data is maintained in a partition stored on a different computing node; and

providing the second event to the different computing node.

6. The method of claim 1, further comprising:

maintaining, by the first computing node, functional modules of the stream processing topology and the context data within a same operating system process.

7. The method of claim 1, further comprising:

receiving, at the stream processing system, the event;

determining a computing node maintaining a partition of context data related to the event; and

routing the event to the determined computing node.

8. The method of claim 7, wherein determining a computing node comprises:

processing an operation to modify a data affinity hint in the event;

determining, from the modified data affinity hint, an identifier of a partition of context data stored by a different computing node of the stream processing system; and

providing the modified event to the different computing node of the stream processing system.

9. A stream processing system comprising:

one or more computing nodes, wherein each computing node of the stream processing system has installed a same stream processing topology of functional modules, wherein context data related to events received by the stream processing system is partitioned among the plurality of computing nodes, and wherein each partition of context data is stored in operational memory of each respective computing node, and one or more storage devices storing instructions that are operable, when executed by the one or more computing nodes, to cause the one or more computing nodes to perform operations comprising;

receiving an event at a first computing node of the stream processing system,

obtaining, by the first computing node of the plurality of computing nodes, context data related to the event from operational memory of the first computing node;

processing, by the first computing node, an operation of a functional module of the stream processing topology using the context data obtained from the operational memory of the first computing node; and

providing, to a subsequent functional module of the stream processing topology, a modified event according to the operation of the functional module of the stream processing topology.

10. The stream processing system of claim 9, wherein each functional module of the stream processing topology installed on each computing node executes within a same operating system process of the computing node.

11. The stream processing system of claim 9, wherein processing the operation of the functional module using the context data comprises modifying the context data in operational memory of the first computing node.

12. The stream processing system of claim 11, wherein the operations further comprise:

receiving, by the first computing node, a different second event;

obtaining, by the first computing node, the modified context data from operational memory; and

processing, by the first computing node;

the operation of the functional module of the stream processing topology using the modified context data obtained from the operational memory of the first computing node.

13. The stream processing system of claim 9, wherein the operations further comprise:

receiving, by the first computing node, a different second event;

determining that an operation of the stream processing topology requests access to context data related to the second event based on a particular data attribute of the different event;

determining that the requested context data is maintained in a partition stored on a different computing node; and

providing the second event to the different computing node.

14. The stream processing system of claim 9, wherein the operations further comprise:

maintaining, by the first computing node, functional modules of the stream processing topology and the context data within a same operating system process.

15. The stream processing system of claim 9, wherein the operations further comprise:

receiving, at the stream processing system, the event;

determining a computing node maintaining a partition of context data related to the event; and

routing the event to the determined computing node.

16. The stream processing system of claim 15, wherein determining a computing node comprises:

processing an operation to modify a data affinity hint in the event;

determining, from the modified data affinity hint, an identifier of a partition of context data stored by a different computing node of the stream processing system; and

providing the modified event to the different computing node of the stream processing system.

17. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an event at a first computing node of a stream processing system, the stream processing system comprising a plurality of computing nodes, wherein each computing node of the stream processing system has installed a same stream processing topology of functional modules, wherein context data related to events received by the stream processing system is partitioned among the plurality of computing nodes, and wherein each partition of context data is stored in operational memory of each respective computing node;

obtaining, by the first computing node of the plurality of computing nodes, context data related to the event from operational memory of the first computing node;

processing, by the first computing node, an operation of a functional module of the stream processing topology using the context data obtained from the operational memory of the first computing node; and

providing, to a subsequent functional module of the stream processing topology, a modified event according to the operation of the functional module of the stream processing topology.

18. The computer program product of claim 17, wherein each functional module of the stream processing topology installed on each computing node executes within a same operating system process of the computing node.

19. The computer program product of claim 17, wherein processing the operation of the functional module using the context data comprises modifying the context data in operational memory of the first computing node.

20. The computer program product of claim 19, wherein the operations further comprise:

receiving, by the first computing node, a different second event;

obtaining, by the first computing node, the modified context data from operational memory; and

processing, by the first computing node, the operation of the functional module of the stream processing topology using the modified context data obtained from the operational memory of the first computing node.

21. The computer program product of claim 17, wherein the operations further comprise:

receiving, by the first computing node, a different second event;

determining that an operation of the stream processing topology requests access to context data related to the second event based on a particular data attribute of the different event;

determining that the requested context data is maintained in a partition stored on a different computing node; and

providing the second event to the different computing node.

22. The computer program product of claim 17, wherein the operations further comprise:

maintaining, by the first computing node, functional modules of the stream processing topology and the context data within a same operating system process.

23. The computer program product of claim 17, wherein the operations further comprise:

receiving, at the stream processing system, the event;

determining a computing node maintaining a partition of context data related to the event; and

routing the event to the determined computing node.

24. The computer program product of claim 23, wherein determining a computing node comprises:

processing an operation to modify a data affinity hint in the event;

determining, from the modified data affinity hint, an identifier of a partition of context data stored by a different computing node of the stream processing system; and

providing the modified event to the different computing node of the stream processing system.