Systems and methods for optimal component composition in a stream processing system

• US 20060200251A1
• Filed: 03/01/2005
• Published: 09/07/2006
• Est. Priority Date: 03/01/2005
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Yu, Philip S., Gu, Xiaohui
• Granted Patent: US 7,562,355 B2
• Time in Patent Office: Days
• US Class Current: 700/1
• CPC Class Codes: H04L 12/4641 Virtual LANs, VLANs, e.g. virtual private networks [VPN] LAN interconnection over a bridge based backbone H04L12/462; encapsulation techniques H04L12/4633; routing of packets H04L45/00; packet switches H04L49/00; virtual private networks for security H04L63/0272

Abstract

A system and method are provided for optimizing component composition in a distributed stream-processing environment having a plurality of nodes capable of being associated with one or more of a plurality of stream processing components. The system includes an adaptive composition probing (ACP) module and a hierarchical state manager. The ACP module probes a subset of the plurality of stream processing components to determine the optimal component composition in response to a stream processing request. The hierarchical state manager manages local and global information for use by said ACP module in determining the optimal component composition.

Claims

1. A system for optimizing component composition in a distributed stream-processing environment having a plurality of nodes capable of being associated with one or more of a plurality of stream processing components, the system comprising:

an adaptive composition probing (ACP) module which probes a subset of the plurality of stream processing components to determine the optimal component composition in response to a stream processing request; and

a hierarchical state manager which manages local and global information for use by said ACP module in determining the optimal component composition.

2. The system according to claim 1, wherein said ACP apparatus adaptively determines the number of probes used for the stream processing request based on at least one of system performance targets and current system conditions.

3. The system according to claim 1, wherein said ACP apparatus adaptively performs per-hop component selection based on at least one of coarse-grain global states, accumulated probing results, and stream rate matching.

4. The system according to claim 1, wherein said hierarchical state manager manages global states in a coarse-grained manner and local states in a fine-grained manner.

5. The system according to claim 1, wherein function, Quality of Service (QoS) and resource needs are allowed to be specified for selecting the component composition.

6. The system according to claim 1, wherein the component composition is determined so as to optimize at least one objective function with respect to load balancing.

7. The system according to claim 6, wherein the determination of the optimal component composition is subject to system and performance constraints.

8. The system according to claim 7, wherein the system and performance constraints comprise at least one of Quality of Service (QoS), resource needs, and rate matching.

9. A method for optimizing component composition in a distributed stream-processing environment having a plurality of nodes capable of being associated with one or more of a plurality of stream processing components, the method comprising the steps of:

performing an adaptive composition probing (ACP) which probes a subset of the plurality of stream processing components to determine the optimal component composition in response to a stream processing request; and

managing local and global information for use by said performing step in determining the optimal component composition.

10. The method according to claim 9, wherein said performing step adaptively determines the number of probes used for the stream processing request based on at least one of system performance targets and current system conditions.

11. The method according to claim 9, wherein said performing step adaptively performs per-hop component selection based on at least one of coarse-grain global state information, accumulated probing results, and stream rate matching.

12. The method according to claim 9, wherein said managing step manages global states in a coarse-grained manner and local states in a fine-grained manner.

13. The method according to claim 9, wherein function, Quality of Service (QoS) and resource needs are allowed to be specified for selecting the component composition.

14. The method according to claim 9, wherein the component composition is determined so as to optimize at least one objective function with respect to load balancing.

15. The method according to claim 14, wherein the determination of the optimal component composition is subject to system and performance constraints.

16. The method according to claim 16, wherein the system and performance constraints comprise at least one of Quality of Service (QoS), resource needs, and rate matching.

17. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for optimizing component composition in a distributed stream-processing environment having a plurality of nodes capable of being associated with one or more of a plurality of stream processing components, the method steps comprising:

performing an adaptive composition probing (ACP) which probes a subset of the plurality of stream processing components to determine the optimal component composition in response to a stream processing request; and

managing local and global information for use by said performing step in determining the optimal component composition.

18. The program storage device according to claim 17, wherein said performing step adaptively determines a number of probes used for the stream processing request based on at least one of system performance targets and current system conditions.

19. The program storage device according to claim 17, wherein said performing step adaptively performs per-hop component selection based on at least one of coarse-grain global state information, accumulated probing results, and stream rate matching.

20. The program storage device according to claim 17, wherein said managing step manages global states in a coarse-grained manner and local states in a fine-grained manner.

21. The program storage device according to claim 17, wherein function, Quality of Service (QoS) and resource needs are allowed to be specified for selecting the component composition.

22. The program storage device according to claim 17, wherein the component composition is determined so as to optimize at least one objective function with respect to load balancing.

23. The program storage device according to claim 22, wherein the determination of the optimal component composition is subject to system and performance constraints.

24. The program storage device according to claim 23, wherein the system and performance constraints comprise at least one of Quality of Service, resource needs, and rate matching.