PORT COMPATIBILTY CHECKING FOR STREAM PROCESSING

• US 20120047467A1
• Filed: 08/17/2010
• Published: 02/23/2012
• Est. Priority Date: 08/17/2010
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Pfeifer, Michael D., Sun, Jingdong
• Application Number:
  US12/857,790
• Publication Number:
  US 20120047467A1
• Time in Patent Office: Days
• US Class Current: 715/853
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247

Abstract

A port compatibility connection engine for a large scale stream processing framework is provided. The port compatibility management unit analyzes port definitions of processing elements (PEs) to validate interconnectivity between said elements. In particular, the port compatibility management unit determines the ability of the PEs to produce and/or consume data streams based on the data stream schema definitions specified on the PE ports. In addition, the port compatibility management unit analyzes security, scope, persistence, and other factors that impact interconnectivity. The port compatibility management unit generates a connection topology snapshot based on the above analysis and identifies the combination of PEs that cannot interconnect and provides the information in an output format that allows for visualization, filtering, and automatic fix capability.

Claims

1. A method for port compatibility checking in data stream processing systems, the method comprising:

analyzing each processing element pair combination in a stream processing application for connection compatibility;

creating a topology snapshot of the stream mining application based on the analysis;

storing the topology snapshot in a topology snapshot repository residing on the data stream processing system;

automatically fixing connection compatibility issues identified by the analysis via an automated fix engine, whenever possible; and

updating the topology snapshot of the stream mining application based on the fixing.

2. The method of claim 1, wherein the method further comprises the step of:

identifying any unresolved connection compatibility issues that remain within the topology snapshot;

interactively reviewing connection compatibility issues identified within the topology snapshot for possible fixes; and

interactively repairing connection compatibility issues identified by the review, whenever possible.

3. The method of claim 2, wherein the method step of interactively reviewing connection compatibility issues identified within the topology snapshot for possible fixes further comprises the step of:

displaying connection compatibility issues to a user via a data visualization interface.

4. The method of claim 1, wherein the method is performed on the stream processing application prior to runtime of the stream processing application.

5. The method of claim 4, wherein the step of analyzing each processing element pair combination in a stream processing application for connection compatibility prior to runtime of the stream processing application includes performing at least one of the following checks between each element in the element pair combination:

schema data type matching, application scope matching, version level matching, security level matching, and operators or PE'"'"'s not running.

6. The method of claim 1, wherein the method is performed on the stream processing application at runtime of the stream processing application.

7. The method of claim 6, wherein the step of analyzing each processing element pair combination in a stream processing application for connection compatibility at runtime includes performing at least one of the following checks:

checking security policies changed at runtime, checking network status, checking data stream monitor service, and checking operators or PE'"'"'s not running.

8. A computer program product for port compatibility checking in data stream processing systems, the computer program product disposed in a computer readable storage medium, the computer program product comprising computer program instructions capable of:

analyzing each processing element pair combination in a stream processing application for connection compatibility;

creating a topology snapshot of the stream mining application based on the analysis;

storing the topology snapshot in a topology snapshot repository residing on the data stream processing system;

automatically fixing connection compatibility issues identified by the analysis via an automated fix engine, whenever possible; and

updating the topology snapshot of the stream mining application based on the fixing.

9. The computer program product of claim 8, further comprising computer program instructions capable of:

identifying any unresolved connection compatibility issues that remain within the topology snapshot;

interactively reviewing connection compatibility issues identified within the topology snapshot for possible fixes; and

interactively repairing connection compatibility issues identified by the review, whenever possible.

10. The computer program product of claim 8, wherein the computer program instructions capable of interactively reviewing connection compatibility issues identified within the topology snapshot for possible fixes further comprises:

displaying connection compatibility issues to a user via a data visualization interface.

11. The computer program product of claim 8, wherein the instructions are performed on the stream processing application prior to runtime of the stream processing application.

12. The computer program product of claim 11, wherein the step of analyzing each processing element pair combination in a stream processing application for connection compatibility prior to runtime of the stream processing application includes performing at least one of the following checks between each element in the element pair combination:

schema data type matching, application scope matching, version level matching, security level matching, and operators or PE'"'"'s not running.

13. The computer program product of claim 8, wherein the instruction are performed on the stream processing application at runtime of the stream processing application.

14. The computer program product of claim 13, wherein the step of analyzing each processing element pair combination in a steam processing application for connection compatibility at runtime include performing at least one of the following checks between each element in the element pair combination:

checking security policies changed at runtime, checking network status, checking data stream monitor service, and checking operators or PE'"'"'s not running.

15. An apparatus for port compatibility checking in a data stream processing system, comprising:

a connection checking engine (CCE) for analyzing every processing element pair combination in a stream processing application for connection compatibility;

a topology snapshot repository (TSR) communicatively coupled to the connection checking engine for storing a topology snapshot generated by the connection checking engine;

a data visualization interface (DVI) communicatively coupled to the topology snapshot repository for displaying connection compatibility information; and

an automated fix engine communicatively coupled to the TSR and the DVI for repairing connection compatibility issues identified by the CCE.

16. The apparatus of claim 15, wherein the connection checking engine further comprises:

a CCE repository for keeping port compatibility checking policies, configuration settings and an original connection model; and

a connection checking component which checks connection pairs within the original connection model based on port compatibility checking policies and configuration settings stored in the CCE repository; and

17. The apparatus of claim 15, wherein the topology snapshot is a true edges object with connection information indicating success or failure for all ports and streams within the stream processing application.

18. The apparatus of claim 15, wherein the automated fix engine includes a self-learning engine.