DYNAMIC RUNTIME CHOOSING OF PROCESSING COMMUNICATION METHODS

• US 20130080652A1
• Filed: 11/20/2012
• Published: 03/28/2013
• Est. Priority Date: 07/26/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cradick, Ryan K., Santosuosso, John M., Schulz, Brandon W.
• Granted Patent: US 9,148,496 B2
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: H04L 69/08 Protocols for interworking; Protocol conversion
• CPC Class Codes: H04L 69/18 Multiprotocol handlers, e.g. single devices capable of handling multiple protocols

Abstract

Techniques are described for assigning and changing communication protocols for a pair of processing elements. The communication protocol determines how the pair of processing elements transmits data in a stream application. The pair may be assigned a communication protocol (e.g., TCP/IP or a protocol that uses a relational database, shared file system, or shared memory) before the operator graph begins to stream data. This assignment may be based on a priority of the processing elements and/or a priority of the communication protocols. After the operator graph begins to stream data, the pair of processing elements may switch to a different communication protocol. The decision to switch the communication protocol may be based on whether the pair of processing elements or assigned communication protocol is meeting established performance standards for the stream application.

Claims

1. A computer-implemented method for processing data, comprising:

establishing an operator graph of a plurality of processing elements, the operator graph defining at least one execution path, wherein a first processing element of the plurality of processing elements is configured to transmit data to at least one downstream processing element, and wherein the first processing element is capable of transmitting data to the at least one downstream processing element using at least one of a first communication protocol and a second communication protocol;

transmitting, by one or more computer processors, data from the first processing element to the at least one downstream processing element using the first communication protocol;

determining whether to switch from the first communication protocol to the second communication protocol; and

upon determining to switch, transmitting data from the first processing element to the at least one downstream processing element using the second communication protocol.

2. The method of claim 1, further comprising:

prioritizing at least one of the plurality of processing elements and the first and second communication protocol based on at least one of a predicted amount of data to be transmitted by each of the plurality of processing elements, a job the plurality of processing elements are associated with, the cost of the first and second communication protocol, a data transfer rate of the first and second communication protocol, and an energy cost of the first and second communication protocol; and

before transmitting data using the first communication protocol, assigning the first processing element to use the first communication protocol based on the prioritization.

3. The method of claim 2, wherein determining whether to switch is based on the prioritization.

4. The method of claim 1, wherein the first or second communication protocol transfer data from the first processing element to the downstream processing element using at least one of transmission control protocol (TCP), user datagram protocol (UDP), Internet Protocol (IP), a shared file system protocol, a relational database protocol, a shared memory protocol, and remote direct memory access protocol.

5. The method of claim 1, wherein the first communication protocol is an inter-nodal communication protocol and wherein the second communication protocol is an intra-nodal communication protocol.

6. The method of claim 5, wherein the first processing element is on a different compute node as the at least one downstream processing element when transmitting data using the first communication protocol, and wherein the first processing element is on a same compute node as the at least one downstream processing element when transmitting data using the second communication protocol.

7. The method of claim 1, wherein the first processing element transmits data using both the first and second communication protocol from a same compute node.

8. The method of claim 1, wherein determining whether to switch from the first communication protocol to the second communication protocol is based on at least one of computer processing unit (CPU) utilization, page faults, system limitations of the first and second communication protocol, and contention between two of the plurality of processing elements using the first communication protocol.

9. The method of claim 1, wherein the first processing element is communicatively coupled to the at least one downstream processing element by both the first and the second communication protocols simultaneously.