Method and apparatus for distributed data processing

• US 10,681,118 B2
• Filed: 12/19/2016
• Issued: 06/09/2020
• Est. Priority Date: 12/28/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Zhao, Junping Frank, Zhang, Fenghao, Cao, Yu, Tao, Jun, Sun, Ricky Yuxi
• Primary Examiner(s): Whipple, Brian
• Assistant Examiner(s): Rotolo, Anthony T
• Application Number:
  US15/382,887
• Publication Number:
  US 20170187792A1
• Time in Patent Office: 1,268 Days
• Field of Search: None
• CPC Class Codes: H04L 47/783 Distributed allocation of resources, e.g. bandwidth brokers
• CPC Class Codes: H04L 47/803 Application aware
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1065 Discovery involving distributed pre-established resource-based relationships among peers, e.g. based on distributed hash tables [DHT] pre-configuration of logical or physical connections H04L67/1053

Abstract

Embodiments of the present disclosure provide a method and apparatus for distributed data processing. The method comprises: receiving from an upstream node an input message attached with a shared count, the shared count being used for determining a processing status of a root message associated with the input message; processing the input message to generate one or more new messages; allocating to each of the one or more new messages a respective new shared count based on the received shared count; and transmitting the one or more new messages to one or more downstream nodes respectively. Compared with the prior art, the methods and apparatuses for distributed data processing according to the embodiments of the present disclosure can effectively reduce network traffic overheads and the consumed CPU and memory resources, and would be scalable for different topologies of various distributed data processing systems.

Claims

1. A method for distributed data processing at a current node, comprising:

receiving from an upstream node an input message attached with a shared count, the shared count being used for determining a processing status of a root message associated with the input message;

processing the input message to generate one or more new messages;

allocating to each of the one or more new messages a respective new shared count based on the received shared count; and

transmitting the one or more new messages to one or more downstream nodes respectively, wherein the current node is one of a root node, a leaf node, and an intermediate working node, wherein, in response to the current node being the root node;

the receiving an input message comprises receiving the root message from an external source, allocating a predetermined shared count for the root message, and, in response to receiving the root message from the external source, reporting first status information to a tracking task, wherein, in response to the current node being the leaf node;

in response to receiving the input message attached with the shared count, reporting second status information to the tracking task, and wherein, in response to the current node being the intermediate working node;

avoiding reporting status information to the tracking task.

2. The method of claim 1, wherein the first status information includes an identifier of the root message, the predetermined shared count and a first operating code, and wherein the first operating code indicates to the tracking task that the predetermined shared count is to be added.

3. The method of claim 1, wherein the second status information includes an identifier of the root message, the shared count received from the leaf node and a second operating code, and wherein the second operating code indicates to the tracking task that the shared count is to be deducted.

4. The method of claim 1, wherein the allocating to each of the one or more new messages a respective new shared count comprises:

allocating the new shared count to the new message based on an allocation policy configured at the current node.

5. The method of claim 4, wherein the allocation policy includes at least one of the following:

dividing, based on a number of the one or more downstream nodes, the shared count equally among the one or more new messages; and

allocating, based on a weight of each of the one or more downstream nodes, a respective new shared count to each of the one or more new messages.

6. The method of claim 1, wherein the allocating to each of the one or more new messages a respective new shared count comprises:

obtaining a total shared count by adding an additional shared count to the shared count; and

dividing the total shared count among the one or more new messages.

7. The method of claim 6, further comprising:

in response to adding the additional shared count to the shared count, reporting third status information to a tracking task.

8. The method of claim 7, wherein the third status information includes an identifier of the root message, the additional shared count and a third operating code, and wherein the third operating code indicates to the tracking task that the additional shared count is to be added.

9. A method for distributed data processing, comprising:

in response to receiving, at a root node, a root message from an external source;

allocating a predetermined shared count for the root message; and

reporting first status information associated with the root message to a tracking task;

in response to receiving, at a leaf node, an input message associated with the root message;

reporting second status information associated with the root message to the tracking task;

receiving, at the tracking task, the first status information associated with the root message from the root node, the first status information including an identifier of the root message, a first shared count allocated to the root message, and a first operating code;

receiving, at the tracking task, the second status information associated with the root message from the leaf node, the second status information including the identifier of the root message, a second shared count received by the leaf node, and a second operating code, the first status information and the second status information being received while avoiding receiving at the tracking task status information associated with the root message from an intermediate working node;

processing, based on the first operating code and the second operating code included in the first status information and the second status information, respectively, the first shared count and the second shared count to determine a total shared count corresponding to the root message; and

determining, based on the total shared count, a processing status of the root message.

10. The method of claim 9, wherein the processing of the first operating code and the second operating code comprises:

adding the first shared count to the total shared count based on the first operating code; and

deducting the second shared count from the total shared count based on the second operating code.

11. The method of claim 9, wherein an initial value of the total shared count is set to zero and the determining a processing status of the root message comprises:

in response to the total shared count being zero, determining that processing of the root message succeeds; and

in response to the total shared counts being non-zero, determining that a failure occurs during the processing of the root message.

12. The method of claim 9, further comprising:

receiving third status information from a node added with a third shared count, the third status information including an identifier of the root message, the additional shared count and a third operating code.

13. The method of claim 12, wherein the processing of the first operating code and the second operating code comprises:

adding the third shared count to the total shared count based on the third operating code.

14. An apparatus for distributed data processing at a current node, comprising:

a processor configured to execute a plurality of software modules including a message receiving module, a message processing module, a shared count allocating module, a message transmitting module, and a first reporting module, wherein the processor is configured to execute the message receiving module to receive from an upstream node an input message attached with a shared count, the shared count being used for determining a processing status of a root message associated with the input message;

wherein the processor is configured to execute the message processing module to process the input message to generate one or more new messages;

wherein the processor is configured to execute the shared count allocating module to allocate to each of the one or more new messages a respective new shared count based on the received shared count; and

wherein the processor is configured to execute the message transmitting module to transmit the one or more new messages to one or more downstream nodes respectively, wherein the current node is one of a root node, a leaf node, and an intermediate working node, wherein, in response to the current node being the root node;

the processor is further configured to execute the message receiving module to receive the root message from an external source, and allocate a predetermined shared count for the root message; and

the processor is configured to execute the first reporting module, in response to receiving the root message from the external source, to report first status information to a tracking task, wherein, in response to the current node being the leaf node;

the processor is further configured to execute the first reporting module, in response to receiving the input message attached with the shared count, to report second status information to the tracking task, and wherein, in response to the current node being the intermediate working node;

the processor is configured to avoid reporting status information to the tracking task.