DISTRIBUTED DATA PROCESSING METHOD AND SYSTEM

• US 20170083579A1
• Filed: 09/15/2016
• Published: 03/23/2017
• Est. Priority Date: 09/18/2015
• Status: Abandoned Application

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: DU, Chuan, Li, Shan, DUAN, Peile, WEI, Pumeng, Sun, Jing
• Application Number:
  US15/266,897
• Publication Number:
  US 20170083579A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1078 Resource delivery mechanisms

Abstract

A distributed data processing method is provided. The distributed data processing method includes: receiving, by a shard node, data uploaded by a client, wherein the data is directed to a table; storing, by the shard node, the data to a storage directory corresponding to the table; and when the storage is successful, sending, by the shard node, the data to each connected stream computing node to perform stream computing.

Claims

1. A distributed data processing method, comprising:

receiving, by a shard node, data uploaded by a client, wherein the data is directed to a table;

storing, by the shard node, the data to a storage directory corresponding to the table; and

when the storing is successful, sending, by the shard node, the data to a connected stream computing node to perform stream computing.

2. The method according to claim 1, wherein storing the data to the storage directory comprises:

searching for a schema corresponding to the table;

verifying the data by using the schema; and

when the verification is passed, storing the data to the storage directory corresponding to the table.

3. The method according to claim 1, wherein the table is divided into one or more partitions, each partition corresponding to a storage sub-directory in the storage directory, and wherein storing the data to the storage directory comprises:

encapsulating a portion of the data into one or more files according to a predetermined file size or duration; and

storing the one or more files to the storage sub-directories corresponding to the partitions.

4. The method according to claim 1, further comprising:

generating, by the shard node, a first storage operation message after data is stored successfully; and

generating, by the shard node, a second storage operation message after a partition is opened or closed, wherein each of the first storage operation message and the second storage operation message includes one or more of the following;

a file to which the data belongs, an offset of the file to which the data belongs, and a storage sequence ID generated according to a storage order.

5. The method according to claim 4, further comprising:

updating, by the stream computing node, first storage meta information based on the first storage operation message; and

updating, by the shard node, second storage meta information based on the second storage operation message.

6. The method according to claim 5, wherein updating the first storage operation message comprises:

determining whether a first target storage operation message exists in the first storage meta information, the first target storage operation message being associated with a same file as the first storage operation message;

if the first target storage operation message exists in the first storage meta information, replacing the first target storage operation message with the first storage operation message; and

if the first target storage operation message does not exist in the first storage meta information, adding the first storage operation message to the first storage meta information.

7. The method according to claim 5, further comprising:

comparing, by the stream computing node, the first storage operation message with the updated first storage meta information to determine whether a portion of the data is lost or duplicated;

when the portion of the data is lost, reading the lost data from the storage directory, and using a first storage operation message of the lost data to update the first storage meta information; and

when the portion of the data is duplicated, discarding the duplicated data.

8. The method according to claim 7, wherein comparing the first storage operation message with the updated first storage meta information to determine whether data is lost or duplicated comprises:

when a storage sequence ID of the first storage operation message is greater than a target storage sequence ID, determining that data is lost; and

when the storage sequence ID of the first storage operation message is less than the target storage sequence ID, determining that data is duplicated, wherein the target storage sequence ID is a storage sequence ID next to a latest storage sequence ID in the first storage meta information.

9. The method according to claim 7, wherein the first storage meta information identifies an open partition, and wherein reading the lost data from the storage directory comprises:

computing a first candidate storage sequence ID based on the storage sequence ID of the first storage operation message and a latest storage sequence ID in the first storage meta information; and

reading data corresponding to the first candidate storage sequence ID from a storage sub-directory corresponding to the open partition.

10. The method according to claim 5, further comprising:

performing, by the stream computing node, a persistence processing on the first storage meta information;

performing, by the stream computing node, a restoration processing by using persistent first storage meta information during failover;

performing, by the shard node, a persistence processing on the second storage meta information; and

performing, by the shard node, a restoration processing by using persistent second storage meta information during failover.

11. The method according to claim 10, wherein the first storage meta information identifies an open partition, and wherein performing the restoration processing by using the persistent first storage meta information during failover comprises:

loading the persistent first storage meta information;

searching for a latest storage sequence ID from a storage sub-directory corresponding to the open partition;

computing a second candidate storage sequence ID based on the latest storage sequence ID in the storage sub-directory and a latest storage sequence ID in the first storage meta information; and

using a first storage operation message of data having the second candidate storage sequence ID to update the first storage meta information.

12. A distributed data processing system comprising:

one or more shard nodes and one or more stream computing nodes, wherein each of the shard nodes comprises;

a data receiving module configured to receive data uploaded by a client, wherein the data is directed to a table;

a data storing module configured to store the data to a storage directory corresponding to the table; and

a data forwarding module configured to, when the storage is successful, send the data to a connected stream computing node to perform stream computing.

13. The distributed data processing system of claim 12, wherein each of the shard nodes further comprises:

a first storage operation message generating module configured to generate a first storage operation message after data is stored successfully; and

a second storage operation message generating module configured to generate a second storage operation message after a partition is opened or closed, wherein each of the first storage operation message and the second storage operation message includes one or more of the following;

a file to which the data belongs, an offset of the file to which the data belongs, and a storage sequence ID generated according to a storage order.

14. The distributed data processing system of claim 13, wherein each of the stream computing nodes further comprises a first updating module configured to update first storage meta information based on the first storage operation message, and wherein each of the shard nodes further comprises a second updating module configured to update second storage meta information based on the second storage operation message.

15. The distributed data processing system of claim 14, wherein each of the stream computing nodes further comprises:

a data checking module configured to compare the first storage operation message with the updated first storage meta information to determine whether a portion of the data is lost or duplicated;

a reading module configured to, when the portion of the data is lost, read the lost data from the storage directory, and use a first storage operation message of the lost data to update the first storage meta information; and

a discarding module configured to, when the portion of the data is duplicated, discard the duplicated data.

16. The distributed data processing system of claim 14, wherein each of the stream computing nodes further comprises:

a first persistence module configured to perform a persistence processing on the first storage meta information; and

a first restoring module configured to perform a restoration processing by using persistent first storage meta information during failover; and

wherein each of the shard nodes further comprises;

a second persistence module configured to perform a persistence processing on the second storage meta information; and

a second restoring module configured to perform a restoration processing by using persistent second storage meta information during failover.

17. A non-transitory computer readable medium that stores a set of instructions that is executable by at least one processor of a shard node to cause the shard node to perform a distributed data processing method, the distributed data processing method comprising:

receiving data uploaded by a client, wherein the data is directed to a table;

storing the data to a storage directory corresponding to the table; and

when the storing is successful, sending the data to each connected stream computing node to perform stream computing.

18. The non-transitory computer readable medium apparatus of claim 17, wherein the set of instructions that is executable by the at least one processor of the shard node to cause the shard node to further perform:

generating a first storage operation message after data is stored successfully; and

generating a second storage operation message after a partition is opened or closed, wherein each of the first storage operation message and the second storage operation message includes one or more of the following;

a file to which the data belongs, an offset of the file to which the data belongs, and a storage sequence ID generated according to a storage order.

19. The non-transitory computer readable medium apparatus of claim 17, wherein storing the data to the storage directory comprises:

searching for a schema corresponding to the table;

verifying the data by using the schema; and

when the verification is passed, storing the data to the storage directory corresponding to the table.

20. The non-transitory computer readable medium apparatus of claim 17, wherein the table is divided into one or more partitions, each partition corresponding to a storage sub-directory in the storage directory, and wherein storing the data to the storage directory comprises:

encapsulating a portion of the data into one or more files according to a predetermined file size or duration; and

storing the one or more files to the storage sub-directories corresponding to the partitions.