STREAM COMPUTING SYSTEM AND METHOD

• US 20160335287A1
• Filed: 05/13/2016
• Published: 11/17/2016
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: LI, Meifang, WEI, Pumeng, DUAN, Peile, LI, Shan
• Granted Patent: US 10,877,935 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

System and method are disclosed for stream computing. An exemplary method may include receive data from a data processing module and determining whether the received data are effective data that are neither incomplete nor duplicative. The method may also include obtaining the effective data when it is determined that the received data are either incomplete or duplicative. In addition, the method may include storing the effective data in a log file of a file system.

Claims

1. A method, implemented by at least one processor device, for processing stream data, the method comprising:

receiving stream data from a data source;

dividing the received stream data into records;

writing the records into a file system;

arranging the records into at least one block; and

sending the at least one block to a logic processing module after the records are successfully written into the file system.

2. The method of claim 1, comprising:

assigning a version number to each record; and

writing the records with the corresponding version numbers into the file system.

3. The method of claim 2, wherein the version number includes a batch number and a sequence number, and wherein the version number uniquely identifies the corresponding record in the file system.

4. The method of claim 1, comprising:

assigning a unique block identifier to each of the at least one block; and

sending the at least one block with the corresponding block identifier to the logic processing module.

5. The method of claim 1, comprising:

temporarily storing the at least one block in a block buffer; and

sending the at least one block by continuously reading the block buffer and broadcasting information read from the block buffer to the logic processing module.

6. The method of claim 1, comprising:

periodically generating a checkpoint; and

write the checkpoint into the file system.

7. A method, implemented by at least one processor device, for processing data, the method comprising:

receiving data from a data processing module;

determining whether the received data are effective data that are neither incomplete nor duplicative;

obtaining the effective data when it is determined that the received data are either incomplete or duplicative; and

storing the effective data in a log file of a file system.

8. The method of claim 7, wherein the received data include at least one block and a block identifier corresponding to each of the at least one block, and the method comprises:

storing an identifier of a latest received block; and

determining whether the received data are effective data by comparing the identifier of the latest received block with the block identifier.

9. The method of claim 8, wherein obtaining the effective data includes:

when it is determined that the received data include a duplicated block, dropping the duplicated block.

10. The method of claim 7, wherein obtaining the effective data includes:

retrieving lost data from the file system when it is determined that the received data are incomplete.

11. The method claim 10, wherein the received data include one or more records and a version number corresponding to each of the one or more records, and the method comprises:

determining the version number corresponding to the lost data; and

retrieving the lost data from the file system based on the determined version number.

12. The method of claim 7, comprising:

transmitting the obtained effective data to an operation unit for processing, the operation unit operating in a secure runtime environment;

receiving processed data from the operation unit; and

sending the processed data to another data processing module.

13. The method of claim 12, comprising:

determining, by the operation unit, whether the operation unit receives duplicative data by comparing a version number corresponding to data received by the operation unit and a latest version number corresponding to latest received data by the operation unit; and

dropping, by the operation unit, the duplicative data when it is determined that the operation unit receives the duplicative data.

14. The method of claim 12, comprising:

generating, by the operation unit, at least one outputs by processing the effective data;

assigning, by the operation unit, a version number to each of the at least one outputs; and

determining whether the received processed data include duplicative data by comparing the version number assigned to each of the at least one outputs with a latest version number corresponding to latest received processed data.

15. The method of claim 7, comprising:

periodically generating a checkpoint including a version number of a latest received record of the effective data;

determining whether to make the checkpoint persistent based on the version number; and

storing the checkpoint in the file system when it is determined to make the checkpoint persistent.

16. The method of claim 15, wherein the version number includes a first batch number indicating a latest record received by a parent operation unit of the at least one processor device, the parent operation unit being configured to update the checkpoint periodically, and the method comprises:

comparing the first batch number with a second batch number, the second batch number corresponding to a latest record received by a child operation unit of the at least one processor device, the child operation unit being configured to receive data from the parent operation unit; and

determining to make the checkpoint persistent when the first batch number is less than or equal to the second batch number.

17. The method of claim 15, comprising:

recovering the effective data from the log file based on the checkpoint in a failover event; and

transmitting at least a portion of the recovered effective data to an operation unit.

18. A system for processing stream data, comprising:

a data receiving module including at least one processor device, the data receiving module being configured to;

receive stream data from a data source;

divide the received stream data into records;

write the records into a file system;

arrange the records into at least one block; and

send the at least one block to a logic processing module after the records are successfully written into the file system; and

the logic processing module including at least one processor device, the logic processing module being configured to;

receive the at least one block from the data receiving module;

determine whether the received at least one block are effective data that are neither incomplete nor duplicative;

obtain the effective data when it is determined that the received data are either incomplete or duplicative; and

store the effective data in a log file of the file system.

19. The system of claim 18, wherein the data receiving module is configured to:

assign a version number to each record, the version number including a batch number and a sequence number, wherein the version number uniquely identifies the corresponding record in the file system; and

write the records with the corresponding version numbers into the file system.

20. The system of claim 18, wherein the received at least one block includes a block identifier corresponding to each of the at least one block, and wherein the logic processing module is configured to:

store an identifier of a latest received block; and

determine whether the received at least one block are effective data by comparing the identifier of the latest received block with the block identifier.

21. The system of claim 18, wherein the logic processing module is configured to:

retrieve lost data from the file system when it is determined that the received at least one block is incomplete.

22. The system of claim 21, wherein the received at least one block includes one or more records and a version number corresponding to each of the one or more records, and wherein the logic processing module is configured to:

determine the version number corresponding to the lost data; and

retrieving lost data from the file system based on the determined version number.

23. The system of claim 18, wherein the logic processing module includes at least one operation unit, and wherein the logic processing module is configured to:

transmit the obtained effective data to the at least one operation unit for processing, the at least one operation unit operating in a secure runtime environment;

receive processed data from the at least one operation unit; and

send the processed data to another logic processing module.

24. The system of claim 23, wherein the at least one operation unit is configured to:

determine whether the at least one operation unit receives duplicative data by comparing a version number corresponding to data received by the at least one operation unit and a latest version number corresponding to latest received data by the at least one operation unit; and

drop the duplicative data when it is determined that the at least one operation unit receives the duplicative data.

25. The system of claim 23, wherein:

the at least one operation unit is configured to;

generate at least one outputs by processing the effective data; and

assign a version number to each of the at least one outputs; and

the logic processing module is configured to;

determine whether the received processed data include duplicative data by comparing the version number assigned to each of the at least one outputs with a latest version number corresponding to latest received processed data.

26. The system of claim 18, wherein the logic processing module is configured to:

periodically generate a checkpoint including a version number of a latest received record of the effective data;

determine whether to make the checkpoint persistent based on the version number; and

store the checkpoint in the file system when it is determined to make the checkpoint persistent.

27. The system of claim 26, wherein the version number includes a first batch number, and the logic processing module is configured to:

compare the first batch number with a second batch number, the second batch number corresponding to a latest record received by an operation unit of the logic processing module; and

determine to make the checkpoint persistent when the first batch number is less than or equal to the second batch number.

28. The system of claim 26, wherein the logic processing module is configured to:

recover the effective data from the log file based on the checkpoint in a failover event; and

transmit at least a portion of the recovered effective data to an operation unit of the logic processing module.

29. An apparatus for processing stream data, comprising:

at least one processor device configured to;

receive stream data from a data source;

divide the received stream data into records;

write the records into a file system;

arrange the records into at least one block; and

send the at least one block to a logic processing device after the records are successfully written into the file system.

30. An apparatus for processing stream data, comprising:

at least one processor device configured to;

receive data from a data processing device;

determine whether the received data are effective data that are neither incomplete nor duplicative;

obtain the effective data when it is determined that the received data are either incomplete or duplicative; and

store the effective data in a log file of a file system.