DISTRIBUTED DATA STREAM PROCESSING METHOD AND SYSTEM

• US 20130139166A1
• Filed: 11/19/2012
• Published: 05/30/2013
• Est. Priority Date: 11/24/2011
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Zhang, Xu, Yang, Zhixiong, Xu, Jia, Deng, Zhonghua
• Granted Patent: US 9,250,963 B2
• Time in Patent Office: Days
• US Class Current: 718/102
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

Embodiments of the present application relate to a distributed data stream processing method, a distributed data stream processing device, a computer program product for processing a raw data stream and a distributed data stream processing system. A distributed data stream processing method is provided. The method includes dividing a raw data stream into a real-time data stream and historical data streams, processing the real-time data stream and the historical data streams in parallel, separately generating respective results of the processing of the real-time data stream and the historical data streams, and integrating the generated processing results.

Claims

1. A distributed data stream processing method, the method comprising:

dividing a raw data stream into a real-time data stream and historical data streams;

processing the real-time data stream and the historical data streams in parallel;

separately generating respective results of the processing of the real-time data stream and the historical data streams; and

integrating the respective generated processing results.

2. The method as described in claim 1, wherein the processing of the real-time data stream is performed based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

3. The method as described in claim 2, wherein the processing of the real-time data stream comprises:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block into a plurality of data units;

processing the plurality of data units in parallel; and

aggregating the results of the processing of the plurality of data units.

4. The method as described in claim 1, wherein the processing of the historical data streams includes dividing the historical data streams based on a dimension and the dividing of the historical data streams is performed in parallel.

5. The method as described in claim 3, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

6. The method as described in claim 1, wherein the real-time data stream and the historical data streams are processed differently.

7. The method as described in claim 4, wherein the dimension relates to a behavior of a user.

8. The method as described in claim 4, wherein different preferences of a user correspond to different dimensions, and data units relating to the different dimensions are processed by separate modules.

9. A distributed data stream processing device, the device comprising:

at least one processor configured to;

divide a raw data stream into a real-time data stream and historical data streams;

process the real-time data stream and the historical data streams in parallel and separately generate respective processing results; and

integrate the respective generated processing results; and

a memory coupled to the at least one processor and configured to provide the at least one processor with instructions.

10. The device as described in claim 9, wherein the processing of the real-time data stream includes dividing the real-time data stream based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

11. The device as described in claim 10, wherein the processing of the real-time data stream includes:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block in parallel into a plurality of data units, and transmitting the plurality of data units to a corresponding plurality of functional modules to process the data units in parallel; and

aggregating results of data processing by the plurality of functional modules.

12. The device as described in claim 9, wherein the processing of the historical data stream includes dividing the historical data streams based on a dimension and performing parallel processing of the historical data streams.

13. The device as described in claim 11, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

14. A computer program product for processing a raw data stream, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

dividing the raw data stream into a real-time data stream and historical data streams;

processing the real-time data stream and the historical data streams in parallel;

separately generating respective results of the processing of the real-time data stream and the historical data streams; and

integrating the respective generated processing results.

15. The computer program product as described in claim 14, wherein the processing of the real-time data stream is performed based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

16. The computer program product as described in claim 15, wherein the processing of the real-time data stream comprises:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block into a plurality of data units in parallel;

separately transmitting the plurality of data units to a corresponding plurality of functional modules for processing in parallel in response to the dividing of each data block; and

aggregating the results of the processing of the plurality of functional modules.

17. The computer program product as described in claim 14, wherein the processing of the historical data streams includes dividing the historical data streams based on a dimension and the dividing of the historical data streams is performed in parallel.

18. The computer program product as described in claim 16, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

19. A distributed data stream processing system, the system comprising:

a plurality of application servers comprising;

a data recognition module configured to divide a raw data stream into a real-time data stream and historical data streams;

a parallel processing module configured to process the real-time data stream and the historical data streams in parallel, and separately generate respective processing results; and

a data integration module configured to integrate the respective generated results of processing.

20. The system as described in claim 19, wherein the parallel processing module divides the real-time data stream based on dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

21. The system as described in claim 20, wherein the parallel processing module comprises:

a horizontal division module configured to divide the real-time data stream into a plurality of data blocks;

a plurality of vertical division modules configured to divide each data block in parallel into a plurality of data units, and transmit the plurality of data units to a corresponding plurality of functional modules to process the data units in parallel; and

a result aggregation module configured to aggregate the results of data processing by the plurality of functional modules.

22. The system as described in claim 19, wherein the parallel processing module divides the historical data streams based on a dimension and performs parallel processing of the historical data streams.

23. The system as described in claim 21, wherein the horizontal division module divides the real-time data stream into the plurality of data blocks based on time.