DISTRIBUTED DATA STREAM PROCESSING METHOD AND SYSTEM

• US 20160179898A1
• Filed: 12/21/2015
• Published: 06/23/2016
• Est. Priority Date: 11/24/2011
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Zhang, Xu, Yang, Zhixiong, Xu, Jia, Deng, Zhonghua
• Granted Patent: US 9,727,613 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

Embodiments of the present application relate to a distributed data stream processing method, a distributed data stream processing device, a computer program product for processing a raw data stream and a distributed data stream processing system. A distributed data stream processing method is provided. The method includes dividing a raw data stream into a real-time data stream and historical data streams, processing the real-time data stream and the historical data streams in parallel, separately generating respective results of the processing of the real-time data stream and the historical data streams, and integrating the generated processing results.

Claims

1. (canceled)

2. A method, comprising:

dividing a raw data stream into a real-time data stream and one or more historical data streams, comprising;

dividing the real-time data stream into a plurality of data units based at least in part on at least one dimension;

processing the real-time data stream and the one or more historical data streams in parallel, wherein the processing of the real-time data stream comprises;

processing the plurality of data units of the real-time data stream in parallel;

separately generating respective results of the processing of the real-time data stream and the one or more historical data streams; and

integrating the respective generated processing results.

3. The method as described in claim 2, wherein the processing of the real-time data stream is performed based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

4. The method as described in claim 2, wherein the processing of the real-time data stream comprises:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block into the plurality of data units; and

aggregating the results of the processing of the plurality of data units.

5. The method as described in claim 2, wherein the processing of the one or more historical data streams includes dividing the one or more historical data streams based on one or more dimensions and the dividing of the one or more historical data streams is performed with the dividing of the real-time data stream in parallel.

6. The method as described in claim 2, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

7. The method as described in claim 2, wherein the real-time data stream and the one or more historical data streams are processed separately.

8. The method as described in claim 2, wherein the dimension is associated with a behavior of a user.

9. The method as described in claim 2, wherein different preferences of a user correspond to different dimensions, and data units associated with different dimensions are processed separately.

10. A device, comprising:

a processor; and

a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to;

divide a raw data stream into a real-time data stream and one or more historical data streams, comprising to;

divide the real-time data stream into a plurality of data units based at least in part on at least one dimension;

process the real-time data stream and the one or more historical data streams in parallel and separately generate respective processing results, wherein the processing of the real-time data stream comprises to;

process the plurality of data units of the real-time data stream in parallel; and

integrate the respective generated processing results; and

a memory coupled to the at least one processor and configured to provide the at least one processor with instructions.

11. The device as described in claim 10, wherein the processing of the real-time data stream includes dividing the real-time data stream based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

12. The device as described in claim 10, wherein the processing of the real-time data stream includes:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block in parallel into the plurality of data units, and transmitting the plurality of data units to a corresponding plurality of functional modules to process the data units in parallel; and

aggregating results of data processing by the plurality of functional modules.

13. The device as described in claim 10, wherein the processing of the one or more historical data stream includes dividing the one or more historical data streams based on one or more dimensions and performing parallel processing of the one or more historical data streams.

14. The device as described in claim 10, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

15. A computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

dividing the raw data stream into a real-time data stream and one or more historical data streams, comprising;

dividing the real-time data stream into a plurality of data units based at least in part on at least one dimension;

processing the real-time data stream and the one or more historical data streams in parallel, wherein the processing of the real-time data stream comprises;

processing the plurality of data units of the real-time data stream in parallel;

separately generating respective results of the processing of the real-time data stream and the one or more historical data streams; and

integrating the respective generated processing results.

16. The computer program product as described in claim 15, wherein the processing of the real-time data stream is performed based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

17. The computer program product as described in claim 15, wherein the processing of the real-time data stream comprises:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block into the plurality of data units in parallel;

separately transmitting the plurality of data units to a corresponding plurality of functional modules for processing in parallel in response to the dividing of each data block; and

aggregating the results of the processing of the plurality of functional modules.

18. The computer program product as described in claim 15, wherein the processing of the one or more historical data streams includes dividing the one or more historical data streams based on a dimension and the dividing of the one or more historical data streams is performed with the dividing of the real-time data stream in parallel.

19. The computer program product as described in claim 15, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

20. A system, comprising:

a plurality of application servers comprising;

a data recognition module configured to divide a raw data stream into a real-time data stream and one or more historical data streams, comprising to;

divide the real-time data stream into a plurality of data units based at least in part on at least one dimension;

a parallel processing module configured to process the real-time data stream and the one or more historical data streams in parallel, and separately generate respective processing results, wherein the processing of the real-time data stream comprises to;

process the plurality of data units of the real-time data stream in parallel; and

a data integration module configured to integrate the respective generated results of processing.

21. The system as described in claim 20, wherein the parallel processing module divides the real-time data stream based on dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

22. The system as described in claim 20, wherein the parallel processing module comprises:

a horizontal division module configured to divide the real-time data stream into a plurality of data blocks;

a plurality of vertical division modules configured to divide each data block in parallel into the plurality of data units, and transmit the plurality of data units to a corresponding plurality of functional modules to process the data units in parallel; and

a result aggregation module configured to aggregate the results of data processing by the plurality of functional modules.

23. The system as described in claim 20, wherein the parallel processing module divides the one or more historical data streams based on a dimension and performs parallel processing of the one or more historical data streams.

24. The system as described in claim 20, wherein the horizontal division module divides the real-time data stream into the plurality of data blocks based on time.