Distributed data stream processing method and system

• US 9,727,613 B2
• Filed: 12/21/2015
• Issued: 08/08/2017
• Est. Priority Date: 11/24/2011
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Yang, Zhixiong, Zhang, Xu, Xu, Jia, Deng, Zhonghua
• Primary Examiner(s): KIM, DONG U
• Application Number:
  US14/977,484
• Publication Number:
  US 20160179898A1
• Time in Patent Office: 596 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

Embodiments of the present application relate to a distributed data stream processing method, a distributed data stream processing device, a computer program product for processing a raw data stream and a distributed data stream processing system. A distributed data stream processing method is provided. The method includes dividing a raw data stream into a real-time data stream and historical data streams, processing the real-time data stream and the historical data streams in parallel, separately generating respective results of the processing of the real-time data stream and the historical data streams, and integrating the generated processing results.

Claims

1. A method, comprising:

determining a number of a plurality of division modules based on flow volume of a raw data stream;

dividing the raw data stream into a real-time data stream and one or more historical data streams based on the number of the plurality of division modules;

processing the real-time data stream and the one or more historical data streams in parallel, comprising;

dividing the real-time data stream into a plurality of data units based on a plurality of dimensions, the plurality of dimensions including a first dimension and a second dimension, wherein a first data unit is associated with the first dimension, and wherein a second data unit is associated with the second dimension;

determining a number of a plurality of functional modules within a functional module group and a number of a plurality of functional module groups based on a number of the plurality of data units and resources available to be used to process the plurality of data units;

processing, via the plurality of functional modules, the first data unit and the second data unit in parallel, the plurality of functional modules including a first functional module and a second functional module, wherein the first data unit is processed by the first functional module, and wherein the second data unit is processed by the second functional module; and

aggregating results of the processing performed by the first functional module and the second functional module;

separately generating respective results of the processing of the real-time data stream and the one or more historical data streams; and

integrating the respective generated processing results.

2. The method as described in claim 1, wherein the processing of the real-time data stream is performed based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

3. The method as described in claim 1, wherein the processing of the real-time data stream comprises:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block into the plurality of data units; and

aggregating the results of the processing of the plurality of data units.

4. The method as described in claim 3, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

5. The method as described in claim 1, wherein the processing of the one or more historical data streams includes dividing the one or more historical data streams based on one or more dimensions and the dividing of the one or more historical data streams is performed with the dividing of the real-time data stream in parallel.

6. The method as described in claim 1, wherein the real-time data stream and the one or more historical data streams are processed separately.

7. The method as described in claim 1, wherein a dimension is associated with a behavior of a user.

8. The method as described in claim 1, wherein different preferences of a user correspond to different dimensions, and data units associated with different dimensions are processed separately.

9. A device, comprising:

a processor; and

a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to;

determine a number of a plurality of division modules based on flow volume of a raw data stream;

divide the raw data stream into a real-time data stream and one or more historical data streams based on the number of the plurality of division modules;

process the real-time data stream and the one or more historical data streams in parallel and separately generate respective processing results, comprising to;

divide the real-time data stream into a plurality of data units based on a plurality of dimensions, the plurality of dimensions including a first dimension and a second dimension, wherein a first data unit is associated with the first dimension, and wherein a second data unit is associated with the second dimension;

determine a number of a plurality of functional modules within a functional module group and a number of a plurality of functional module groups based on a number of the plurality of data units and resources available to be used to process the plurality of data units;

process, via the plurality of functional modules, the first data unit and the second data unit in parallel, the plurality of functional modules including a first functional module and a second functional module, wherein the first data unit is processed by the first functional module, and wherein the second data unit is processed by the second functional module; and

aggregate results of the processing performed by the first functional module and the second functional module; and

integrate the respective generated processing results.

10. The device as described in claim 9, wherein the processing of the real-time data stream includes dividing the real-time data stream based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

11. The device as described in claim 9, wherein the processing of the real-time data stream includes:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block in parallel into the plurality of data units, and transmitting the plurality of data units to a corresponding plurality of functional modules to process the data units in parallel; and

aggregating results of data processing by the plurality of functional modules.

12. The device as described in claim 11, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

13. The device as described in claim 9, wherein the processing of the one or more historical data stream includes dividing the one or more historical data streams based on one or more dimensions and performing parallel processing of the one or more historical data streams.

14. A computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

determining a number of a plurality of division modules based on flow volume of a raw data stream;

dividing the raw data stream into a real-time data stream and one or more historical data streams based on the number of the plurality of division modules;

processing the real-time data stream and the one or more historical data streams in parallel, comprising;

dividing the real-time data stream into a plurality of data units based on a plurality of dimensions, the plurality of dimensions including a first dimension and a second dimension, wherein a first data unit is associated with the first dimension, and wherein a second data unit is associated with the second dimension;

determining a number of a plurality of functional modules within a functional module group and a number of a plurality of functional module groups based on a number of the plurality of data units and resources available to be used to process the plurality of data units;

processing, via the plurality of functional modules, the first data unit and the second data unit in parallel, the plurality of functional modules including a first functional module and a second functional module, wherein the first data unit is processed by the first functional module, and wherein the second data unit is processed by the second functional module; and

aggregating results of the processing performed by the first functional module and the second functional module; and

integrating the respective generated processing results.

15. The computer program product as described in claim 14, wherein the processing of the real-time data stream is performed based on a dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

16. The computer program product as described in claim 14, wherein the processing of the real-time data stream comprises:

dividing the real-time data stream into a plurality of data blocks;

dividing each data block into the plurality of data units in parallel;

separately transmitting the plurality of data units to a corresponding plurality of functional modules for processing in parallel in response to the dividing of each data block; and

aggregating the results of the processing of the plurality of functional modules.

17. The computer program product as described in claim 16, wherein the dividing of the real-time data stream into the plurality of data blocks is based on time.

18. The computer program product as described in claim 14, wherein the processing of the one or more historical data streams includes dividing the one or more historical data streams based on a dimension and the dividing of the one or more historical data streams is performed with the dividing of the real-time data stream in parallel.

19. A system, comprising:

a plurality of application servers comprising;

a determination module configured to determine a number of a plurality of division modules based on flow volume of a raw data stream;

a data recognition module configured to divide the raw data stream into a real-time data stream and one or more historical data streams based on the number of the plurality of division modules;

a parallel processing module configured to process the real-time data stream and the one or more historical data streams in parallel, and separately generate respective processing results, comprises to;

divide the real-time data stream into a plurality of data units based on a plurality of dimensions, the plurality of dimensions including a first dimension and a second dimension, wherein a first data unit is associated with the first dimension, and wherein a second data unit is associated with the second dimension;

determine a number of a plurality of functional modules within a functional module group and a number of a plurality of functional module groups based on a number of the plurality of data units and resources available to be used to process the plurality of data units;

process, via the plurality of functional modules, the first data unit and the second data unit in parallel, the plurality of functional modules including a first functional module and a second functional module, wherein the first data unit is processed by the first functional module, and wherein the second data unit is processed by the second functional module; and

aggregate results of the processing performed by the first functional module and the second functional module; and

a data integration module configured to integrate the respective generated results of processing.

20. The system as described in claim 19, wherein the parallel processing module divides the real-time data stream based on dimension, and wherein one data block of the real-time data stream is processed in parallel with another data block of the real-time data stream.

21. The system as described in claim 19, wherein the parallel processing module comprises:

a horizontal division module configured to divide the real-time data stream into a plurality of data blocks;

a plurality of vertical division modules configured to divide each data block in parallel into the plurality of data units, and transmit the plurality of data units to a corresponding plurality of functional modules to process the data units in parallel; and

a result aggregation module configured to aggregate the results of data processing by the plurality of functional modules.

22. The system as described in claim 21, wherein the horizontal division module divides the real-time data stream into the plurality of data blocks based on time.

23. The system as described in claim 19, wherein the parallel processing module divides the one or more historical data streams based on a dimension and performs parallel processing of the one or more historical data streams.