Efficient data backflow processing for data warehouse

• US 8,374,995 B2
• Filed: 11/22/2010
• Issued: 02/12/2013
• Est. Priority Date: 11/24/2009
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Li, Xuesheng
• Primary Examiner(s): Hu, Jensen
• Application Number:
  US12/951,464
• Publication Number:
  US 20110125707A1
• Time in Patent Office: 813 Days
• Field of Search: 707/602
• US Class Current: 707/602
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Processing backflow data includes obtaining backflow data from a data warehouse, determining, according to a backflow rule for the backflow data, a respective destination table in a production system for each data item in the backflow data, and sending the data item according to the determined respective destination table in the production system.

Claims

1. A method for processing backflow data, comprising:

obtaining backflow data from a data warehouse;

determining, according to a backflow rule for the backflow data, a respective destination table in a production system for each data item in the backflow data; and

sending the data item according to the determined respective destination table in the production system, wherein sending the data item includes grouping backflow data to the determined destination tables in the production system, wherein data items in the backflow data associated with the same destination table are included in the same group, and sending the backflow data according to the grouping using a plurality of threads, wherein data items in each group are sent to the same destination table in the production system by the same thread;

wherein sending the data item according to the determined respective destination table in the production system further comprises distributing groups of backflow data to the plurality of threads based at least in part on an amount of data associated with at least one of the groups of backflow data.

2. The method of claim 1, wherein obtaining the backflow data from the data warehouse comprises extracting backflow data from the data warehouse into a memory concurrently by using a plurality of threads.

3. The method of claim 1, wherein the backflow rule is determined according to the number of destination tables in the production system and an attribute of the backflow data.

4. The method of claim 3, wherein the attribute of the backflow data comprises a value of a field in the data item.

5. The method of claim 4, wherein the backflow rule includes applying a hash function to the value of the field in the data item.

6. A backflow processing system, comprising:

a processor configured to;

obtain backflow data from a data warehouse;

determine, according to a backflow rule for the backflow data, a respective destination table in a production system for each data item in the backflow data; and

send the data item according to the determined respective destination table in the production system, wherein sending the data item includes grouping backflow data to the determined destination tables in the production system, wherein data items in the backflow data associated with the same destination table are included in the same group, and sending the backflow data according to the grouping using a plurality of threads, wherein data items in each group are sent to the same destination table in the production system by the same threat

wherein sending the data item according to the determined respective destination table in the production system further comprises distributing groups of backflow data to the plurality of threads based at least in part on an amount of data associated with at least one of the groups of backflow data; and

a memory coupled to the processor, configured to provide the processor with instructions.

7. The system of claim 6, wherein obtaining the backflow data from the data warehouse comprises extracting backflow data from the data warehouse into a memory concurrently by using a plurality of threads.

8. The system of claim 6, wherein the backflow rule is determined according to the number of destination tables in the production system and an attribute of the backflow data.

9. The system of claim 8, wherein the attribute of the backflow data comprises a value of a field in the data item.

10. The system of claim 9, wherein the backflow rule includes applying a hash function to the value of the field in the data item.