EFFICIENT DATA BACKFLOW PROCESSING FOR DATA WAREHOUSE

• US 20110125707A1
• Filed: 11/22/2010
• Published: 05/26/2011
• Est. Priority Date: 11/24/2009
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Li, Xuesheng
• Granted Patent: US 8,374,995 B2
• Time in Patent Office: Days
• US Class Current: 707/602
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Processing backflow data includes obtaining backflow data from a data warehouse, determining, according to a backflow rule for the backflow data, a respective destination table in a production system for each data item in the backflow data, and sending the data item according to the determined respective destination table in the production system.

Claims

1. A method for processing backflow data, comprising:

obtaining backflow data from a data warehouse;

determining, according to a backflow rule for the backflow data, a respective destination table in a production system for each data item in the backflow data; and

sending the data item according to the determined respective destination table in the production system.

2. The method of claim 1, wherein obtaining the backflow data from the data warehouse comprises extracting backflow data from the data warehouse into a memory concurrently by using a plurality of threads.

3. The method of claim 1, wherein sending the data item according to the determined respective destination table in the production system comprises:

grouping backflow data to the determined destination tables in the production system; and

sending the backflow data according to the grouping using a plurality of threads;

wherein data items in each group are sent to the same destination table in the production system by the same thread.

4. The method of claim 3, wherein sending the data item according to the determined respective destination table in the production system further comprises distributing groups of backflow data to the plurality of threads.

5. The method of claim 1, wherein the backflow rule is determined according to the number of destination tables in the production system and an attribute of the backflow data.

6. The method of claim 5, wherein the attribute of the backflow data comprises a value of a field in the data item.

7. The method of claim 6, wherein the backflow rule includes applying a hash function to the value of the field in the data item.

8. A backflow processing system, comprising:

a processor configured to;

obtain backflow data from a data warehouse;

determine, according to a backflow rule for the backflow data, a respective destination table in a production system for each data item in the backflow data; and

send the data item according to the determined respective destination table in the production system; and

a memory coupled to the processor, configured to provide the processor with instructions.

9. The system of claim 8, wherein obtaining the backflow data from the data warehouse comprises extracting backflow data from the data warehouse into a memory concurrently by using a plurality of threads.

10. The system of claim 8, wherein sending the data item according to the determined respective destination table in the production system comprises:

grouping backflow data to the determined destination tables in the production system; and

sending the backflow data according to the grouping using a plurality of threads;

wherein data items in each group are sent to the same destination table in the production system by the same thread.

11. The system of claim 10, wherein sending the data item according to the determined respective destination table in the production system further comprises distributing groups of backflow data to the plurality of threads.

12. The system of claim 8, wherein the backflow rule is determined according to the number of destination tables in the production system and an attribute of the backflow data.

13. The system of claim 12, wherein the attribute of the backflow data comprises a value of a field in the data item.

14. The system of claim 13, wherein the backflow rule includes applying a hash function to the value of the field in the data item.