Extracting Incremental Data

• US 20130073516A1
• Filed: 06/22/2012
• Published: 03/21/2013
• Est. Priority Date: 06/23/2011
• Status: Abandoned Application

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Fan, Xin
• Application Number:
  US13/574,162
• Publication Number:
  US 20130073516A1
• Time in Patent Office: Days
• US Class Current: 707/602
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation

Abstract

The present disclosure introduces a method, an apparatus, and a system for extracting incremental data. Primary key information of incremental data is obtained from a backup database. The incremental data is inquired based on the primary key information from a main database that synchronizes with the backup database. The found incremental data is then inserted into a target data warehouse. The present techniques not only save a lot of time and system resources but also improve the efficiency of incremental data extraction.

Claims

1. A method performed by one or more processors configured with computer-executable instructions, the method comprising:

obtaining primary key information of incremental data from a backup database;

inquiring incremental data at a main database, based on the obtained primary key information, synchronized between the main database and the backup database; and

inserting found incremental data into a target data warehouse.

2. The method as recited in claim 1, wherein the data synchronized between the main database and the backup database includes one or more key items of the data without inclusion of all items of the data, the one or more key items including primary key information of the data.

3. The method as recited in claim 1, wherein the backup database is a backup database of a front-end website and the main database is a main database of the front-end website.

4. The method as recited in claim 1, wherein the obtaining comprises:

parsing a log file of the backup database to obtain parsed contents;

based on the parsed contents in the log file of the backup database, inversely parsing changed data in the backup database; and

retrieving the primary key information of the changed data from the backup database.

5. The method as recited in claim 1, wherein the inquiring comprises using a search function or search instruction to inquire one or more entire pieces of incremental data from the main database based on the obtained primary key information.

6. The method as recited in claim 5, wherein each of the one or more entire pieces of incremental data includes:

a type of change of the incremental data;

a time of change of the incremental data; and

the primary key information of the incremental data.

7. The method as recited in claim 1, further comprising obtaining a type of change of the incremental data.

8. The method as recited in claim 7, wherein the type of change includes at least one of the following:

insertion arising from an insertion operation;

updating arising from an updating operation;

deletion arising from a deletion operation.

9. The method as recited in claim 1, wherein the inserting comprises merging the incremental data with an original data table at the target data warehouse.

10. An apparatus comprising:

one or more processors; and

computer storage media having stored thereon computer-executable instructions that are executable by the one or more processors to perform actions comprising;

obtaining primary key information of incremental data from a backup database, the obtaining including;

parsing a log file of the backup database;

based on the parsed contents in the log file of the backup database, inversely parsing changed data in the backup database; and

retrieving the primary key information of the changed data from the backup database;

inquiring incremental data at a main database, based on the obtained primary key information, synchronized between the main database and the backup database; and

inserting found incremental data into a target data warehouse.

11. The apparatus as recited in claim 10, wherein the inquiring comprises using a search function or search instruction to inquire one or more entire pieces of incremental data from the main database based on the obtained primary key information.

12. The apparatus as recited in claim 11, wherein the found one or more entire pieces of incremental data includes:

a type of change of the incremental data;

a time of change of the incremental data; and

the primary key information of the incremental data.

13. The apparatus as recited in claim 12, wherein the type of change includes at least one of the following:

insertion arising from an insertion operation;

updating arising from an updating operation;

deletion arising from a deletion operation.

14. The apparatus as recited in claim 10, wherein the inquiring comprises:

comparing found one or more entire pieces of incremental data with an original table at the target data warehouse; and

updating the found one or more entire pieces of incremental data into the original table based on a result of the comparing.

15. The apparatus as recited in claim 10, wherein the data synchronized between the main database and the backup database includes one or more key items of the data without inclusion of all items of the data, the one or more key items including primary key information of the data.

16. The apparatus as recited in claim 10, wherein the backup database is a backup database of a front-end website and the main database is a main database of the front-end website.

17. A system comprising:

a main database;

a backup database;

a target warehouse; and

an apparatus including;

one or more processors; and

computer storage media having stored thereon computer-executable instructions that are executable by the one or more processors to perform actions comprising;

obtaining primary key information of incremental data from a backup database, the obtaining including;

parsing a log file of the backup database;

based on the parsed contents in the log file of the backup database, inversely parsing changed data in the backup database; and

retrieving the primary key information of the changed data from the backup database;

inquiring one or more entire pieces of incremental data at a main database, based on the obtained primary key information, synchronized between the main database and the backup database; and

inserting found one or more entire pieces of incremental data into a target data warehouse.

18. The system as recited in claim 17, wherein the data synchronized between the main database and the backup database includes one or more key items of the data without inclusion of all items of the data, the one or more key items including primary key information of the data.

19. The system as recited in claim 17, wherein the one or more entire pieces of incremental data includes:

a type of change of the incremental data;

a time of change of the incremental data; and

the primary key information of the incremental data.

20. The system as recited in claim 19, wherein the type of change includes at least one of the following:

insertion arising from an insertion operation;

updating arising from an updating operation;

deletion arising from a deletion operation.