System and method for incrementally replicating investigative analysis data

• US 8,886,601 B1
• Filed: 06/20/2013
• Issued: 11/11/2014
• Est. Priority Date: 06/20/2013
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Landau, Alexander, Ducott, Richard Allen III, Wilson, Tim
• Primary Examiner(s): Young, Kevin L
• Application Number:
  US13/922,437
• Time in Patent Office: 509 Days
• Field of Search: None
• US Class Current: 707/625
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation

Abstract

A method of incrementally replicating investigative analysis data is disclosed along with a system for performing the same. The method and system provide the ability to break a data replication job into multiple “replication chunks” which can be exported and imported separately. By doing so, the method and system can efficiently replicate large numbers of data object changes over an unreliable data network.

Claims

1. A method comprising:

receiving, at an exporting system, a command to execute an incremental replication job;

wherein the exporting system tracks changes to database data in a revisioning database by creating, in the revisioning database, change records for changes to the database data, the change records comprising logical clock values reflecting an order of the changes, the logical clock values obtained from a logical clock used to order changes to the revisioning database;

based on the command, the exporting system creating and storing an incremental replication plan for the incremental replication job;

wherein creating and storing the incremental replication plan comprises;

determining a number of incremental replication chunks to divide the incremental replication job into,

determining a current logical clock value of the logical clock, and

storing the current logical clock value as part of the incremental replication plan;

wherein the stored incremental replication plan comprises a number of incremental replication chunk specifications, one for each of the number of incremental replication chunks that the incremental replication job is divided into;

wherein each of the number of incremental replication chunk specifications comprises information that identifies a range of data objects that are to be exported as part of the incremental replication chunk;

after creating and storing the incremental replication plan, the exporting system executing the incremental replication job according to the incremental replication plan;

wherein executing the incremental replication job comprises, for each of the number of incremental replication chunk specifications;

collecting, from the revisioning database, based on the information in the incremental replication chunk specification and based on the logical clock value stored as part of the incremental replication plan, change data from one or more change records for the data objects to be exported as part of the incremental replication chunk, and

exporting the collected change data to an importing system;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the command specifies the number of replication chunks to divide the replication job into.

3. The method of claim 1, wherein the command specifies a number of data objects per replication chunk for the incremental replication job;

and wherein determining the number of incremental replication chunks to divide the incremental replication job into is based on the specified number of data objects per replication chunk.

4. The method of claim 1, wherein the command specifies a time at which to start execution of the incremental replication job;

and wherein executing the incremental replication job comprises starting execution of the incremental replication job at the specified time.

5. The method of claim 1, wherein the command is received via a command line interface.

6. The method of claim 1, wherein executing the incremental replication job comprises, for at least one of the number of incremental replication chunk specifications, querying the revisioning database, based on the information in the at least one incremental replication chunk specification and based on the logical clock value stored as part of the incremental replication plan, for change records for the data objects to be exported as part of the at least one incremental replication chunk that are associated with a logical clock value that is less than or equal to the logical clock value stored as part of the incremental replication plan.

7. The method of claim 1, further comprising:

after creating and storing the incremental replication plan and before executing the incremental replication job, updating in the revisioning database a particular data object that, according to a particular incremental replication chunk specification, is to be exported as part of the particular incremental replication chunk.

8. The method of claim 7, wherein change data resulting from the updating of the particular data object is not provided to the importing system as part of executing the incremental replication job.

9. The method of claim 7, wherein change data resulting from the updating of the particular data object is associated in the revisioning database with a logical clock value greater than the logical clock value stored as part of the incremental replication plan.

10. An exporting system comprising:

one or more processors;

memory storing instructions which, when executed by the one or more processors, cause performance of a method comprising;

receiving, at the exporting system, a command to execute an incremental replication job;

wherein the exporting system tracks changes to database data in a revisioning database by creating, in the revisioning database, change records for changes to the database data, the change records comprising logical clock values reflecting an order of the changes, the logical clock values obtained from a logical clock used to order changes to the revisioning database;

based on the command, the exporting system creating and storing an incremental replication plan for the incremental replication job;

wherein creating and storing the incremental replication plan comprises;

determining a number of incremental replication chunks to divide the incremental replication job into,

determining a current logical clock value of the logical clock, and

storing the current logical clock value as part of the incremental replication plan;

wherein the stored incremental replication plan comprises a number of incremental replication chunk specifications, one for each of the number of incremental replication chunks that the incremental replication job is divided into;

wherein each of the number of incremental replication chunk specifications comprises information that identifies a range of data objects that are to be exported as part of the incremental replication chunk;

after creating and storing the incremental replication plan, the exporting system executing the incremental replication job according to the incremental replication plan;

wherein executing the incremental replication job comprises, for each of the number of incremental replication chunk specifications;

collecting, from the revisioning database, based on the information in the incremental replication chunk specification and based on the logical clock value stored as part of the incremental replication plan, change data from one or more change records for the data objects to be exported as part of the incremental replication chunk, and

exporting the collected change data to an importing system.

11. The system of claim 10, wherein the command specifies the number of replication chunks to divide the replication job into.

12. The system of claim 10, wherein the command specifies a number of data objects per replication chunk for the incremental replication job;

and wherein determining the number of incremental replication chunks to divide the incremental replication job into is based on the specified number of data objects per replication chunk.

13. The system of claim 10, wherein the command specifies a time at which to start execution of the incremental replication job;

and wherein executing the incremental replication job comprises starting execution of the incremental replication job at the specified time.

14. The system of claim 10, wherein the command is received via a command line interface.

15. The system of claim 10, wherein executing the incremental replication job comprises, for at least one of the number of incremental replication chunk specifications, querying the revisioning database, based on the information in the at least one incremental replication chunk specification and based on the logical clock value stored as part of the incremental replication plan, for change records for the data objects to be exported as part of the at least one incremental replication chunk that are associated with a logical clock value that is less than or equal to the logical clock value stored as part of the incremental replication plan.

16. The system of claim 10, the method further comprising:

after creating and storing the incremental replication plan and before executing the incremental replication job, updating in the revisioning database a particular data object that, according to a particular incremental replication chunk specification, is to be exported as part of the particular incremental replication chunk.

17. The system of claim 16, wherein change data resulting from the updating of the particular data object is not provided to the importing system as part of executing the incremental replication job.

18. The system of claim 16, wherein change data resulting from the updating of the particular data object is associated in the revisioning database with a logical clock value greater than the logical clock value stored as part of the incremental replication plan.