Systems and methods for adaptive data replication

• US 10,068,002 B1
• Filed: 07/31/2017
• Issued: 09/04/2018
• Est. Priority Date: 04/25/2017
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Wilczynski, Peter, Zheng, Ryan, Carrino, John, Wilson, Timothy, Freiberg, Stephen, Garrod, John, Waldrep, William
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US15/664,184
• Time in Patent Office: 400 Days
• Field of Search: 707600-899
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods are provided for adaptive data replication. A first data object is stored. One or more first network characteristics of a communication network are detected. A first replication mode is triggered (e.g., a single-master replication mode) based on the one or more first network characteristics of the communication network. The single-master replication may be configured to facilitate real-time synchronizations (e.g., for critical updates). One or more second network characteristics of the communication network are detected. A second replication mode (e.g., a multi-master replication mode) is triggered based on the one or more second network characteristics. The second replication may be configured to facilitate a delayed synchronization (e.g., for non-critical updates) of the first data object and the second data object using the second update file.

Claims

1. A system comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to perform;

storing a first data object, the first data object being associated with a corresponding second data object stored by a remote master replication system;

detecting one or more first network characteristics of a communication network;

triggering a first replication mode based on the one or more first network characteristics of the communication network, the first replication mode comprising a single-master replication mode, the first replication mode including;

receiving a first update of the first data object;

generating a first update file in response to receiving the first update of the first data object, the first update file indicating the first update of the first data object; and

providing the first update file to the remote master replication system over the communication network, the remote master replication system being configured to facilitate a real-time synchronization of the first data object and the second data object using the first update file;

detecting one or more second network characteristics of the communication network; and

triggering a second replication mode based on the one or more second network characteristics, the second replication mode comprising a multi-master replication mode, the second replication mode including;

receiving a second update of the first data object;

generating a second update file, the second update file indicating the second update of the first data object;

providing the second update file to a second replication system, the second replication system comprising a multi-master replication system, the second replication system facilitating a delayed synchronization of the first data object and the second data object using the second update file;

providing, by the second replication system, the second update file to a remote multi-master replication system;

generating, by the remote multi-master replication system, a third update file, the third update file indicating the second update of the first data object;

providing, by the remote multi-master replication system, the third update file to the remote master replication system; and

updating, by the remote master replication system, the second data object using the third update file.

2. The system of claim 1, wherein the one or more first network characteristics comprise any of latency, bandwidth, throughput, or network connection status.

3. The system of claim 1, wherein the second replication mode further includes:

generating an alert based on any of the one or more second network characteristics, or the second replication mode; and

presenting the alert to a user of the system.

4. The system of claim 1, wherein the instructions further cause the system to perform:

determining a priority status of at least a portion of the first update of the first data object; and

wherein the first replication mode is triggered based on the priority of status of the at least a portion of the first update of the first data object.

5. The system of claim 1, wherein the second replication mode further includes:

obtaining, by the second replication system, one or more third data objects, the one or more third data objects being associated with corresponding one or more fourth data objects stored by the remote multi-master replication system, the second update file further indicating the one or more third data objects; and

synchronizing, by the remote multi-master replication system, the one or more third data objects and the one or more fourth data objects using the second update file.

6. The system of claim 1, wherein the third and fourth data objects comprise immutable data objects and the first and second data objects comprises mutable data objects.

7. The system of claim 1, wherein the third and fourth data objects represent a geographic map, and the first and second data objects represent a point of interest of the geographic map.

8. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:

storing a first data object, the first data object being associated with a corresponding second data object stored by a remote master replication system;

detecting one or more first network characteristics of a communication network;

triggering a first replication mode based on the one or more first network characteristics of the communication network, the first replication mode comprising a single-master replication mode, the first replication mode including;

receiving a first update of the first data object;

generating a first update file in response to receiving the first update of the first data object, the first update file indicating the first update of the first data object; and

providing the first update file to the remote master replication system over the communication network, the remote master replication system being configured to facilitate a real-time synchronization of the first data object and the second data object using the first update file;

detecting one or more second network characteristics of the communication network; and

triggering a second replication mode based on the one or more second network characteristics, the second replication mode comprising a multi-master replication mode, the second replication mode including;

receiving a second update of the first data object;

generating a second update file, the second update file indicating the second update of the first data object;

providing the second update file to a second replication system, the second replication system comprising a multi-master replication system, the second replication system facilitating a delayed synchronization of the first data object and the second data object using the second update file;

providing, by the second replication system, the second update file to a remote multi-master replication system;

generating, by the remote multi-master replication system, a third update file, the third update file indicating the second update of the first data object;

providing, by the remote multi-master replication system, the third update file to the remote master replication system; and

updating, by the remote master replication system, the second data object using the third update file.

9. The method of claim 8, wherein the one or more first network characteristics comprise any of latency, bandwidth, throughput, or network connection status.

10. The method of claim 8, wherein the second replication mode further includes:

generating an alert based on any of the one or more second network characteristics, or the second replication mode; and

presenting the alert to a user of the system.

11. The method of claim 8, further comprising:

determining a priority status of at least a portion of the first update of the first data object; and

wherein the first replication mode is triggered based on the priority of status of the at least a portion of the first update of the first data object.

12. The method of claim 8, further comprising:

obtaining, by the second replication system, one or more third data objects, the one or more third data objects being associated with corresponding one or more fourth data objects stored by a remote multi-master replication system, the second update file further indicating the one or more third data objects; and

synchronizing, by the remote multi-master replication system, the one or more third data objects and the one or more fourth data objects using the second update file.

13. The method of claim 8, wherein the third and fourth data objects comprise immutable data objects and the first and second data objects comprises mutable data objects.

14. The method of claim 8, wherein the third and fourth data objects represent a geographic map, and the first and second data objects represent a point of interest of the geographic map.

15. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:

storing a first data object, the first data object being associated with a corresponding second data object stored by a remote master replication system;

detecting one or more first network characteristics of a communication network;

triggering a first replication mode based on the one or more first network characteristics of the communication network, the first replication mode comprising a single-master replication mode, the first replication mode including;

receiving a first update of the first data object;

generating a first update file in response to receiving the first update of the first data object, the first update file indicating the first update of the first data object; and

providing the first update file to the remote master replication system over the communication network, the remote master replication system being configured to facilitate a real-time synchronization of the first data object and the second data object using the first update file;

detecting one or more second network characteristics of the communication network; and

triggering a second replication mode based on the one or more second network characteristics, the second replication mode comprising a multi-master replication mode, the second replication mode including;

receiving a second update of the first data object;

generating a second update file, the second update file indicating the second update of the first data object;

providing the second update file to a second replication system, the second replication system comprising a multi-master replication system, the second replication system facilitating a delayed synchronization of the first data object and the second data object using the second update file;

providing, by the second replication system, the second update file to a remote multi-master replication system;

generating, by the remote multi-master replication system, a third update file, the third update file indicating the second update of the first data object;

providing, by the remote multi-master replication system, the third update file to the remote master replication system; and

updating, by the remote master replication system, the second data object using the third update file.