Optimizing data processing across server clusters and data centers using checkpoint-based data replication

• US 10,296,425 B2
• Filed: 04/20/2017
• Issued: 05/21/2019
• Est. Priority Date: 04/20/2017
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Huang, Tao, Agrawal, Archie, Siripurapu, Sateesh, Zhang, Xianhong
• Primary Examiner(s): Butler, Sarai E
• Application Number:
  US15/492,122
• Publication Number:
  US 20180307572A1
• Time in Patent Office: 761 Days
• Field of Search: 714 12, 714 13
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence

Abstract

Aspects of the disclosure relate to optimizing data processing across server clusters and data centers using checkpoint-based data replication. A computing platform may determine to initiate a data processing job associated with identifying one or more features of a source dataset, and the data processing job may include multiple processing steps. Based on determining to initiate the data processing job, the computing platform may generate one or more commands directing one or more cluster server nodes associated with a data center to execute the multiple processing steps. The one or more commands may direct the one or more cluster server nodes to update a checkpoint table as each processing step is completed, and may further direct the one or more cluster server nodes to replicate processing results data to at least one other data center. Subsequently, the computing platform may send the generated commands to the cluster server nodes.

Claims

1. A computing platform, comprising:

at least one processor;

a communication interface communicatively coupled to the at least one processor; and

memory storing computer-readable instructions that, when executed by the at least one processor, cause the computing platform to;

determine to initiate a data processing job associated with identifying one or more features of a source dataset, the data processing job comprising multiple processing steps;

based on determining to initiate the data processing job, generate one or more first commands directing one or more first cluster server nodes associated with a first data center to execute the multiple processing steps associated with the data processing job to identify the one or more features of the source dataset, the one or more first commands further directing the one or more first cluster server nodes associated with the first data center to update a checkpoint table as each processing step of the multiple processing steps associated with the data processing job is completed, and the one or more first commands further directing the one or more first cluster server nodes associated with the first data center to replicate processing results data to at least one other data center different from the first data center as each processing step of the multiple processing steps associated with the data processing job is completed; and

send, via the communication interface, to the one or more first cluster server nodes associated with the first data center, the one or more first commands.

2. The computing platform of claim 1, wherein determining to initiate the data processing job comprises determining to initiate the data processing job based on receiving user input requesting to initiate the data processing job.

3. The computing platform of claim 1, wherein determining to initiate the data processing job comprises determining to initiate the data processing job based on receiving, via the communication interface, from an administrative computing device, a script defining the data processing job.

4. The computing platform of claim 1, wherein the processing results data is determined by the one or more first cluster server nodes associated with the first data center during execution of each processing step of the multiple processing steps associated with the data processing job, and the processing results data identifies at least one feature of the source dataset as determined by the one or more first cluster server nodes associated with the first data center.

5. The computing platform of claim 1, wherein the checkpoint table is also replicated with the processing results data to the at least one other data center different from the first data center as each processing step of the multiple processing steps associated with the data processing job is completed, and the checkpoint table indicates which processing steps of the multiple processing steps associated with the data processing job have been completed.

6. The computing platform of claim 1, wherein the memory stores additional computer-readable instructions that, when executed by the at least one processor, cause the computing platform to:

detect a failure associated with the data processing job.

7. The computing platform of claim 6, wherein detecting the failure associated with the data processing job comprises detecting that the data processing job has stopped prior to completion.

8. The computing platform of claim 6, wherein detecting the failure associated with the data processing job comprises detecting that the first data center has failed.

9. The computing platform of claim 6, wherein the memory stores additional computer-readable instructions that, when executed by the at least one processor, cause the computing platform to:

based on detecting the failure associated with the data processing job, generate a notification identifying the failure associated with the data processing job; and

send, via the communication interface, to an administrative computing device, the notification identifying the failure associated with the data processing job.

10. The computing platform of claim 9, wherein the memory stores additional computer-readable instructions that, when executed by the at least one processor, cause the computing platform to:

receive, via the communication interface, from the administrative computing device, a response to the notification identifying the failure associated with the data processing job.

11. The computing platform of claim 6, wherein the memory stores additional computer-readable instructions that, when executed by the at least one processor, cause the computing platform to:

determine to resume execution of the data processing job at a second data center of the at least one other data center different from the first data center.

12. The computing platform of claim 11, wherein determining to resume execution of the data processing job at the second data center comprises determining to resume the execution of the data processing job at the second data center based on one or more automatic failover criteria.

13. The computing platform of claim 11, wherein determining to resume execution of the data processing job at the second data center comprises determining to resume the execution of the data processing job at the second data center based on a response to a notification identifying the failure associated with the data processing job received from an administrative computing device.

14. The computing platform of claim 11, wherein the memory stores additional computer-readable instructions that, when executed by the at least one processor, cause the computing platform to:

based on determining to resume execution of the data processing job at the second data center, generate one or more second commands directing one or more second cluster server nodes associated with the second data center to execute at least one uncompleted processing step of the multiple processing steps associated with the data processing job based on the checkpoint table and the processing results data replicated from the first data center; and

send, via the communication interface, to the one or more second cluster server nodes associated with the second data center, the one or more second commands.

15. The computing platform of claim 14, wherein the one or more second commands further direct the one or more second cluster server nodes associated with the second data center to update the checkpoint table as each processing step of the multiple processing steps associated with the data processing job is completed, and the one or more second commands further direct the one or more second cluster server nodes associated with the second data center to replicate processing results data to at least one other data center different from the second data center as each processing step of the multiple processing steps associated with the data processing job is completed.

16. A method, comprising:

at a computing platform comprising at least one processor, memory, and a communication interface;

determining, by the at least one processor, to initiate a data processing job associated with identifying one or more features of a source dataset, the data processing job comprising multiple processing steps;

based on determining to initiate the data processing job, generating, by the at least one processor, one or more first commands directing one or more first cluster server nodes associated with a first data center to execute the multiple processing steps associated with the data processing job to identify the one or more features of the source dataset, the one or more first commands further directing the one or more first cluster server nodes associated with the first data center to update a checkpoint table as each processing step of the multiple processing steps associated with the data processing job is completed, and the one or more first commands further directing the one or more first cluster server nodes associated with the first data center to replicate processing results data to at least one other data center different from the first data center as each processing step of the multiple processing steps associated with the data processing job is completed; and

sending, by the at least one processor, via the communication interface, to the one or more first cluster server nodes associated with the first data center, the one or more first commands.

17. The method of claim 16, wherein determining to initiate the data processing job comprises determining to initiate the data processing job based on receiving user input requesting to initiate the data processing job.

18. The method of claim 16, wherein determining to initiate the data processing job comprises determining to initiate the data processing job based on receiving, via the communication interface, from an administrative computing device, a script defining the data processing job.

19. The method of claim 16, wherein the processing results data is determined by the one or more first cluster server nodes associated with the first data center during execution of each processing step of the multiple processing steps associated with the data processing job, and the processing results data identifies at least one feature of the source dataset as determined by the one or more first cluster server nodes associated with the first data center.

20. One or more non-transitory computer-readable media storing instructions that, when executed by a computing platform comprising at least one processor, memory, and a communication interface, cause the computing platform to:

determine to initiate a data processing job associated with identifying one or more features of a source dataset, the data processing job comprising multiple processing steps;

based on determining to initiate the data processing job, generate one or more first commands directing one or more first cluster server nodes associated with a first data center to execute the multiple processing steps associated with the data processing job to identify the one or more features of the source dataset, the one or more first commands further directing the one or more first cluster server nodes associated with the first data center to update a checkpoint table as each processing step of the multiple processing steps associated with the data processing job is completed, and the one or more first commands further directing the one or more first cluster server nodes associated with the first data center to replicate processing results data to at least one other data center different from the first data center as each processing step of the multiple processing steps associated with the data processing job is completed; and

send, via the communication interface, to the one or more first cluster server nodes associated with the first data center, the one or more first commands.