Multi-cloud bi-directional storage replication system and techniques

• US 10,635,642 B1
• Filed: 05/09/2019
• Issued: 04/28/2020
• Est. Priority Date: 05/09/2019
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation), Insulet Corporation
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Haggerty, Timothy, Long, Steven, Rao, Deepa, Henry, Eric, Zhou, Yuting
• Primary Examiner(s): Pyzocha, Michael
• Application Number:
  US16/408,425
• Time in Patent Office: 355 Days
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/1794 Details of file format conversion
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/105 Multiple levels of security
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Disclosed are systems, apparatuses and techniques for replicating data between different cloud computing platforms. Examples include storage replicator components operable in different cloud computing platforms. The first storage replicator component may identify the second cloud computing platform as a location to copy a data file in response to an event related to the data file stored in a first cloud computing platform. The first storage replicator component may request a copy of the data file via an application programming interface of the first cloud computing platform. The attributes of the copy of the data file which involve modification to conform to data management conventions of the second cloud computing platform may be determined and modified to comply with conventions of the second cloud computing platform. The modified copy of the data file may be forwarded to the second cloud computing platform for storage.

Claims

1. A method, comprising:

in response to an event related to a data file that is one of a plurality of data files stored in a cloud computing data storage of a first cloud computing platform, identifying, by a first storage replicator component in the first cloud computing platform, a second cloud computing platform to receive replicated data, wherein the event indicates a change to an attribute of the data file;

requesting, by the first storage replicator component via an application programming interface of the first cloud computing platform, a copy of the data file;

determining which attributes of the copy of the data file involve modification to conform to data management conventions of the second cloud computing platform by;

accessing a rules engine;

providing an identifier of the second cloud computing platform to the rules engine;

in response to providing the identifier of the second cloud computing platform, retrieving attribute rules related to the second cloud computing platform; and

determining which attributes of copy of the data file involve modification to conform to the attribute rules related to the second cloud computing platform;

while copying the data file, modifying the determined attributes of the copy of the data file to comply with the conventions of the second cloud computing platform;

encrypting the modified copy of the data file; and

forwarding the encrypted, modified copy of the data file to the second cloud computing platform for storage.

2. The method of claim 1, wherein attributes of the copy of the data file include at least one of content of the data file, a filename of the data file, data file metadata format or a size of the data file.

3. The method of claim 1, wherein an event is one of when a file is created, a file is updated, a file is modified, a file is sent to a bucket for storage, an indication of a need to load balance between the first cloud computing platform and the second cloud computing platform.

4. The method of claim 1, further comprising:

obtain, by the first storage replicator component, authentication credentials of a customer associated with a customer network platform; and

authenticate the customer as being permitted to copy the data file in the first cloud computing platform based on the obtained authentication credentials.

5. The method of claim 1, further comprising:

in response to the event being of a particular event type, selecting the second cloud computing platform from a plurality of cloud computing platforms coupled to a customer network platform.

6. The method of claim 1, further comprising:

in response to a status of the data file related to the event, selecting the second cloud computing platform from a plurality of cloud computing platforms coupled to a customer network platform.

7. The method of claim 1, further comprising:

detecting the event related to the data file.

8. The method of claim 1, further comprising:

receiving, by a second storage replicator component executing in the second cloud computing platform, the encrypted modified data file forwarded from the first cloud computing platform;

authenticating the second storage replicator component via an application programming interface of the second cloud computing platform;

decrypting the encrypted modified copy of the data file forwarded from the first cloud computing platform; and

storing the modified data file in data storage of the second cloud computing platform.

9. A system, comprising:

a customer network platform comprising a processor and a memory, wherein the customer network platform is operable to perform a number of functions for a customer entity; and

a plurality of cloud computing platforms coupled to the customer network platform and each cloud computing platform comprising a cloud computing data storage, wherein;

each respective cloud computing platform of the plurality of cloud computing platforms is operable to perform functions via one or more respective application programming interfaces of each respective cloud computing platform,

each respective cloud computing platform includes a respective storage replicator component within the respective cloud computing platform operating as a serverless function operable to provide data replication services,

each respective cloud computing platform has a same customer and each respective storage replicator component is associated with the customer, and

wherein a first respective storage replicator component in a first respective cloud computing platform of the plurality of cloud computing platforms is operable to perform functions, including functions to;

in response to an event related to a data file which is one of a plurality of data files stored in a data storage of the first respective cloud computing platform, access a file copy application programming interface provided by the first respective cloud computing platform;

determine that the data file is to be replicated to a second respective cloud computing platform;

utilize a rules engine to determine which attributes of the data file require modification to comply with data conventions of the second respective cloud computing platform;

based on an output of the rules engine, modify the data file to conform with the data conventions of the second respective cloud computing platform; and

forward the modified data file from the first respective cloud computing platform to the second respective cloud computing platform.

10. The system of claim 9, wherein upon accessing the file copy application programming interface, the first respective storage replicator component is further operable to perform functions, including functions to:

obtain authentication credentials of the customer associated with the customer network platform; and

authenticate the customer based on the obtained authentication credentials.

11. The system of claim 9, wherein the first respective storage replicator component is further operable to perform functions, including functions to:

determine a size of the data file;

in response to the determined size of the data file exceeding a predetermined size threshold, instantiating a dividing container operable to divide the data file into chunks of data;

instantiating a respective copy container is assigned a respective chunk of the chunks of data, wherein each respective copy container is operable to replicate an assigned respective chunk of data;

modify, by each respective copy container, the respective chunk of data assigned to the respective copy container; and

provide the modified respective chunk of data to the first respective storage replicator component to be forwarded to the second respective cloud computing platform.

12. The system of claim 9, wherein a second respective storage replicator component in the second respective cloud computing platform of the plurality of cloud computing platform is operable to perform functions, including functions to:

receive the modified data file forwarded from the first respective cloud computing platform;

authenticate the first storage replicator component via an application programming interface of the second respective cloud computing platform; and

store the modified data file in data storage of the second respective cloud computing platform.

13. The system of claim 9, wherein attributes of the data file include at least one of content of the data file, a filename of the data file, data file metadata format or a size of the data file.

14. The system of claim 9, wherein an event related to a respective cloud computing platform of the plurality of cloud computing platforms, including the first respective cloud computing platform and the second respective cloud computing platform, is:

one of when a file is created, a file is updated, a file is modified, a file is sent to a bucket for storage, an indication of a need to back up a file from the first respective cloud computing platform and the second respective cloud computing platform, or an indication of a need to load balance between the first respective cloud computing platform and the second respective cloud computing platform.

15. A non-transitory computer readable medium embodying programming code that when executed by at least one processor causes the at least one processor to perform functions, including functions to:

detect, from within a first cloud computing platform, an event related to a data file which is one of a plurality of data files stored in the first cloud computing platform;

based on the detected event, identify a second cloud computing platform from a plurality of other cloud computing platforms;

request, via a file copy application programming interface provided by the first cloud computing platform, a copy of the data file stored on the first cloud computing platform, wherein the copy of the data file is generated through the file copy application programing interface related to a copy file application;

obtain the requested copy of the data file;

determine, via a copy container executing in the first cloud computing platform, which attributes of the requested copy of the data file require modification to conform to data management conventions of the second cloud computing platform, by performing functions to;

access a rules engine;

provide an identifier of the second cloud computing platform to the rules engine;

in response to providing the identifier of the second cloud computing platform, retrieve attribute rules related to the second cloud computing platform; and

determine which attributes of copy of the data file involve modification to conform to the attribute rules related to the second cloud computing platform;

modify the determined attributes of the copy of the data file to comply with the conventions of the second cloud computing platform and generate a modified copy of the data file;

encrypt the modified copy of the data file; and

forward the encrypted, modified requested copy of the data file in a transmission to the second cloud computing platform.

16. The non-transitory computer readable medium of claim 15, further comprising programming code that when executed by the processor causes the processor to perform functions, including functions to:

including with the encrypted, modified requested copy of the data file forwarded to the second cloud computing platform, a request for an application programming interface of the second cloud computing platform that is compatible with the file copy application programming interface in the first cloud computing platform.

17. The non-transitory computer readable medium of claim 15, further comprising programming code that when executed by the processor causes the processor to perform functions, including functions to:

upon detecting the event related to the data file, obtain authentication credentials stored in an authentication key storage within the first cloud computing platform; and

provide the authentication credentials to the file copy application programming interface of the first cloud computing platform.

18. The non-transitory computer readable medium of claim 15, further comprising programming code that when executed by the processor causes the processor to perform functions, including functions to:

determine that a size of the requested copy of the data file is greater than a predetermined size threshold;

in response to the determination that the data file is greater than a predetermined size threshold, instantiate another copy container via the file copy application programming interface; and

provide a portion of the data contained in the data file to the other copy container to be copied.

19. The non-transitory computer readable medium of claim 15, further comprising programming code that when executed by the processor causes the processor to perform functions, including functions to:

prior to forwarding the encrypted modified, requested copy of the data file to the second cloud computing platform, further encrypting the transmission forwarding the encrypted modified, requested copy of the data file to the second cloud computing platform.