Local key-value database synchronization

• US 10,242,084 B2
• Filed: 07/10/2017
• Issued: 03/26/2019
• Est. Priority Date: 10/15/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Shams, Khawaja Salman, Pandey, Prashant, Sivasubramanian, Swaminathan, Zaki, Omer Ahmed, Yanacek, David Craig, Olson, Johanna S, de Kock, Hendrik, Wu, Johnny Ying
• Primary Examiner(s): Stevens, Robert
• Application Number:
  US15/645,897
• Publication Number:
  US 20170308558A1
• Time in Patent Office: 624 Days
• Field of Search: 707638
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation

Abstract

A remote distributed data store may be configured to process data updates received through invocation of a common API with reference to a common schema. A local data store may also be configured to process updates through the common API with reference to the common schema. Updates to the local data store may be mapped from the local data store schema to the common schema, and applied to the distributed data store. Updates to the distributed data store may be mapped from the common schema to the local data store schema. User identity may be verified to limit data synchronization to authorized users.

Claims

1. A system, comprising:

a memory to store program instructions which, if performed by at least one processor, cause the at least one processor to perform a method to at least;

receive, at an emulator, an invocation of an application programming interface (API) for a data storage service to access an item of a data set stored in a data store according to a first data schema from an application running on the at least one processor, wherein the API includes a reference to the item specified according to a second data schema for data stored in the storage service;

responsive to the invocation of the API;

translate, by the emulator, the reference to the item in the API to specify the item according to the first data schema;

access, by the emulator, the item in the data set at the data store using the translated reference to the item instead of the reference to the item included in the API; and

return, by the emulator, a result of the access to the application according to the API.

2. The system of claim 1, wherein the access to the item updates the item, and wherein the program instructions further cause the at least one processor to perform the method to send, by the emulator, an indication of the update to the data storage service to synchronize a copy of the item stored at the data storage service.

3. The system of claim 1, wherein the access to the item reads the item, and wherein the program instructions further cause the at least one processor to perform the method to:

prior to the return of the result, modify the result to include a prior version of the item than a version of the item stored in the data store when the invocation of the API is received in order to simulate an eventually consistent replication model for a distributed data store.

4. The system of claim 1, wherein the program instructions further cause the at least one processor to perform the method to:

receive, at the emulator, another invocation of the API to access another item of the data set stored in the data store from the application, wherein the API includes a reference to the other item specified according to the second data schema;

responsive to the other invocation of the API;

determine, by the emulator, that a throughput rate for access to the data set from the application exceeds a provisioned throughput; and

in response to the determination that the throughput rate for the access to the data set from the application exceeds the provisioned throughput amount, return, by the emulator, a rejection response for the other invocation to the application according to the API.

5. The system of claim 1, wherein the program instructions further cause the at least one processor to perform the method to:

prior to the return of the result, delay the return of the result in order to simulate a network latency between the application and the data storage service.

6. The system of claim 1, wherein the program instructions further cause the at least one processor to perform the method to:

receive, at the emulator, another invocation of the API to access another item of the data set stored in the data store from the application, wherein the API includes a reference to the other item specified according to the second data schema;

responsive to the other invocation of the API;

determine, by the emulator, a data storage service error to inject into processing of the invocation of the API; and

return, by the emulator, the determined error in a response for the other invocation to the application according to the API.

7. The system of claim 1, wherein the first data schema is a relational data schema, and wherein the second data schema is a key-value data schema.

8. A method, comprising:

receiving, at an emulator, an invocation of an application programming interface (API) for a data storage service to access an item of a data set stored in a data store according to a first data schema from an application running on the at least one processor, wherein the API includes a reference to the item specified according to a second data schema for data stored in the storage service;

responsive to the invocation of the API;

translating, by the emulator, the reference to the item in the API to specify the item according to the first data schema;

accessing, by the emulator, the item in the data set at the data store using the translated reference to the item instead of the reference to the item included in the API; and

returning, by the emulator, a result of the access to the application according to the API.

9. The method of claim 8, wherein the access to the item updates the item, and wherein the method further comprises sending, by the emulator, an indication of the update to the data storage service to synchronize a copy of the item stored at the data storage service.

10. The method of claim 8, wherein the access to the item reads the item, and wherein the method comprises:

prior to returning the result, modifying the result to include a prior version of the item than a version of the item stored in the data store when the invocation of the API is received in order to simulate an eventually consistent replication model for a distributed data store.

11. The method of claim 8, further comprising:

receiving, at the emulator, another invocation of the API to access another item of the data set stored in the data store from the application, wherein the API includes a reference to the other item specified according to the second data schema;

responsive to the other invocation of the API;

determining, by the emulator, that a throughput rate for access to the data set from the application exceeds a provisioned throughput; and

in response to determining that the throughput rate for the access to the data set from the application exceeds the provisioned throughput amount, returning, by the emulator, a rejection response for the other invocation to the application according to the API.

12. The method of claim 8, further comprising:

prior to the return of the result, delaying the return of the result in order to simulate a network latency between the application and the data storage service.

13. The method of claim 8, further comprising:

receiving, at the emulator, another invocation of the API to access another item of the data set stored in the data store from the application, wherein the API includes a reference to the other item specified according to the second data schema;

responsive to the other invocation of the API;

determining, by the emulator, a data storage service error to inject into processing of the invocation of the API; and

returning, by the emulator, the determined error in a response for the other invocation to the application according to the API.

14. The method of claim 8, wherein the first data schema is a relational data schema, and wherein the second data schema is a key-value data schema.

15. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices, causes the one or more computing devices to implement:

receiving, at an emulator, an invocation of an application programming interface (API) for a data storage service to access an item of a data set stored in a data store according to a first data schema from an application running on the at least one processor, wherein the API includes a reference to the item specified according to a second data schema for data stored in the storage service;

responsive to the invocation of the API;

translating, by the emulator, the reference to the item in the API to specify the item according to the first data schema;

accessing, by the emulator, the item in the data set at the data store using the translated reference to the item instead of the reference to the item included in the API; and

returning, by the emulator, a result of the access to the application according to the API.

16. The non-transitory, computer-readable storage medium of claim 15, wherein the access to the item updates the item, and wherein the program instructions cause the one or more computing devices to further implement sending, by the emulator, an indication of the update to the data storage service to synchronize a copy of the item stored at the data storage service.

17. The non-transitory, computer-readable storage medium of claim 15, wherein the access to the item reads the item, and wherein the program instructions cause the one or more computing devices to further implement:

prior to returning the result, modifying the result to include a prior version of the item than a version of the item stored in the data store when the invocation of the API is received in order to simulate an eventually consistent replication model for a distributed data store.

18. The non-transitory, computer-readable storage medium of claim 15, wherein the program instructions cause the one or more computing devices to further implement:

receiving, at the emulator, another invocation of the API to access another item of the data set stored in the data store from the application, wherein the API includes a reference to the other item specified according to the second data schema;

responsive to the other invocation of the API;

determining, by the emulator, that a throughput rate for access to the data set from the application exceeds a provisioned throughput; and

in response to determining that the throughput rate for the access to the data set from the application exceeds the provisioned throughput amount, returning, by the emulator, a rejection response for the other invocation to the application according to the API.

19. The non-transitory, computer-readable storage medium of claim 15, wherein the program instructions cause the one or more computing devices to further implement:

prior to the return of the result, delaying the return of the result in order to simulate a network latency between the application and the data storage service.

20. The non-transitory, computer-readable storage medium of claim 15, wherein the program instructions cause the one or more computing devices to further implement:

receiving, at the emulator, another invocation of the API to access another item of the data set stored in the data store from the application, wherein the API includes a reference to the other item specified according to the second data schema;

responsive to the other invocation of the API;

determining, by the emulator, a data storage service error to inject into processing of the invocation of the API; and

returning, by the emulator, the determined error in a response for the other invocation to the application according to the API.