LIFECYCLE TRANSITIONS IN LOG-COORDINATED DATA STORES

• US 20160086260A1
• Filed: 09/19/2014
• Published: 03/24/2016
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: VERMEULEN, ALLAN HENRY, ZAKI, OMER AHMED
• Granted Patent: US 10,373,247 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes

Abstract

Based at least in part on an optimization goal determined for a storage group comprising a plurality of data stores for which writes requests are coordinated by a log-based transaction manager, one or more lifecycle transitions are identified for data objects stored in a first data store. In accordance with a first lifecycle transition, a data object is to be transferred from the first data store to a second data store based at least in part on a duration for which the data object has been resident at the first data store. A transition agent is assigned by a configuration manager to implement the first lifecycle transition.

Claims

1. A system, comprising:

a configuration manager of a heterogeneous storage service, and

one or more transition agents;

wherein the configuration manager is configured to;

determine one or more lifecycle transitions to be performed for at least a portion of a data set to be stored at a first data store of a plurality of data stores of a log-coordinated storage group established on behalf of a client, wherein, in accordance with a first lifecycle transition of the one or more lifecycle transitions, a data object that has been accepted for inclusion in the first data store is to be transferred to a second data store of the plurality of data stores based at least in part on one or more time-based transition criteria, and wherein the second data store is designated as a destination of the first lifecycle transition based at least in part on a difference in service capability of the first data store and the second data store; and

assign a particular transition agent of the one or more transition agents to implement the first lifecycle transition;

and wherein the particular transition agent is configured to;

detect that a particular data object of the first data store meets a particular time-based transfer criterion of the one or more time-based transition criteria;

store (a) a representation of the particular data object in the second data store and (b) a transition record indicating that the representation has been stored, wherein the transition record is to be utilized to generate a billing amount for a client based on a pricing policy associated with lifecycle transitions.

2. The system as recited in claim 1, wherein the configuration manager is configured to:

receive a lifecycle flow request from the client via a programmatic interface, wherein the lifecycle flow request indicates the first data store, the second data store, and the one or more time-based transfer criterion.

3. The system as recited in claim 2, wherein the configuration manager is configured to:

receive an indication of a budget goal of the client; and

identify the first lifecycle transition as a candidate lifecycle transition based at least in part on the budget goal.

4. The system as recited in claim 1, wherein the difference in service capability comprises a difference in one or more of:

(a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.

5. The system as recited in claim 1, further comprising a transaction manager of the log-coordinated storage group, wherein the transaction manager is configured to:

receive a transaction request from the client, indicating (a) a particular write operation directed to the second data store and (b) a read descriptor indicating one or more reads on which the particular write depends;

determine, based at least in part on the read descriptor and at least in part on one or more write records stored in a persistent log, that the particular write is to be accepted for commit; and

initiate a propagation of the particular write to the second data store, without propagating the write to the first data store.

6. A method, comprising:

determining, at a configuration manager of a storage service, one or more lifecycle transitions to be implemented for at least a portion of a first data store of a plurality of data stores for which write requests are coordinated by a log-based transaction manager on behalf of a client, wherein, in accordance with a first lifecycle transition of the one or more lifecycle transitions, a data object is to be transferred to a second data store of the plurality of data stores based at least in part on a transfer criterion; and

assigning, by the configuration manager, a particular transition agent of one or more transition agents of the storage service to perform the first lifecycle transition; and

,

in response to detecting, by the particular transition agent that a particular data object meets the transfer criterion, storing, by the particular transition agent, (a) a representation of the data object in the second data store and (b) a transition record indicating that the representation has been stored.

7. The method as recited in claim 6, wherein the second data store is selected as a destination of the first lifecycle transition based at least in part on a difference in a service capability between the first data store and the second data store.

8. The method as recited in claim 7, wherein the difference in service capability comprises a difference in one or more of:

(a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.

9. The method as recited in claim 6, wherein in accordance with the first lifecycle transition, the data object is to be transferred to the second data store based at least in part on a difference in a first pricing policy associated with the first data store, and a second pricing policy associated with the second data store.

10. The method as recited in claim 6, wherein the first data store includes a plurality of storage devices utilizing a first medium type of a set of medium types comprising (a) main memory-resident storage, (b) solid-state-drive storage, (c) rotating-disk-based storage and (d) tape-based storage, and wherein the second data store includes at least one storage device utilizing the first medium type.

11. The method as recited in claim 6, wherein the particular data object is stored at a first storage device of the first data store at a first data center of a provider network, and wherein the representation of the particular data object is stored by the particular transition agent at a different storage device of the second data store at a different data center.

12. The method as recited in claim 6, further comprising:

receiving, by the configuration manager, a lifecycle flow request from the client via a programmatic interface, wherein the lifecycle flow request indicates the first data store, the second data store, and the transfer criterion.

13. The method as recited in claim 12, further comprising:

receiving, by the configuration manager, an indication of a budget goal of the client; and

identifying, by the configuration manager, the first lifecycle transition as a candidate lifecycle transition based at least in part on the budget goal.

14. The method as recited in claim 6, further comprising:

generating, by the first transition agent, the representation based at least in part on an examination of a commit record associated with the particular data object, wherein the commit record is stored in a persistent write log of the log-based transaction manager, wherein the persistent write log is implemented as a replication graph.

15. The method as recited in claim 6, wherein the first data store comprises one of:

a non-relational database system, a relational database system, a storage service that implements a web services interface allowing access to unstructured data objects, an in-memory database, or an instance of a distributed cache.

16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implements a configuration manager of a storage service, wherein the configuration manager is configured to:

determine an optimization goal for a storage group established on behalf of a client, wherein the storage group comprises a plurality of data stores for which write requests are coordinated by a log-based transaction manager;

identify, in accordance with the optimization goal, one or more data lifecycle transitions to be implemented for at least a portion of a first data store of the plurality of data stores, wherein, in accordance with a first data lifecycle transition of the one or more data lifecycle transitions, a data object is to be transferred to a second data store of the plurality of data stores based at least in part on a duration for which the data object has resided in the first data store; and

assign, by the configuration manager, at least a first transition agent to implement the first lifecycle transition.

17. The non-transitory computer-accessible storage medium claim as recited in claim 16, wherein the configuration manager is configured to:

designate the second data store as a destination of the first lifecycle transition based at least in part on a difference in service capability between the first data store and the second data store.

18. The non-transitory computer-accessible storage medium claim as recited in claim 17, wherein the difference in service capability comprises a difference in one or more of:

(a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.

19. The non-transitory computer-accessible storage medium claim as recited in claim 16, wherein the optimization goal comprises a budget goal.

20. The non-transitory computer-accessible storage medium claim as recited in claim 16, wherein the configuration manager is configured to:

determine, based at least in part on monitoring one or more metrics associated with operations performed by the first transition agent, that an additional transition agent is to be configured; and

instantiate a second transition agent to implement the first lifecycle transition.

21. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the optimization goal is indicated in a request received via a programmatic interface of the storage service, wherein the programmatic interface comprises one or more of:

(a) an application programming interface (API), (b) a web-based console, (c) a graphical user interface, (d) a menu of pre-defined service levels, or (e) a command-line tool.