Tiered middleware framework for data storage

• US 8,402,061 B1
• Filed: 08/27/2010
• Issued: 03/19/2013
• Est. Priority Date: 08/27/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Briggs, Gregory J., Rohr, Vincent M.
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US12/870,466
• Time in Patent Office: 935 Days
• Field of Search: 707600-899
• US Class Current: 707/791
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems

Abstract

A tiered middleware framework for transacting data between an application layer and a data storage layer includes modular components. The tiered middleware framework is implemented when the arrangement of the modular components enables the tiered middleware framework to meet or exceed certain data transaction criteria. However, when the data transaction criteria are not met by the tiered middleware framework with arrangement of the modular components, the modular components in the tiered middleware framework is rearranged.

Claims

1. A computer-implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

receiving an arrangement of modular components for a tiered middleware framework that transacts blob data between an application layer that includes at least one application and a data storage layer that includes at least one data store, the arrangement being developed based at least in part on one or more data usage attributes of the at least one application and at least in part on one or more specifications of the at least one data store in the data storage layer;

testing the tiered middleware framework via transfer of blob data between the application layer and the data storage layer to obtain at least one of a quantitative measurement or a qualitative measurement;

implementing the tiered middleware framework for transacting blob data between the application layer and the data storage when the at least one of the quantitative measurement or qualitative measurement indicates that the arrangement of the modular components meets or exceeds data transaction criteria; and

receiving a rearrangement of the modular components for the tiered middleware framework for additional testing when the at least one of the quantitative measurement or qualitative measurement indicates that the arrangement of the modular components does not meet the data transaction criteria, the rearrangement being optimized based at least in part on tradeoffs between desired objectives for at least two of latency, availability, durability, operation cost, or throughput of each data transaction within the tiered middleware framework.

2. The computer-implemented method of claim 1, wherein the receiving the arrangement includes receiving modular components for facilitating blob transactions that include at least one of a transactionally consistent indexer component, a multi-blob consistency component, a blob transaction component, or a sub-containment concurrency component.

3. The computer-implemented method of claim 1, wherein the one or more data usage attributes include at least one of a type of data to be transacted, access frequency of the data being transacted, data latency, data availability, data throughput, or chunk size of the data.

4. The computer-implemented method of claim 1, wherein the data transaction criteria includes desired objectives for at least two of latency, availability, consistency, durability, operation cost, or throughput of each data transaction within the tiered middleware framework.

5. A computer-implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

receiving an arrangement of modular components for a tiered middleware framework that transacts blob data between an application layer and a data storage layer;

implementing the tiered middleware framework for transacting the blob data between the application layer and the data storage when the arrangement of the modular components meets or exceeds data transaction criteria; and

receiving a rearrangement of the modular components for the tiered middleware framework when the arrangement of the modular components does not meet the data transaction criteria.

6. The computer-implemented method of claim 5, wherein the data storage layer include at least one data server that provides one or more data stores for storing the blob data.

7. The computer-implemented method of claim 5, wherein the receiving the arrangement includes receiving modular components for facilitating blob transactions that include at least one of a transactionally consistent indexer component, a multi-blob consistency component, a blob transaction component, or a sub-containment concurrency component.

8. The computer-implemented method of claim 5, wherein the receiving the arrangement includes receiving the arrangement that is developed based at least in part on one or more data usage attributes of at least one application in the application layer and at least in part on one or more specifications of at least one data store in the data storage layer.

9. The computer-implemented method of claim 8, wherein the one or more data usage attributes include at least one of a type of data to be transacted, access frequency of the data being transacted, data latency, data availability, data throughput, or chunk size of the data.

10. The computer-implemented method of claim 8, wherein the one or more specifications includes at least one of the latency, availability, durability, throughput, operation cost, a type of each data store, or a number of data stores in the data storage layer.

11. The computer-implemented method of claim 5, wherein the data transaction criteria includes desired objectives for at least two of latency, availability, consistency, durability, throughput of each data transaction in the tiered middleware framework.

12. The computer-implemented method of claim 5, wherein the receiving the arrangement includes receiving an arrangement that is optimized based at least in part on tradeoffs between desired objectives for at least two of latency, availability, consistency, durability, or throughput of each data transaction in the tiered middleware framework.

13. The computer-implemented method of claim 5, wherein the receiving the rearrangement includes receiving an rearrangement that is optimized via at least one of a re-reorganization of the modular components, integration of one or more additional instances of a modular component in the arrangement of modular components, addition of one or more other modular components or different versions of a modular component, elimination of one or more modular components, or replacement of at least one modular component with another modular component or another version of the at least one modular component.

14. The computer-implemented method of claim 5, further comprising, prior to the implementing and the receiving of the rearrangement:

defining the data transaction criteria; and

testing the tiered middleware framework via transfer of blob data between the application layer and the data storage layer to obtain at least one of a quantitative measurement or a qualitative measurement;

and wherein the implementing of the tiered middleware framework occurs when the at least one of the quantitative measurement or qualitative measurement indicates that the arrangement of the modular components meets or exceeds data transaction criteria, and the receiving of the rearrangement occurs when the at least one of the quantitative measurement or qualitative measurement indicates that the arrangement of the modular components does not meet the data transaction criteria.

15. A server, comprising:

a processor; and

memory storing modular components of a tiered middleware storage layer that are executable by the processor to transact blob data between an application layer and a blob data storage layer, the modular components comprising;

an application interface component to receive data transactions from the application layer, each of the data transaction to update, insert, or delete one or more data blobs in at least one blob data store of the data storage layer;

a data store adapter component to communicate the data transactions to the at least one blob data store of the data storage layer;

a plurality of components that are integrated between the application interface component and the data store adapter component to perform the data transactions on the at least one blob data store, the plurality of components including at least one of;

a transitionally consistent indexer component to provide a transactional index that at least tracks one or more non-primary key property fields of each data blob in the at least one data store;

a multi-blob consistency component to enable the data storage layer to insert, update, or delete a plurality of data blobs from the at least one blob data store in an atomic manner via a master blob;

a blob transaction component to facilitate a data transaction across a plurality of blobs that are stored in different blob data stores in an optimistically consistent manner via a record blob;

or

a sub-containment concurrency component to ensure optimistic concurrency for the data transactions performed on a plurality of data blobs that are stored in the at least one blob data store of the data storage layer via a sub-containment hierarchical structure.

16. The server of claim 15, wherein the plurality of components includes the transitionally consistent indexer component, and wherein an application in the application layer is to perform a search query for data in the at least one data blob by performing a query against the transactional index.

17. The server of claim 15, wherein the modular components of the tiered middleware storage layer are placed in an arrangement based at least in part on one or more data usage attributes of at least one application in the application layer and at least in part on one or more specifications of blob data stores in the data storage layer to meet a plurality of data transaction criteria.

18. The server of claim 17, wherein the one or more data usage attributes include at least one of a type of data to be transacted, access frequency of the data being transacted, data latency, data availability, data throughput, or chunk size of the data.

19. The server of claim 17, wherein the one or more specifications includes at least one of the latency, availability, durability, throughput, operation cost, or a number of the blob data stores in the data storage layer.

20. The server of claim 17, wherein the data transaction criteria includes desired objectives for at least two of latency, availability, consistency, durability, throughput of each data transaction.

21. The server of claim 17, wherein the arrangement is optimized based at least in part on tradeoffs between desired objectives for at least two of latency, availability, consistency, durability, or throughput of each data transaction.