Recommending provisioned throughput capacity for generating a secondary index for an online table

• US 10,146,814 B1
• Filed: 09/18/2015
• Issued: 12/04/2018
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gupta, Aanchal, Muniswamy Reddy, Kiran Kumar, Marshall, Stuart Henry Seelye
• Primary Examiner(s): NGUYEN, MERILYN P
• Application Number:
  US14/859,075
• Time in Patent Office: 1,173 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A data storage system may implement recommending throughput capacity for generating a secondary index for an online table. Secondary indexes may be created for a table stored in multiple partitions in the data storage system. The data storage system may allow users to provision throughput capacity to process access requests at secondary indexes. The throughput capacity may also be used to perform updates at the secondary index as part of generating the secondary index. A creation throughput capacity may be determined that compensates for performing the updates to generate the secondary index. A user may receive a recommendation with the creation throughput capacity for creating the secondary index and may accept or modify the creation throughput capacity. The creation throughput may be automatically provisioned to create the secondary index, in some embodiments.

Claims

1. A distributed data store, comprising:

a plurality of storage nodes that store different partitions of a table;

an interface for the distributed data store;

one or more hardware processors and associated memory to implement a control plane for the distributed data store, configured to;

receive a request, via the interface, to create a secondary index for the table at one or more other storage nodes in the distributed data store;

generate a recommendation for provisioning throughput capacity at the other storage nodes of the secondary index, wherein the recommendation identifies a creation throughput capacity that compensates for processing updates received at the other storage nodes as part of creating the secondary index in addition to baseline throughput capacity for processing other access requests received at the other storage nodes, wherein the updates received at the other storage nodes as part of creating the secondary index are received from different ones of the storage nodes storing the partitions of the table;

provision a total throughput capacity for the secondary index that includes the recommended creation throughput capacity and the baseline throughput capacity; and

initiate creation of the secondary index using the provisioned creation throughput capacity.

2. The distributed data store of claim 1, wherein to generate the recommendation, the control plane is configured to:

identify a maximum indexing rate for the table according to a number of partitions of the table; and

calculate additional throughput capacity to be added to the baseline throughput capacity such that a total throughput capacity for processing the updates received as part of creating the index at the maximum indexing rate, wherein the additional throughput capacity is identified as the creation throughput capacity.

3. The distributed data store of claim 1, wherein the control plane is further configured to determine an estimated creation time for the secondary index based, at least in part, on the creation throughput capacity, and wherein the estimated creation time is included in the recommendation sent to the user.

4. The distributed data store of claim 1, wherein the distributed data store is a non-relational storage service, wherein the table is maintained for a customer account of the non-relational storage service, and wherein the interface for the distributed data store is a graphical user interface.

5. A method, comprising:

performing, by one or more computing devices;

for a secondary index for a table stored across a plurality of partitions in a distributed data store;

determining a creation throughput capacity to be provisioned for the secondary index that compensates for processing updates received at the secondary index as part of creating the secondary index in addition to other throughput capacity for processing other access requests received at the secondary index, wherein the updates received at the secondary index as part of creating the secondary index are received from different ones of the plurality of partitions of the table;

providing a provisioning recommendation indicating the creation throughput capacity via an interface for the distributed data store; and

initiating creation of the secondary index using throughput capacity provisioned based on the provisioning recommendation.

6. The method of claim 5, wherein determining the creation throughput capacity to be provisioned for the secondary index comprises:

determining a number of partitions of the table to identify a maximum indexing rate for the table; and

calculating additional throughput capacity to be added to the other throughput capacity such that a total throughput capacity at the secondary index is sufficient to process the updates received as part of creating the index at the maximum indexing rate, wherein the additional throughput capacity is identified as the creation throughput capacity.

7. The method of claim 6, wherein determining the creation throughput capacity to be provisioned for the secondary index further comprises reducing the additional throughput capacity based, at least in part, on an estimated number of partitions of the secondary index sufficient to provide the total throughput capacity, wherein the reduced throughput capacity is identified as the creation throughput capacity.

8. The method of claim 5, further comprising:

receiving a request to create the secondary index, wherein the request comprises a creation time for the secondary index; and

wherein determining the creation throughput capacity comprises calculating additional throughput capacity to be provisioned to create the secondary index within the creation time.

9. The method of claim 8, wherein determining the estimated creation time comprises:

determining a normalized size of the table according to a number of input/output (I/O) operations to access items in the table at a point in time;

calculating a time that partitions of the table the partitions of the table can index the normalized size of the table in parallel according to a maximum indexing rate;

calculating a time that the secondary index can process the normalized sized of the table according to the creation throughput capacity; and

selecting as the estimated time the greater of the calculated times.

10. The method of claim 5, wherein the secondary index is stored across a plurality of other partitions in the distributed data store.

11. The method of claim 5, further comprising:

determining another creation throughput capacity that compensates for processing access requests at the partitions of the table; and

wherein the other creation throughput capacity is included in the recommendation provided to the user.

12. The method of claim 5, wherein the creation throughput capacity is identified as a number of I/O operations per second (IOPS) for write operations.

13. The method of claim 5, wherein the distributed data store is a network-based storage service, wherein the table is maintained for a customer account of the non-relational storage service, and wherein the interface for the distributed data store is a network-based interface.

14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:

receiving a request to create a secondary index for a table stored across a plurality of partitions in a distributed data store;

generating a recommendation for provisioning throughput capacity at the secondary index, wherein the recommendation identifies a creation throughput capacity that compensates for processing updates received at the secondary index in addition to baseline throughput capacity for processing other access requests received at the secondary index, wherein the updates received at the secondary index as part of creating the secondary index are received from different ones of the partitions of the table;

provisioning a total throughput capacity that includes the creation throughput capacity identified in the recommendation and the baseline throughput capacity for the secondary index; and

initiating creation of the secondary index using the creation total throughput capacity.

15. The non-transitory, computer-readable storage medium of claim 14, wherein, in generating the recommendation, the programming instructions cause the one or more computing devices to implement:

determining a number of partitions of the table to identify a maximum indexing rate for the table; and

calculating additional throughput capacity to be added to the baseline throughput capacity such that a total throughput capacity at the secondary index is sufficient to process the updates received as part of creating the index at the maximum indexing rate, wherein the additional throughput capacity is identified as the creation throughput capacity.

16. The non-transitory, computer-readable storage medium of claim 14, wherein, in generating the recommendation, the programming instructions cause the one or more computing devices to implement:

modifying a candidate creation throughput capacity in response to determining that an estimated creation time for the secondary index determined according to the candidate creation throughput capacity exceeds a creation time limit.

17. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to further implement:

determining an estimated creation time for the secondary index based, at least in part, on the creation throughput capacity; and

wherein the estimated creation time is included in the recommendation provided to the user.

18. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to implement:

wherein provisioning the total throughput capacity that includes the creation throughput capacity and the baseline throughput capacity for the secondary index is performed in response to a user request that includes the creation throughput capacity in a request to provision the total throughput capacity for the secondary index.

19. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to implement:

upon creation of the secondary index, reducing the total throughput capacity to remove the creation throughput capacity.

20. The non-transitory, computer-readable storage medium of claim 14, wherein the distributed data store is a network-based storage service, wherein the table is maintained for a customer account of the non-relational storage service, and wherein the interface for the distributed data store is a graphical user interface.