Generating creation performance metrics for a secondary index of a table

• US 10,135,703 B1
• Filed: 09/18/2015
• Issued: 11/20/2018
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gupta, Aanchal, Muniswamy Reddy, Kiran Kumar, Ly, Shirley Xue Yi, Panwar, Lokendra Singh, Das, Pejus Manoj
• Primary Examiner(s): Chacko, Joe
• Application Number:
  US14/858,360
• Time in Patent Office: 1,159 Days
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/067 using time frame reporting
• CPC Class Codes: H04L 43/0888 Throughput

Abstract

A data storage system may implement generating creation performance metrics for a secondary index of a table. Metrics may be collected from different partitions of a table and different partitions of a secondary index in a table store upon initiating creation of the secondary index for the table that utilize provisioned resources for creating the secondary index. The metrics may be analyzed to generate performance metrics for the creation of the table. The performance metrics may be provided to clients of the data store or to monitoring systems. Monitoring systems may send performance event notifications to clients or automatically perform modifications to the provisioned throughput capacity.

Claims

1. A data store, comprising:

a plurality of storage nodes, wherein different ones of the storage nodes store a plurality of partitions of a table;

one or more hardware computers to implement a control plane for the data store, configured to;

initiate creation of a secondary index for the table, wherein the secondary index is stored separately from the partitions of the table and at one or more other storage nodes of the data store, wherein the different storage nodes index the partitions of the table and send updates to the one or more other storage nodes of the secondary index to store items in the secondary index, wherein the one or more other storage nodes store the received updates according to a provisioned throughput capacity for the secondary index, wherein other updates to the secondary index sent from the different storage nodes in response to access requests to the table are processed according to the same provisioned throughput capacity;

during the creation of the secondary index;

collect metrics with respect to utilization of the provisioned throughput capacity by the sent updates from the different storage nodes of the partitions of the table and the one or more other storage nodes of the secondary index;

determine one or more performance metrics for the creation of the secondary index based, at least in part, on the collected metrics; and

report the one or more performance metrics.

2. The system of claim 1, wherein the control plane is further configured to:

detect a performance event for the creation of the secondary index based, at least in part, on the one or more performance metrics; and

in response to the detection of the performance event, automatically modify the provisioned throughput capacity for the secondary index.

3. The system of claim 1, wherein the data store further comprises a graphical user interface for the data store and wherein the one or more performance metrics are reported via the graphical user interface.

4. The system of claim 3, wherein the data store is a network-based storage service that stores the table for a client of the network-based storage service, wherein the graphical user interface is implemented as part of network-based site for the network-based storage service and is accessible to the client via the network-based site.

5. A method, comprising:

performing, by one or more computing devices;

initiating creation of a secondary index for a table maintained in a plurality of partitions at a data store, wherein the secondary index is stored separately from the partitions of the table, wherein creation of the secondary index utilizes provisioned resources for at least one of the table or the secondary index, wherein the table is available for access during the creation of the secondary index;

during the creation of the secondary index;

generating one or more performance metrics for the creation of the secondary index based, at least in part, on the utilizing of the provisioned resources; and

providing the one or more performance metrics.

6. The method of claim 5, wherein generating one or more performance metrics for the creation of the secondary index comprises determining as a performance metric a number of throttling events for a period of time that throttled processing of updates for the secondary index.

7. The method of claim 5, wherein generating one or more performance metrics for the creation of the secondary index comprises determining as a performance metric a creation progress point that indicates progress of indexing the table across the partitions.

8. The method of claim 5, further comprising:

detecting a performance event for the creation of the secondary index based, at least in part, on the one or more performance metrics; and

in response to detecting the performance event, automatically modifying the provisioned resources.

9. The method of claim 5, further comprising:

detecting a performance event for the creation of the secondary index based, at least in part, on the one or more performance metrics; and

in response to detecting the performance event, sending a notification of the performance event to a client associated with the table, wherein the notification comprises a recommended modification to the provisioned resources.

10. The method of claim 9, further comprising receiving a request to modify the provisioned resources according to the recommend modification.

11. The method of claim 5,

wherein the secondary index is stored in multiple index partitions;

wherein the method further comprises requesting creation metrics from storage hosts of the partitions of the table and other storage hosts of the index partitions that indicate the processing of the updates utilizing the provisioned resources; and

wherein the one or more performance metrics are generated according to creation metrics received from the storage hosts of the partition table and the other storage hosts of the index partitions.

12. The method of claim 5, wherein the one or more performance metrics are provided via a programmatic interface.

13. The method of claim 5, wherein the data store is a non-relational data store.

14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:

initiating creation of a secondary index for a table maintained in a plurality of partitions at a data store, wherein the secondary index is stored separately from the partitions of the table, wherein creation of the secondary index utilizes provisioned resources for at least one of the table or the secondary index, wherein the table is available for access during the creation of the secondary index;

during the creation of the secondary index;

collecting metrics with respect to processing of the updates utilizing the provisioned resources from a plurality of different storage nodes hosting the partitions of the table and the secondary index;

generating one or more performance metrics for the creation of the secondary index based, at least in part, on the collected metrics; and

providing the one or more performance metrics.

15. The non-transitory, computer-readable storage medium of claim 14,

wherein the provisioned resources are a provisioned throughput capacity for the secondary index, wherein other updates to the secondary index received from the partitions of the table in response to different access requests are processed from the same provisioned throughput capacity for the secondary index;

wherein, in generating the one or more performance metrics for the creation of the secondary index, the program instructions cause the one or more computing devices to implement determining as a performance metric a consumed amount of the provisioned throughput capacity for a period of time as a result of processing updates for the secondary index.

16. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to further implement:

detecting a performance event for the creation of the secondary index based, at least in part, on the one or more performance metrics; and

in response to detecting the performance event, automatically modifying the provisioned resources.

17. The non-transitory, computer-readable storage medium of claim 14, wherein the program instructions cause the one or more computing devices to further implement:

detecting a performance event for the creation of the secondary index based, at least in part, on the one or more performance metrics;

in response to detecting the performance event, sending a notification of the performance event to a client associated with the table, wherein the notification comprises a recommended modification to the provisioned resources; and

receiving a request to modify the provisioned resources according to the recommendation modification.

18. The non-transitory, computer-readable storage medium of claim 14, wherein the data store is a relational data store.

19. The non-transitory, computer-readable storage medium of claim 14, wherein the one or more performance metrics are provided via graphical user interface for the data store.

20. The non-transitory, computer-readable storage medium of claim 19, wherein the data store is a network-based storage service that stores the table for a client of the network-based storage service, wherein the graphical user interface is implemented as part of network-based site for the network-based storage service and is accessible to the client via the network-based site.