System and method for automated run-tme scaling of cloud-based data store

• US 10,346,431 B1
• Filed: 04/16/2015
• Issued: 07/09/2019
• Est. Priority Date: 04/16/2015
• Status: Active Grant

• Current Assignee: Akamai Technologies, Inc.
• Original Assignee: Akamai Technologies, Inc.
• Inventors: Broda, Tal, Solnit, Matthew, Hong, Albert, Wood, Ben, Kritzer, Bobby, Xie, Fei, Hemmert, Michael, Ellis, Eric, Nayak, Shilpi
• Primary Examiner(s): Thomas, Ashish
• Assistant Examiner(s): Eyers, Dustin D
• Application Number:
  US14/688,245
• Time in Patent Office: 1,545 Days
• Field of Search: None
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A computer-implemented method includes providing a user interface (UI) that allows a user to select an existing database having cloud storage volumes that contain historical data of one or more customers. A first command input on the UI replicates the existing database is to create one or more new databases. Each of the new databases has new cloud storage volumes created by taking a snapshot of a corresponding one of the volumes of the existing database. Each new database contains the historical data of the existing database such that a new set of databases results. A mapping algorithm is executed in a cloud-computing instance that maps the historical data of each of the one or more customers to produce a shard map. The mapping algorithm includes a round-robin partitioning of the historical data striped by date in a circular order across the new set of databases.

Claims

1. A computer-implemented method for scaling of cloud-based data stores during run-time comprising:

providing a graphical user interface (GUI) that allows a user to select an existing database having a plurality of cloud storage volumes for use with one or more cloud-computing instances, the cloud storage volumes containing historical data of one or more customers, the historical data collected from a plurality of first servers configured to receive beacons from a plurality of client devices associated with real users, each first server including a processor coupled with a memory for receiving the beacons, the client devices being distributed among a plurality of geographic locations, each beacon comprising one or more data items which includes a performance metric representative of an experience of a real user on a website or application of a customer, each beacon being generated in real-time from a real user session on the website or application, the GUI allowing the user to input operational commands;

replicating, during run-time as beacons are being received, in response to a first operational command input via the GUI, the existing database to create one or more new databases for use by the one or more cloud-computing instances, each of the new databases having new cloud storage volumes that are an exact replica of the cloud storage volumes of the existing database, each of the new cloud storage volumes being created by taking a snapshot of a corresponding one of the volumes of the existing database such that each new database contains the historical data of the existing database, the replicating resulting in a new set of n databases, where n is an integer greater than 1, which includes the existing database and the one or more new databases, the cloud storage volumes of the existing database being momentarily frozen during the replicating;

executing a mapping algorithm in a cloud-computing instance that maps the historical data of each of the one or more customers according to a round-robin partitioning of the historical data striped by date across the new set of n databases, the cloud-computing instance including a processor coupled with a memory; and

compacting, in response to a second operational command input via the GUI, the new set of n databases by dropping data tables in the cloud storage volumes of each of the new set of n databases.

2. The computer-implemented method of claim 1 wherein the compacting comprises dropping (n−

1)/n of all the data tables in the cloud storage volumes of each of the new set of n databases.

3. The computer-implemented method of claim 1 wherein the new set of n databases are configured in a circular order for the mapping algorithm, and the round-robin partitioning of the historical data comprises locating data of a customer obtained on a first date in a first one of the new set of n databases, locating data of the customer obtained on a next date in a next one of the new set of n databases, and so on, continuing in the circular order.

4. The computer-implemented method of claim 1 wherein the new set of n databases are configured in a circular order for the mapping algorithm, and the mapping algorithm further comprises a future mapping for a following day wherein data of a customer obtained for the following day is a mapped to a next database in the circular order, the next database being a present database in the circular order incremented by one.

5. The computer-implemented method of claim 1 wherein the mapping algorithm further comprises:

determining, for each database, whether a total size of the historical data written to the database a certain number of days ago exceeds a specific threshold amount above or below a statistical criteria;

re-balancing, in response to the specific threshold being exceeded, the mapping of the historical data across the new set of n databases for write workload by customer.

6. The computer-implemented method of claim 5 wherein the re-balancing comprises an iterative process that distributes a portion of the write workload for the database found to exceed the specific threshold, among one or more of a remainder of the of new set of n databases.

7. A non-transitory computer-readable storage medium encoded with computer instructions, which, when executed by one or more processors, operable to:

provide a graphical user interface (GUI) that allows a user to select an existing database having a plurality of cloud storage volumes for use with one or more cloud-computing instances, the cloud storage volumes containing historical data of one or more customers, the historical data collected from a plurality of first servers configured to receive beacons from a plurality of client devices associated with real users, each first server including a processor coupled with a memory for receiving the beacons, the client devices being distributed among a plurality of geographic locations, each beacon comprising one or more data items which includes a performance metric representative of an experience of a real user on a website or application of a customer, each beacon being generated in real-time from a real user session on the website or application, the GUI allowing the user to input operational commands;

replicate, during run-time as beacons are being received, in response to a first operational command input via the GUI, the existing database to create one or more new databases for use by the one or more cloud-computing instances, each of the new databases having new cloud storage volumes that are an exact replica of the cloud storage volumes of the existing database, each of the new cloud storage volumes being created by taking a snapshot of a corresponding one of the volumes of the existing database such that each new database contains the historical data of the existing database, replication resulting in a new set of n databases, where n is an integer greater than 1, which includes the existing database and the one or more new databases, the cloud storage volumes of the existing database being momentarily frozen during replication;

map the historical data of each of the one or more customers according to a round-robin partitioning of the historical data striped by date across the new set of n databases, the cloud-computing instance including a processor coupled with a memory; and

compact, in response to a second operational command input via the GUI, the new set of n databases by dropping data tables in the cloud storage volumes of each of the new set of n databases.

8. The non-transitory computer-readable storage medium of claim 7 wherein execution of the computer instructions is further operable to drop (n−

1)/n of all the data tables in the cloud storage volumes of each of the new set of n databases.

9. The non-transitory computer-readable storage medium of claim 7 wherein the new set of n databases are configured in a circular order and the round-robin partitioning of the historical data comprises locating data of a customer obtained on a first date in a first one of the new set of n databases, locating data of the customer obtained on a next date in a next one of the new set of n databases, and so on, continuing in the circular order.

10. The non-transitory computer-readable storage medium of claim 7 wherein the new set of n databases are configured in a circular order, and execution of the computer instructions is further operable to execute a future mapping for a following day wherein data of a customer obtained for the following day is a mapped to a next database in the circular order, the next database being a present database in the circular order incremented by one.

11. The non-transitory computer-readable storage medium of claim 7 wherein execution of the computer instructions is further operable to:

determine, for each database, whether a total size of the historical data written to the database a certain number of days ago exceeds a specific threshold amount above or below a statistical criteria;

re-balance, in response to the specific threshold being exceeded, the mapping of the historical data across the new set of n databases for write workload by customer.

12. The non-transitory computer-readable storage medium of claim 11 wherein the re-balance comprises an iterative process that distributes a portion of the write workload for the database found to exceed the specific threshold, among one or more of a remainder of the of new set of n databases.

13. A system comprising:

a plurality of first servers configured to receive beacons from a plurality of client devices associated with real users, each first server including a processor coupled with a memory for receiving the beacons, the client devices being distributed among a plurality of geographic locations, each beacon comprising one or more data items which includes a performance metric representative of an experience of a real user on a website or application of a customer, each beacon being generated in real-time from a real user session on the website or application;

one or more cloud-computing instances;

a first database having a plurality of cloud storage volumes for use with the one or more cloud-computing instances, the cloud storage volumes containing historical data of one or more customers, the historical data collected from the plurality of first servers;

a main computing instance that includes a processor coupled with a memory configured to run a computer program that provides a graphical user interface (GUI) that allows a user to input operational commands, the computer program being operable, in response to a first operational command, to replicate during run-time, as beacons are being received, the first database to create one or more new databases for use by the one or more cloud-computing instances, each of the new databases having new cloud storage volumes that are an exact replica of the cloud storage volumes of the existing database, each of the new cloud storage volumes being created by taking a snapshot of a corresponding one of the volumes of the first database such that each new database contains the historical data of the first database, replication resulting in a new set of n databases, where n is an integer greater than 1, which includes the existing database and the one or more new databases, the cloud storage volumes of the existing database being momentarily frozen during replication;

wherein the computer program is further operable, when executed, to map the historical data of each of the one or more customers according to a round-robin partitioning of the historical data striped by date across the new set of n databases; and

wherein the computer program is further operable, in response to a second operational command, to compact the new set of n databases by dropping data tables in the cloud storage volumes of each of the new set of n databases.

14. The system of claim 13 wherein the computer program is further operable, in response to a second operational command, to drop (n−

1)/n of all the data tables in the cloud storage volumes of each of the new set of n databases.

15. The system of claim 13 wherein the new set of n databases are configured in a circular order and the round-robin partitioning of the historical data comprises locating data of a customer obtained on a first date in a first one of the new set of n databases, locating data of the customer obtained on a next date in a next one of the new set of n databases, and so on, continuing in the circular order.