Resource provisioning systems and methods

• US 10,534,794 B2
• Filed: 01/11/2017
• Issued: 01/14/2020
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Snowflake, Inc.
• Inventors: Dageville, Benoit, Cruanes, Thierry, Zukowski, Marcin
• Primary Examiner(s): Dao, Thuy
• Application Number:
  US15/403,654
• Publication Number:
  US 20170123854A1
• Time in Patent Office: 1,098 Days
• CPC Class Codes: A61F 5/566 Intra-oral devices
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5044 : considering hardware capabilities
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.

Claims

1. A system comprising a resource manager having one or more hardware processors configured to:

monitor received data processing requests from a plurality of computerized query sources to be executed by an execution platform, the data processing requests directed to database data stored on a plurality of shared storage devices collectively storing the database data, wherein the execution platform comprises a plurality of nodes, each node independent of the plurality of computerized query sources and comprising at least one processor and at least one local cache caching at least a portion of the database data;

monitor query response rates corresponding to the database data;

determine that at least one of additional data storage capacity and additional processing capacity are needed based on the data processing requests and the query response rates; and

increase, in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes, wherein each of the one or more nodes added comprise at least one processor and at least one local cache.

2. The system of claim 1, wherein each of the plurality of nodes of the execution platform can access each of the plurality of shared storage devices.

3. The system of claim 1, wherein a number of shared storage devices is independent from a number of nodes of the execution platform.

4. The system of claim 1, wherein the resource manager is further configured to determine, based on the data processing requests, that at least one of the additional data storage capacity and the additional processing capacity are no longer needed.

5. The system of claim 4, wherein the resource manager is further programmed to decrease, in response to the determining that the at least one of the additional data storage capacity and the additional processing capacity are no longer needed, the amount of the database data cached by the execution platform by removing at least one node from the plurality of nodes.

6. The system of claim 1, wherein the plurality of nodes are arranged in a plurality of virtual warehouses.

7. The system of claim 6, wherein the plurality of virtual warehouses are grouped into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different group of users.

8. The system of claim 6, wherein the plurality of virtual warehouses are grouped into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different entity.

9. A method comprising:

controlling, by one or more processors, an execution platform comprising a plurality of nodes, each node thereof comprising at least one processor and at least one local cache, the execution platform configured to process database queries corresponding to database data stored by a plurality of shard storage devices independent from the execution platform;

monitoring, by the one or more processors, the data processing requests that originate from a plurality of query sources independent from the execution platform, the data processing requests directed to the database data;

monitoring, by the one or more processors, query response rates corresponding to the database data;

determining, by the one or more processors, that at least one of additional data storage capacity and additional processing capacity are needed based on the data processing requests and the query response rates; and

increasing, by the one or more processors, in response to the determining, an amount of the database data cached by the execution platform by adding one or more nodes to the plurality of nodes of the execution platform, wherein each of the one or more nodes added comprise at least one processor and at least one local cache.

10. The method of claim 9, further comprising determining, by the one or more processors based on the data processing requests, that at least one of the additional data storage capacity and the additional processing capacity are no longer needed.

11. The method of claim 9, further comprising decreasing, by the one or more processors in response to the determining that at least one of the additional data storage capacity and the additional processing capacity are no longer needed, the amount of the database data cached by the execution platform by removing nodes at least one node from the plurality of nodes.

12. The method of claim 9, wherein the determining that at least one of the additional data storage capacity and the additional processing capacity are no longer needed comprises:

determining, by the one or more processors, current query processing delays; and

determining, by the one or more processors, that the current query processing delays exceed a threshold.

13. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:

allocate storage resources in a storage platform, the storage resources comprising a plurality of shared storage devices storing database data;

allocate computing resources in an execution platform comprising a plurality of nodes, each node comprising at least one processor and at least one local cache, wherein the execution platform is independent from the storage resources and processes queries corresponding to database data stored by the plurality of shared storage devices of the storage platform;

monitor data processing requests that originate from a plurality of query sources independent from the execution platform, the data processing requests directed to the database data;

monitor query response rates corresponding to the database data;

increase or decrease a number of nodes of the execution platform based on the data processing requests and the query response rates; and

increase or decrease the storage resources in the storage platform based on the data processing requests and current data allocation in the plurality of shard storage devices;

wherein the number of nodes of the execution platform is independent from the amount of storage resources in the storage platform.

14. The non-transitory computer readable storage media of claim 13, wherein the instructions cause the one or more processors to determine that at least a portion of data storage capacity or processing capacity are no longer needed by:

determining current query processing delays; and

determining that the current query processing delays exceed a threshold.

15. The non-transitory computer readable storage media of claim 13, wherein the instructions cause the one or more processors to arrange the plurality of nodes of the execution platform in a plurality of virtual warehouses.

16. The non-transitory computer readable storage media of claim 15, wherein the instructions cause the one or more processors to group the plurality of virtual warehouses into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different group of users.

17. The non-transitory computer readable storage media of claim 15, wherein the instructions cause the one or more processors to group the plurality of virtual warehouses into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different entity.