Resource provisioning systems and methods

• US 10,776,388 B2
• Filed: 01/13/2020
• Issued: 09/15/2020
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Snowflake, Inc.
• Inventors: Dageville, Benoit, Cruanes, Thierry, Zukowski, Marcin
• Primary Examiner(s): Dao, Thuy
• Application Number:
  US16/741,660
• Publication Number:
  US 20200151192A1
• Time in Patent Office: 246 Days
• CPC Class Codes: A61F 5/566 Intra-oral devices
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5044 : considering hardware capabilities
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A system is described inkling a cache memory and a hardware processor. The hardware processor to determine that at least one of additional data storage capacity and additional processing capacity are needed based on data processing requests of a plurality of computerized query sources to be executed by an execution platform comprising a plurality of nodes, where the data processing requests are directed to database data stored on a plurality of shared storage devices collectively storing the database data. The hardware processor further to increase, in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes.

Claims

1. A system comprising:

a memory; and

a processor, operatively coupled with the memory, the processor to;

monitor received data processing requests, each data processing request directed to data stored in a plurality of shared storage devices and to be executed by an execution platform comprising a plurality of execution nodes, wherein the plurality of execution nodes is logically divided into one or more virtual warehouses, and each virtual warehouse establishes a communication link with each of one or more of the plurality of shared storage devices based at least in part on a respective received data processing request;

monitor for each of the one or more virtual warehouses, a current resource utilization, query response rates, and a number of users interacting with the virtual warehouse;

determine whether additional processing resources are required based at least in part on the received data processing requests, the current utilization of the one or more virtual warehouses, query response rates of the one or more virtual warehouses and a number of users interacting with each of the one or more virtual warehouses;

determine whether additional data storage capacity is needed based on the received data processing requests;

allocate, in response to determining that additional data storage capacity is needed, one or more additional shared storage devices to the plurality of shared storage devices; and

allocate, in response to determining that additional processing resources are required, one or more additional execution nodes to the plurality of execution nodes.

2. The system of claim 1, wherein each of the plurality of execution nodes comprise at least one processor and at least one cache, and wherein the current resource utilization of a virtual warehouse comprises a processor utilization and a cache utilization of one or more of the execution nodes of the virtual warehouse.

3. The system of claim 1, wherein each of the plurality of execution nodes of the execution platform can access each of the plurality of shared storage devices.

4. The system of claim 1, wherein the plurality of shared storage devices is separate and independent from the execution platform.

5. The system of claim 1, wherein the processor is further to determine, based on subsequently received data processing requests, that at least one of the additional shared storage devices are no longer needed.

6. The system of claim 5, wherein the processor is further to decrease, in response to the determining that at least one of the additional shared storage devices are no longer needed, the number of shared storage devices from the plurality of shared storage devices.

7. The system of claim 1, wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different group of users.

8. The system of claim 1 wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different entity.

9. A method, comprising:

monitoring received data processing requests, each data processing request directed to data stored in a plurality of shared storage devices and to be executed by an execution platform comprising a plurality of execution nodes, wherein the plurality of execution nodes is logically divided into one or more virtual warehouses, and each virtual warehouse establishes a communication link with each of one or more of the plurality of shared storage devices based at least in part on a respective received data processing request;

monitoring, for each of the one or more virtual warehouses, a current resource utilization, query response rates, and a number of users interacting with the virtual warehouse;

determining whether additional processing resources are required based at least in part on the received data processing requests, the current utilization of the one or more virtual warehouses, query response rates of the one or more virtual warehouses and a number of users interacting with each of the one or more virtual warehouses;

determining, by a processor, whether additional data storage capacity is needed based on the received data processing requests;

allocating, in response to determining that additional data storage capacity is needed, one or more additional shared storage devices to the plurality of shared storage devices; and

allocating, in response to determining that additional processing resources are required, one or more additional execution nodes to the plurality of execution nodes.

10. The method of claim 9, wherein each of the plurality of execution nodes comprise at least one processor and at least one cache, and wherein the current resource utilization of a virtual warehouse comprises a processor utilization and a cache utilization of one or more of the execution nodes of the virtual warehouse.

11. The method of claim 9, further comprising determining, based on subsequently received data processing requests, that at least one of the additional shared storage devices are no longer needed.

12. The method of claim 11, wherein the determining that at least one of the shared storage devices are no longer needed comprises:

determining current query processing delays; and

determining that the current query processing delays exceed a threshold.

13. The method of claim 9, further comprising decreasing, in response to the determining that at least one of the additional shared storage devices are no longer needed, the number of shared storage devices from the plurality of nodes.

14. The method of claim 9, wherein each of the plurality of execution nodes of the execution platform can access each of the plurality of shared storage devices.

15. The method of claim 9, wherein the plurality of shared storage devices is independent from the execution platform.

16. The method of claim 9, wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different group of users.

17. The method of claim 9, wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different entity.

18. A non-transitory computer readable storage medium storing instructions that, when executed by a processor, cause the processor to:

monitor received data processing requests, each data processing request directed to data stored in a plurality of shared storage devices and to be executed by an execution platform comprising a plurality of execution nodes, wherein the plurality of execution nodes is logically divided into one or more virtual warehouses, and each virtual warehouse establishes a communication link with each of one or more of the plurality of shared storage devices based at least in part on a respective received data processing request;

monitor for each of the one or more virtual warehouses, a current resource utilization, query response rates, and a number of users interacting with the virtual warehouse;

determine whether additional processing resources are required based at least in part on the received data processing requests, the current utilization of the one or more virtual warehouses, query response rates of the one or more virtual warehouses and a number of users interacting with each of the one or more virtual warehouses;

determine whether additional data storage capacity is needed based on data the received data processing requests; and

allocate, in response to determining that additional data storage capacity is needed, one or more additional shared storage devices to the plurality of shared storage devices; and

allocate, in response to determining that additional processing resources are required, one or more additional execution nodes to the plurality of execution nodes.

19. The non-transitory computer readable storage medium of claim 18, wherein the processor is further to determine, based on subsequently received data processing requests, that at least one of the additional shared storage devices are no longer needed.

20. The non-transitory computer readable storage medium of claim 19, wherein to determine that at least one of the additional shared storage devices are no longer needed, the processor is to:

determine current query processing delays; and

determine that the current query processing delays exceed a threshold.

21. The non-transitory computer readable storage medium of claim 18, wherein the processor is further to group the one or more virtual warehouses into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different group of users.

22. The non-transitory computer readable storage medium of claim 18, wherein the processor is further to group the one or more virtual warehouses into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different entity.