RESOURCE PROVISIONING SYSTEMS AND METHODS

• US 20170123854A1
• Filed: 01/11/2017
• Published: 05/04/2017
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Snowflake Computing Inc. (Snowflake, Inc.)
• Inventors: Dageville, Benoit, Cruanes, Thierry, Zukowski, Marcin
• Granted Patent: US 10,534,794 B2
• Time in Patent Office: Days
• CPC Class Codes: A61F 5/566 Intra-oral devices
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5044 : considering hardware capabilities
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.

Claims

1. An apparatus comprising:

a plurality of shared storage devices collectively storing data;

an execution platform independent of the plurality of shared storage devices, the execution platform executing data processing requests directed to the data by a plurality of computerized query sources;

the execution platform comprising a plurality of nodes, each node thereof being independent of the plurality of computerized query sources and comprising (1) at least one processor and (2) at least one local cache caching at least a portion of the data; and

a resource manager comprising a software program stored in memory and executed by one or more processors connected to the execution platform, the resource manager programmed to;

monitor received data processing requests directed to the data;

determine that at least one of additional data storage capacity and additional processing capacity are needed based on the data processing requests; and

increase, in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes, wherein each of the one or more nodes added comprise (1) at least one processor and (2) at least one local cache.