Executing queries for structured data and not-structured data

• US 10,713,247 B2
• Filed: 03/31/2017
• Issued: 07/14/2020
• Est. Priority Date: 03/31/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Petropoulos, Michail, Pandis, Ippokratis
• Primary Examiner(s): Alam, Hosain T
• Assistant Examiner(s): Gemignani, Anthony G
• Application Number:
  US15/476,273
• Publication Number:
  US 20180285418A1
• Time in Patent Office: 1,201 Days
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/3332 Query translation

Abstract

Queries received at a query engine may be executed for structured data and not-structured data. A query execution plan may be generated for the query that includes stateless operations to apply the query to the not-structured data at remote query processing engines. The remote query processing engines may perform the stateless operations and return results to the query engine. The query engine may generate a result for the query based on the results received from the remote query engine as well as results determined as part of applying the query to structured data. The result to the query may be returned.

Claims

1. A system, comprising:

a memory to store program instructions which, if performed by at least one processor, cause the at least one processor to perform a method to at least;

receive a query on structured data and not-structured data at a query engine, wherein the structured data is stored according to a pre-defined data model and the not-structured data is not stored according to a pre-defined data model;

generate, by the query engine, a query execution plan for the query that comprises;

one or more stateful operations to apply the query to the structured data; and

stateless operations to apply the query to the not-structured data at one or more remote query processing engines;

send, by the query engine, the stateless operations to the remote query processing engines as part of performing the query execution plan to execute the query on the structured data and not-structured data; and

generate, by the query engine, a result for the query based, at least in part, on;

respective results for the stateless operations received from the remote query processing engines; and

respective results for the stateful operations to apply the query to the structured data; and

return, by the query engine the result for the query.

2. The system of claim 1, wherein the stateless operations are sent to the remote query processing engines according to a programmatic interface for the query processing engines.

3. The system of claim 2, wherein the query engine is implemented as a part of a leader node of a cluster of compute nodes, wherein the leader node provides instructions to one or more compute nodes of the cluster to send the stateless operations to the remote query processing engines, and wherein the leader node receives the respective results as part of other results generated by the one or more compute nodes in order to generate the result for the second query.

4. The system of claim 2, wherein the query engine is implemented as part of a network-based data warehouse service, wherein the not-structured data is a log comprising a plurality of records stored in a network-based data storage service, wherein the remote query engines are implemented as part of a not-structured data processing service, and wherein the data warehouse service, data storage service and not-structured data processing service are implemented as part of a provider network.

5. A method, comprising:

receiving a query on structured data and not-structured data, wherein the structured data is stored according to a pre-defined data model and the not-structured data is not stored according to a pre-defined data model;

generating a query execution plan for the query that comprises;

one or more stateful operations to apply the query to the structured data; and

stateless operations to apply the query to the not-structured data at one or more remote query processing engines;

causing performance of the stateless operations at the remote query processing engines as part of performing the query execution plan to execute the query on the structured data and not-structured data; and

returning a result for the query based, at least in part, on;

respective results for the stateless operations received from the remote query processing engines; and

respective results for the stateful operations to apply the query to the structured data.

6. The method of claim 5, wherein causing performance of the stateless operations at the remote query processing engines as part of performing the query execution plan to execute the query on the structured data and not-structured data comprises sending the stateless operations to the remote query processing engines according to a programmatic interface for the remote query processing engines.

7. The method of claim 6, wherein causing performance of the stateless operations at the remote query processing engines comprises sending a respective operation to a different remote query engine for one or more partitions of the not-structured data.

8. The method of claim 6, wherein returning the result for the query comprises:

determining that schema information for at least one of the respective results is received from the remote query processing engines along with the respective results; and

modifying execution of the query execution plan with respect to the at least one respective result according to the received schema information.

9. The method of claim 5,

wherein the not-structured data is a log comprising a plurality of log records;

wherein the method further comprises;

receiving the log from a data source;

storing log records of the log into different partitions in a remote data store according to a partitioning scheme; and

wherein causing performance of the stateless operations at the remote query processing engines comprises accessing, by the remote query processing engines, the remote data store to separately apply the query to one or more of the partitions.

10. The method of claim 9, wherein generating the query execution plan for the query that comprises:

determining that at least one of the partitions does not satisfy the query; and

removing a stateless operation from the query execution plan to apply the query to the at least one partition.

11. The method of claim 5, further comprising:

receiving the not-structured data from a data source;

extracting schema information for at least part of the not-structured data;

storing the extracted schema information in a separate metadata store; and

wherein generating the query execution plan for the query is based, at least in part, on the extracted schema information for the not-structured data.

12. The method of claim 5, wherein the query is performed in response to a data event detected for the not-structured data.

13. The method of claim 5, wherein the receiving, the generating, the causing, and the returning are performed by a query engine that is implemented as part of a network-based data warehouse service, wherein the not-structured data is stored in a network-based data storage service, and wherein the data warehouse service and the data storage service are implemented as part of a same provider network.

14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:

receiving a query on structured data and not-structured data, wherein the structured data is stored according to a pre-defined data model and the not-structured data is not stored according to a pre-defined data model;

generating a query execution plan for the query that comprises;

one or more stateful operations to apply the query to the structured data; and

stateless operations to apply the query to the not-structured data at one or more remote query processing engines;

causing performance of the stateless operations at the remote query processing engines as part of performing the query execution plan to execute the query on the structured data and not-structured data; and

returning a result for the query based, at least in part, on;

respective results for the stateless operations received from the remote query processing engines; and

respective results for the stateful operations to apply the query to the structured data.

15. The non-transitory, computer-readable storage medium of claim 14, wherein, in causing performance of the stateless operations at the remote query processing engines as part of performing the query execution plan to execute the query on the structured data and not-structured data, the program instructions cause sending the stateless operations to the remote query processing engines according to a programmatic interface for the remote query processing engines.

16. The non-transitory, computer-readable storage medium of claim 15, wherein, in returning the result for the query, the program instructions cause the one or more computing devices to implement:

determining that schema information for at least one of the respective results is received from the remote query processing engines along with the respective results; and

modifying execution of the query execution plan to the at least one respective result according to the received schema information.

17. The non-transitory, computer-readable storage medium of claim 14,

wherein the program instructions cause the one or more computing devices to further implement:

receiving the not-structured data from a data source;

storing data objects of the not-structured into different partitions in a remote data store according to a partitioning scheme; and

wherein, in causing performance of the stateless operations at the remote query processing engines, the program instructions cause the one or more computing devices to implement accessing, by the remote query processing engines, the remote data store to separately apply the query to one or more of the partitions.

18. The non-transitory, computer-readable storage medium of claim 17, wherein the program instructions cause the one or more computing devices to further implement:

extracting schema information for at least part of the not-structured data;

storing the extracted schema information in a separate metadata store; and

wherein generating the query execution plan for the query is based, at least in part, on the extracted schema information for the not-structured data.

19. The non-transitory, computer-readable storage medium of claim 14, wherein the structured data is a relational database table and wherein the not-structured data is log comprising a plurality of log records.

20. The non-transitory, computer-readable storage medium of claim 14, wherein the one or more computing devices implement a query engine as part of a network-based structured data processing service, wherein the not-structured data is stored in a network-based data storage service, and wherein the structured data processing service and the data storage service are implemented as part of a same provider network.