CUSTOM OPERATORS FOR A PARALLEL QUERY ENGINE

• US 20120066250A1
• Filed: 09/14/2010
• Published: 03/15/2012
• Est. Priority Date: 09/14/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Toub, Stephen Harris, Ostrovsky, Igor, Liddell, Mike
• Granted Patent: US 8,521,721 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/24532 of parallel queries

Abstract

Embodiments are directed to implementing custom operators in a query for a parallel query engine and to generating a partitioned representation of a sequence of query operators in a parallel query engine. A computer system receives a portion of partitioned input data at a parallel query engine, where the parallel query engine is configured to process data queries in parallel, and where the queries include a sequence of built-in operators. The computer system incorporates a custom operator into the sequence of built-in operators for a query and accesses the sequence of operators to determine how the partitioned input data is to be processed. The custom operator is accessed in the same manner as the built-in operators. The computer system also processes the sequence of operators including both the built-in operators and at least one custom operator according to the determination indicating how the data is to be processed.

Claims

1. At a computer system including a processor and a memory, in a computer networking environment including a plurality of computing systems, a computer-implemented method for implementing one or more custom operators in a query for a parallel query engine, the method comprising:

an act of receiving a portion of partitioned input data at a parallel query engine, wherein the parallel query engine is configured to process data queries in parallel, the queries including a sequence of built-in operators;

an act of incorporating at least one custom operator into the sequence of built-in operators for a query;

an act of accessing the sequence of operators to determine how the partitioned input data is to be processed, wherein the at least one custom operator is accessed in the same manner as the built-in operators; and

an act of processing the sequence of operators including both the built-in operators and at least one custom operator according to the determination indicating how the data is to be processed.

2. The method of claim 1, wherein the parallel query engine maintains a set of threads to pull data from a final query operator in the sequence of operators, and wherein the final query pulls data from the operator before it using the same set of threads.

3. The method of claim 2, wherein a processing thread is instantiated for each operator task that is ready for processing.

4. The method of claim 1, wherein the last operator in the sequence of operators polls its predecessor operator in the sequence of operators to determine the predecessor operator'"'"'s output information.

5. The method of claim 4, wherein the output information includes information regarding the number of partitions that are requestable by a subsequent operator.

6. The method of claim 4, wherein the output information includes one or more ordering guarantees that apply to the output of the custom operator.

7. The method of claim 6, wherein the custom operator is configured to determine whether changes have occurred that affect the ordering guarantees established and recorded in the representation of the partitioned data that forms the input to the custom operator.

8. The method of claim 7, wherein the custom operator is configured to modify at least a portion of the ordering guarantees that apply to the output of the custom operator, based on the determination that changes have occurred that affect the ordering guarantees associated with the input to the custom operator.

9. The method of claim 4, wherein the output information includes one or more configuration options identifying how the data is configured.

10. The method of claim 2, wherein one or more different execution mechanisms are employed including at least one of single-thread, multi-thread and one-thread-per-task parallelism.

11. The method of claim 1, wherein the custom operator is configured to re-partition the received input data into an arbitrary number of partitions.

12. The method of claim 4, further comprising complying with one or more execution time exception policies of the query processing engine.

13. The method of claim 12, further comprising:

an act of determining that processing of the sequence of operators has failed; and

an act of stopping the processing of the query based on the received exception management data.

14. A computer program product for implementing a method for generating a partitioned representation of a sequence of query operators in a parallel query engine, the computer program product comprising one or more computer-readable storage media having stored thereon computer-executable instructions that, when executed by one or more processors of the computing system, cause the computing system to perform the method, the method comprising:

an act of accessing a sequence of operators configured to process a portion of partitioned input data in a parallel query system, the sequence of operators comprising at least one custom operator;

an act of generating a list of the partitions into which the input data has been partitioned;

an act of determining the number of partitions that will be made during a re-partitioning operation at an indicated operator; and

an act of generating a partitioned representation of the sequence of query operators, wherein the partitioned representation provides internal information regarding the processing of the input data.

15. The computer program product of claim 14, wherein the operators are configured to re-partition the input data.

16. The computer program product of claim 15, wherein the input data is re-partitioned into a fewer number of partitions.

17. The computer program product of claim 14, further comprising:

an act of passing the partitioned representation to one or more other operators in the sequence of operators; and

an act of the receiving operators adding their own internal partition information to the partitioned representation.

18. The computer program product of claim 14, wherein the partitioned representation provides an indication of those characteristics that are necessary for operators to be included as part of the sequence of operators.

19. The computer program product of claim 14, wherein the partitioned representation provides an indication of those characteristics that are necessary for operators to be efficiently executed.

20. A computer system comprising the following:

one or more processors;

system memory;

one or more computer-readable storage media having stored thereon computer-executable instructions that, when executed by the one or more processors, causes the computing system to perform a method for implementing one or more custom operators in a query for a parallel query engine, the method comprising the following;

an act of receiving a portion of partitioned input data at a parallel query engine, wherein the parallel query engine is configured to process data queries in parallel, the queries including a sequence of built-in operators;

an act of incorporating at least one custom operator into the sequence of built-in operators for a query;

an act of accessing the sequence of operators to determine how the partitioned input data is to be processed, wherein the at least one custom operator is accessed in the same manner as the built-in operators; and

an act of processing the sequence of operators in reverse order such that the last operator in the sequence of operators polls its predecessor operator in the sequence of operators to determine the predecessor operator'"'"'s output information.