SYSTEM AND METHOD OF MASSIVELY PARALLEL DATA PROCESSING

• US 20100241646A1
• Filed: 05/21/2010
• Published: 09/23/2010
• Est. Priority Date: 03/18/2009
• Status: Active Grant

• Current Assignee: Teradata US, Inc. (Teradata Corporation)
• Original Assignee: Aster Data Systems, Inc. (Teradata Corporation)
• Inventors: Pawlowski, Peter, Friedman, Eric
• Granted Patent: US 8,903,841 B2
• Time in Patent Office: Days
• US Class Current: 707/764
• CPC Class Codes: G06F 16/24532 of parallel queries

Abstract

A system and method of massively parallel data processing are disclosed. In an embodiment, a method includes generating an interpretation of a customizable database request which includes an extensible computer process and providing an input guidance to available processors of an available computing environment. The method further includes automatically distributing an execution of the interpretation across the available computing environment operating concurrently and in parallel, wherein a component of the execution may be limited to at least a part of an input data. The method also includes automatically assembling a response using a distributed output of the execution.

Claims

1. A method comprising:

generating an interpretation of a customizable database request which includes an extensible computer process;

providing an input guidance to available processors of an available computing environment;

automatically distributing an execution of the interpretation across the available computing environment operating concurrently and in parallel, wherein a component of the execution is limited to at least a part of an input data; and

automatically assembling a response using a distributed output of the execution, wherein the input guidance is provided to each of the available processors and is comprised of certain portions of the input data, and wherein the input guidance is used to determine which of the available processors are to perform functions related to the at least the part of the input data.

2. The method of claim 1 further comprising:

providing an information to the extensible computer process about its context in the customizable database request; and

processing an interpretation of the customizable database request based on the information provided, wherein the extensible computer process is a developer provided-computer program, and wherein the information provided includes at least one of a format of the input data and an output data, whether the input data and the output data may is ordered and in which form, grouping information, statistics of the input data and the output data, a distribution information, a length of the input data and the output data, and a custom parameter, wherein the custom parameter is at least one of a number, a string, a list of numbers of strings, a content of a file in the available computing environment, and a result of the customizable database request; and

post processing an output of each of the available processors when automatically assembling the response, wherein the post processing includes at least one database operation including at least one of an aggregation operation, a sorting operation, and an invocation of another extensible computer process.

3. The method of claim 1 further comprising pre-processing an input of each of the available processors when providing the input guidance to the available processors, wherein the available computing environment is comprised of at least two servers, wherein the customizable database request specifies the input data for the extensible computer process,

wherein the input data is structured in a form comprising at least one of a database table and an output of a different database query,

wherein the input data is unstructured in a form comprising a content of at least one file in a computing environment.

4. The method of claim 1, further comprising:

detecting a fault in the execution of the interpretation; and

automatically rectifying an output effect of the fault, wherein rectifying the output effect of the fault includes at least one of reprocessing an operation, excluding a corrupted data, and logging the corrupted data,

wherein the customizable database request is comprised of at least one of a predetermined function, a developer created function, and an analyst created function.

5. A system comprising:

a query planning module to generate an interpretation of a database request which includes an extensible computer process;

a parallelization module to provide an information to available processors of an available computing environment and to automatically distribute an execution of the interpretation across the available computing environment operating concurrently and in parallel, wherein a component of the execution is limited to at least a part of an input data;

a response organization module to automatically assemble a response using a distributed output of the execution, wherein the information is used to provide each of the available processors certain portions of the input data, and to determine which of the available processors are to perform functions related to the at least the part of the input data;

a reference module to provide an extensible computer process information about its context in the database request; and

a dynamic interpretation module to process information that affects the interpretation of the database request based on the information provided, wherein the extensible computer process is a developer provided-computer program, and wherein the information provided includes a format of the input data and an output data, whether the input data and the output data is ordered and in which form, grouping information, statistics of the input data, a distribution information, a length of the input data and the output data, and custom parameters, wherein the custom parameters are at least one of a number, a string, a list of numbers of strings, a content of a file in the available computing environment, and a result of the database request.