System and method of massively parallel data processing

• US 8,903,841 B2
• Filed: 05/21/2010
• Issued: 12/02/2014
• Est. Priority Date: 03/18/2009
• Status: Active Grant

• Current Assignee: Teradata US, Inc. (Teradata Corporation)
• Original Assignee: Teradata US, Inc. (Teradata Corporation)
• Inventors: Friedman, Eric, Pawlowski, Peter
• Primary Examiner(s): KINSAUL, DANIEL W
• Application Number:
  US12/784,527
• Publication Number:
  US 20100241646A1
• Time in Patent Office: 1,656 Days
• US Class Current: 707/764
• CPC Class Codes: G06F 16/24532 of parallel queries

Abstract

A system and method of massively parallel data processing are disclosed. In an embodiment, a method includes generating an interpretation of a customizable database request which includes an extensible computer process and providing an input guidance to available processors of an available computing environment. The method further includes automatically distributing an execution of the interpretation across the available computing environment operating concurrently and in parallel, wherein a component of the execution may be limited to at least a part of an input data. The method also includes automatically assembling a response using a distributed output of the execution.

Claims

1. A method of interpreting a customizable database request through a computing system comprising:

receiving the customizable database request;

generating, in a data processing device of the computing system, an interpretation of the customizable database request through an input data processing device of the computing system configured to execute an extensible computer process thereon, the data processing device being one of the input data processing device and another data processing device of the computing system, wherein the data processing device is configured to convert a plurality of programming languages into a preferred programming language, and wherein the data processing device executes instructions that convert the customizable database request into the preferred programming language when the customizable database request is in a programming language from the plurality of programming languages different from the preferred programming language, wherein the interpretation includes alternate responses to the customizable database request;

providing an input guidance to two or more processors of the computing system;

automatically distributing an execution of the interpretation in the computing system to operate concurrently, wherein a component of the execution is limited to at least a part of an input data associated with the customizable database request;

automatically assembling a response from the alternate responses using a distributed output of the concurrent execution, wherein the input guidance comprises certain portions of the input data, and wherein the customizable database request includes a parameter comprising one or both of a prediction or expectation regarding the response, and wherein the input guidance and the parameter are used to determine which of the two or more processors is to perform functions related to the at least the part of the input data;

detecting a fault in the execution of the interpretation; and

automatically rectifying an output effect of the fault, wherein rectifying the output effect of the fault includes reprocessing an operation, excluding a corrupted data, and logging the corrupted data, wherein the customizable database request comprises at least one of a predetermined function, a developer created function, and an analyst created function.

2. The method of claim 1, further comprising:

providing an information to the extensible computer process about a context thereof in the customizable database request;

processing an interpretation of the customizable database request based on the information provided, wherein the extensible computer process is a developer provided-computer program, and wherein the information provided includes at least one of a format of the input data and an output data, whether the input data and the output data are ordered and in which form, grouping information, statistics of the input data and the output data, a distribution information, a length of the input data and the output data, and a custom parameter, wherein the custom parameter is at least one of a number, a string, a list of numbers of strings, a content of a file in the available computing environment, and a result of the customizable database request; and

post-processing an output of each of the processors when automatically assembling the response, wherein the post-processing includes at least one database operation including at least one of an aggregation operation, a sorting operation, and an invocation of another extensible computer process.

3. The method of claim 1, further comprising pre-processing an input of each of the processors when providing the input guidance thereto wherein the customizable database request specifies the input data for the extensible computer process, wherein the input data is structured in a form comprising at least one of a database table and an output of a different database query, and wherein the input data is unstructured in a form comprising a content of at least one file in a computing environment.

4. A computing system comprising:

two or more processors;

at least one data processing device comprising at least one processor associated therewith, receiving a customizable database request and configured to execute an extensible computer process thereon, wherein the at least one data processing device is configured convert a plurality of programming languages to a preferred programming language, and wherein the at least one data processing device executes instructions that convert the customizable database request into the preferred programming language when the customizable database request is in a programming language from the plurality of programming languages different from the preferred programming language, and wherein the plurality of programming languages includes at least one non-database query language, and wherein the at least one data processing device is configured to;

generate an interpretation of the customizable database request,

provide input guidance to the two or more processors of the computing system,

automatically distribute an execution of the interpretation to the two or more processors of the computing system to operate concurrently, wherein a component of the execution is limited to at least a part of an input data associated with the customizable database request,

automatically assemble a response using a distributed output of the concurrent execution, wherein the input guidance comprises certain portions of the input data, and wherein the customizable database request includes a parameter comprising one or both of a prediction or expectation regarding the response, and wherein the input guidance and the parameter are used to determine which of the two or more processors of the computing system is to perform functions related to the at least the part of the input data,

provide the extensible computer process information about a context thereof in the database request,

process information that affects the interpretation of the database request based on the information provided,

detect a fault in the execution of the interpretation, and automatically rectify an output effect of the fault, wherein rectifying the output effect of the fault includes reprocessing an operation, excluding a corrupted data, and logging the corrupted data, wherein the customizable database request comprises at least one of a predetermined function, a developer created function, and an analyst created function, wherein the extensible computer process is a developer provided-computer program, and wherein the information provided includes a format of the input data and an output data.