Field Oriented Pipeline Architecture For A Programmable Data Streaming Processor

• US 20100257537A1
• Filed: 04/06/2010
• Published: 10/07/2010
• Est. Priority Date: 09/18/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: Netezza Corp (International Business Machines Corporation)
• Inventors: Stewart, William K., Meyers, David L., Miller, Sharon L., Andraka, Raymond A., Hinshaw, Foster D.
• Granted Patent: US 8,880,551 B2
• Time in Patent Office: Days
• US Class Current: 718/105
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure

Abstract

A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

Claims

1. A distributed data processing system for executing database operations comprising:

(a) a first group including at least one host processor configured to accept queries for data stored on mass storage devices;

(b) a second group including two or more Job Processing Units (JPUs), each JPU configured to receive requests from the at least one host processor;

(c) a network infrastructure providing network connection of the first and second groups as network nodes, to support communication of data among network nodes, the network infrastructure using a communication protocol;

(d) a communication process responsible for communicating data blocks in one or more messages communicated among network nodes;

(e) an application process for executing database operations, with portions of the application process executing on the JPUs, and portions of the application process executing on at least one host processor; and

(f) the communication process implementing a load balancing function in front of the at least one host processors, the load balancing function directing individual transactions to selected ones of the at least one host processors so as to evenly distribute workload.