Field oriented pipeline architecture for a programmable data streaming processor

• US 8,880,551 B2
• Filed: 04/06/2010
• Issued: 11/04/2014
• Est. Priority Date: 09/18/2002
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: IBM International Group BV (International Business Machines Corporation)
• Inventors: Hinshaw, Foster D., Andraka, Raymond A., Meyers, David L., Miller, Sharon L., Stewart, William K.
• Primary Examiner(s): PHAM, KHANH B
• Application Number:
  US12/755,052
• Publication Number:
  US 20100257537A1
• Time in Patent Office: 1,673 Days
• Field of Search: 707/769, 707/770, 718/105
• US Class Current: 707/770
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure

Abstract

A distributed data processing system executes database operations, and includes a first group of host processors and a second group of job processing units (JPUs). A software application for executing database operations executes in a distributed fashion with portions of the database application executing on at least one central database processor and other portions executing on the data storage processors. At least a portion of the database application is implemented within and/or coordinated by a communication process that is executing the communication protocol. The communication process implements a load balancing function in front of the host processors, the load balancing function directing individual transactions to selected processors so as to evenly distribute workload.

Claims

1. A distributed data processing system for executing database operations comprising:

(a) a first group including two or more host processors configured to accept queries with filtering criteria for data stored on mass storage devices of a database;

(b) a second group including two or more Job Processing Units (JPUs), each JPU configured to receive requests from the two or more host processors and including a first processor and a second processor, the second processor including an interface for accessing the mass storage devices of the database;

(c) a network infrastructure providing network connection of the first and second groups as network nodes, to support communication of data among network nodes, the network infrastructure using a communication protocol;

(d) a communication process responsible for communicating data blocks in one or more messages communicated among network nodes; and

(e) an application process for executing database operations, with portions of the application process executing on the JPUs, and portions of the application process executing on at least one of the two or more host processors, the portions executing on the JPUs including the second processor;

parsing streaming data in accordance with a format of the database, wherein the streaming data is received from the mass storage devices of the database via the interface; and

filtering the parsed data based on the filtering criteria of a corresponding query;

wherein the parsing and filtering are performed as the streaming data is being read from the mass storage devices of the database for transference of the filtered data from the second processor to the first processor.

2. A system as in claim 1 wherein each of the two or more JPUs provides exclusive access to at least one of the mass storage devices.

3. A system as in claim 1 wherein each of the two or more host processors provides the requests to the two or more JPUs based on the queries for data stored on the mass storage devices, the two or more JPUs providing a response to the host processors based on the request.

4. A system as in claim 1 wherein the second processor of each of the two or more JPUs comprises a programmable streaming data processor (PSDP).

5. A system as in claim 4 wherein the PSDP comprises at least two temporary registers for storing field delineated data from a field buffer prior to use of the field delineated data by a logic unit at the JPU.

6. A system as in claim 5 wherein:

a first data field is stored in a first register from the field buffer;

a second data field is stored into a second register; and

the logic unit is connected to compare a third data field from the field buffer with the first data field and a fourth data field from the field buffer with the second data field.

7. A system as in claim 6 wherein:

the logic unit compares the third data field from the field buffer with the first data field and the fourth data field from the field buffer with the second data field in two instructions.

8. A system as in claim 5 wherein:

a first data field is stored in a first temporary register from the field buffer;

a second data field is stored into a second temporary register; and

the logic unit is connected to compare a third data field from the field buffer with the first data field and with the second data field.

9. A system as in claim 8 wherein:

the logic unit compares the third data field from the field buffer with the first data field and with the second data field in a single instruction.

10. A system as in claim 4 wherein the PSDP comprises a data string register that stores data received from one of the two or more host processors to be used as an operand by a logic unit at the JPU.

11. A system as in claim 10 wherein:

at least one pointer specifies a location in the data string register to be used as the operand.

12. A system as in claim 11 wherein:

the logic unit is connected to compare a data field from a field buffer with a data field from the data string register as specified by a first pointer.

13. A system as in claim 12 wherein:

the data field from the field buffer is simultaneously compared with a second data field from the data string register specified by a second pointer.

14. A system as in claim 10 wherein:

at least one logic unit performs a bit vector join operation using an operand from the data string register to determine the presence or absence of a particular field value in field delineated data.

15. A system as in claim 1, wherein the communication process implements a load balancing function connected to the two or more host processors, the load balancing function directing individual transactions to selected ones of the two or more host processors so as to evenly distribute workload.

16. A method of executing database operations comprising:

operating a communication process to communicate data blocks in one or more messages communicated among two or more Job Processing Units (JPUs) and two or more host processors as network nodes, each host processor configured to accept queries with filtering criteria for data stored on mass storage devices of a database, each JPU configured to receive requests from the two or more host processors and including a first processor and a second processor, the second processor including an interface for accessing the mass storage devices of the database; and

operating an application process to execute database operations, with portions of the application process executing on the two or more JPUs, and portions of the application process executing on the two or more host processors, the portions executing on the JPUs including the second processor;

parsing streaming data in accordance with a format of the database, wherein the streaming data is received from the mass storage devices via the interface; and

filtering the parsed data based on the filtering criteria of a corresponding query;

wherein the parsing and filtering are performed as the streaming data is being read from the mass storage devices of the database for transference of the filtered data from the second processor to the first processor.

17. A method as in claim 16 wherein each of the two or more JPUs provides exclusive access to at least one of the mass storage devices.

18. A method as in claim 16 wherein each of the two or more host processors provides the requests to the two or more JPUs based on the queries for data stored on the mass storage devices, the two or more JPUs providing a response to the host processors based on the request.

19. A method as in claim 16, wherein the second processor of each of the two or more JPUs comprises a programmable streaming data processor (PSDP).

20. A method as in claim 19 further comprising storing field delineated data from a field buffer to at least two registers prior to use of the field delineated data by a logic unit at the JPU.

21. A method as in claim 20 additionally comprising:

storing a first data field in a first register from the field buffer;

storing a second data field into a second register; and

comparing, at the logic unit, a third data field from the field buffer with the first data field, and a fourth data field from the field buffer with the second data field.

22. A method as in claim 21 further comprising comparing, at the logic unit, the third data field from the field buffer with the first data field and the fourth data field from the field buffer with the second data field in two instructions.

23. A method as in claim 20 further comprising:

storing a first data field in a first temporary register from the field buffer;

storing a second data field into a second temporary register; and

comparing, at the logic unit, a third data field from the field buffer with the first data field and with the second data field.

24. A method as in claim 23 further comprising comparing, at the logic unit, the third data field from the field buffer with the first data field and with the second data field in a single instruction.

25. A method as in claim 19 further comprising storing data received from one of the two or more host processors to a data string register at the PSDP, the data to be used as an operand by a logic unit at the JPU.

26. A method as in claim 16, further comprising operating a load balancing function, the load balancing function directing individual transactions to selected ones of the two or more host processors so as to evenly distribute workload.

27. A computer program product for executing database operations comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured to;

operate a communication process to communicate data blocks in one or more messages communicated among two or more Job Processing Units (JPUs) and two or more host processors as network nodes, each host processor configured to accept queries with filtering criteria for data stored on mass storage devices of a database, each JPU configured to receive requests from the two or more host processors and including a first processor and a second processor, the second processor including an interface for accessing the mass storage devices of the database; and

operate an application process to execute database operations, with portions of the application process executing on the two or more JPUs, and portions of the application process executing on the two or more host processors, the portions executing on the JPUs including the second processor;

parsing streaming data in accordance with a format of the database, wherein the streaming data is received from the mass storage devices of the database via the interface; and

filtering the parsed data based on the filtering criteria of a corresponding query;

wherein the parsing and filtering are performed as the streaming data is being read from the mass storage devices of the database for transference of the filtered data from the second processor to the first processor.

28. A computer program product as in claim 27, wherein the computer readable program code is further configured to operate a load balancing function, the load balancing function directing individual transactions to selected ones of the two or more host processors so as to evenly distribute workload.