Intelligent storage device controller

• US 20040133565A1
• Filed: 09/18/2003
• Published: 07/08/2004
• Est. Priority Date: 09/18/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: Netezza (International Business Machines Corporation)
• Inventors: Andraka, Raymond A., Zane, Barry M., Hinshaw, Foster D., Stewart, William K., Meyers, David L., Miller, Sharon L., Sporer, Michael
• Granted Patent: US 7,730,077 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure

Abstract

A programmable streaming data processor that can be programmed to recognize record and field structures of data received from a streaming data source such as a mass storage device. Being programmed with, for example, field information, the unit can locate record and field boundaries and employ logical arithmetic methods to compare fields with one another or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The remaining records arrive and are discarded by the streaming data processor or are tagged with status bits to indicate to the more general purpose processor that they are to be ignored. In a preferred embodiment, the streaming data processor may analyze and discard records for several reasons. The first reason may be an analysis of contents of the field. Other reasons for record blocking may have to do with tagging records that are to be visible to particular users depending upon a series of concurrent transactions.

Claims

1. A Progammable Streaming Data Processor (PSDP) which is arranged to perform primitive functions directly on data received from a streaming data interface, PSDP performing initial processing on a set of data comprising:

a streaming data interface, for receiving data from a streaming data source;

a streaming interface First In First Out (FIFO), arranged for temporarily storing streaming data from the streaming data interface;

a data engine, arranged to receive output data from the streaming interface FIFO, the data engine for determining field boundaries therein, and for processing fields to select one or more fields to be output tuples, the data engine also containing logic to determine whether an output tuple is to be selected for further processing by additional processing Job Processing Units, and for asserting a use/lose decision value according to that determination;

a tuple generator for assembling fields into the output tuple, and if the use/lose decision value indicates that such output tuple is to be discarded, for preventing such tuple set from being transferred from the output FIFO to the memory of the JPU; and

an output First In First Out (FIFO) device, for forming tuples and temporarily storing them prior to conditionally forwarding them to the Job Processing Unit.

2. An apparatus as in claim 1 wherein the use/lose value indicates a result from logic processing of fields read from the streaming data interface.

3. An apparatus as in claim 1 wherein the use/lose decision value indicates a result from Transaction Identifier (TID) processing.

4. An apparatus as in claim 3 wherein the TID processing and data engine logic execute in parallel.

5. An apparatus as in claim 1 wherein the output tuple is greater in length than an expected predetermined size, and the use/lose decision value is then used to set an overflow field in the output tuple.

6. An apparatus as in claim 5 wherein the use/lose decision value is not asserted when a buffer local to the programmable data streaming processor is full;

and

means for appending an overflow filter bit to a tuple that indicates a transfer of a tuple that should be ignored.

7. An apparatus as in claim 1 additionally comprising:

a Direct Memory Access (DMA) interface, coupled to the output FIFO, to provide direct access to a memory in the Job Processing Unit.

8. An apparatus as in claim 1 wherein the use/lose value is used to reset the output FIFO write pointer so any prior fields in the present tuple are discarded.

9. An apparatus as in claim 1 wherein the overflow filter bit is inserted in a length field appended to record fragments.

10. An apparatus as in claim 1 wherein an invalid field is appended to a tuple to indicate the results of transaction ID processing.

11. An apparatus as in claim 10 wherein the invalid field indicates that the TID mode marks return tuple

12. An apparatus as in claim 10 wherein the invalid field indicates that the tuple should not have been returned but the output FIFO overflowed.

13. An apparatus as in claim 1 further comprising:

a register reflecting the final PSDP status which is read by the CPU to identify whether any overflow or TID status bits are set in any of the tuples.

14. An apparatus as in claim 1 wherein the use/lose decision value represents DeMorgan'"'"'s Law reduction of multiple instructions.