Intelligent data storage and processing using FPGA devices

• US 8,095,508 B2
• Filed: 05/21/2004
• Issued: 01/10/2012
• Est. Priority Date: 04/07/2000
• Status: Expired due to Term

• Current Assignee: Ip Reservoir, LLC
• Original Assignee: Washington University In St Louis
• Inventors: Chamberlain, Roger D., Franklin, Mark Allen, Indeck, Ronald S., Cytron, Ron K., Cholleti, Sharath R.
• Primary Examiner(s): Fleurantin, Jean B
• Application Number:
  US10/550,323
• Publication Number:
  US 20060294059A1
• Time in Patent Office: 2,790 Days
• Field of Search: 707600-831, 709/214, 711/103, 711/111, 711/112, 711/145, 713/189, 714/752, 705/51
• US Class Current: 707/636
• CPC Class Codes: G06F 16/90339 by using parallel associative memories or content-addressable memories
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 21/76 in application-specific integrated circuits [ASIC] or field-programmable devices, e.g. field-programmable gate arrays [FPGA] or programmable logic devices [PLD]

Abstract

A data storage and retrieval device and method is disclosed. The device includes at least one magnetic storage medium configured to store target data and at least one re-configurable logic device comprising an FPGA coupled to the at least one magnetic storage medium and configured to read a continuous stream of target data therefrom, having been configured with a template or as otherwise desired to fit the type of search and data being searched. The re-configurable logic device is configured to receive at least one search inquiry in the form of a data key and to determine a match between the data key and the target data as it is being read from the at least one magnetic storage medium. This device and method can perform a variety of searches on the target data including without limitation exact and approximate match searches, sequence match searches, image match searches and data reduction searches. This device and method may be provided as part of a stand-alone computer system, embodied in a network attached storage device, or can otherwise be provided as part of a computer LAN or WAN. In addition to performing search and data reduction operations, this device may also be used to perform a variety of other processing operations including encryption, decryption, compression, decompression, and combinations thereof.

Claims

1. A data processing system comprising:

a processing device; and

a computer system having a system bus, wherein the computer system is configured to communicate with the processing device over the system bus to control an operation of the processing device;

wherein the processing device comprises a re-configurable logic device configured to receive and process streaming data through a pipeline deployed on the re-configurable logic device, the pipeline comprising a plurality of pipelined data processing engines, the plurality of pipelined data processing engines being configured to perform different processing operations, wherein the pipeline comprises a multi-functional pipeline, and wherein the re-configurable logic device further comprises a control processor, wherein the control processor is configured to controllably activate or deactivate each of the pipelined data processing engines in the pipeline and thereby define a function for the pipeline, the pipeline function being the combined functionality of each activated pipelined data processing engine in the pipeline.

2. The system of claim 1 wherein the pipelined data processing engines in the pipeline comprise a decryption engine and a downstream search engine, and wherein the control processor is configured to controllably activate the decryption engine and the downstream search engine.

3. The system of claim 2 wherein the streaming data comprises data in an encrypted format, and wherein the pipeline is further configured to (1) receive the encrypted data stream, (2) decrypt the received encrypted data stream using the decryption engine to create a decrypted data stream, and (3) perform a search operation within the decrypted data stream using the search engine.

4. The system of claim 3 further comprising a data storage medium in which data is stored in the encrypted format, wherein the pipeline is configured to receive the encrypted data stream from the data storage medium, and wherein the search engine is configured to determine whether a pattern match exists between a search key that is representative of data desired to be retrieved from the data storage medium and a data signal that is representative of the decrypted data stream.

5. The system of claim 2 wherein the streaming data comprises data in an encrypted compressed format, and wherein the pipeline comprises a decompression engine positioned between the decryption engine and the search engine, wherein the control processor is further configured to also controllably activate the decompression engine, and wherein the pipeline is further configured to (1) receive the encrypted compressed data stream, (2) decrypt the received encrypted compressed data stream using the decryption engine to create a decrypted compressed data stream, (3) decompress the decrypted compressed data stream using the decompression engine to create a decompressed decrypted data stream, and (4) perform a search operation within the decompressed decrypted data stream using the search engine.

6. The system of claim 5 further comprising a data storage medium in which data is stored in the encrypted and compressed format, wherein the pipeline is configured to receive the encrypted compressed data stream from the data storage medium, and wherein the search engine is configured to determine whether a pattern match exists between a search key that is representative of data desired to be retrieved from the data storage medium and a data signal that is representative of the decompressed decrypted data stream.

7. The system of claim 1 wherein the re-configurable logic device comprises a Field Programmable Gate Array (FPGA).

8. The system of claim 1 wherein at least one of the pipelined data processing engines in the pipeline comprises a compression engine.

9. The system of claim 1 wherein at least one of the pipelined data processing engines in the pipeline comprises a decompression engine.

10. The system of claim 1 wherein at least one of the pipelined data processing engines in the pipeline comprises a data reduction engine.

11. The system of claim 10 wherein the streaming data comprises streaming financial information, the streaming financial information comprising data representative of a plurality of stocks and their associated prices, wherein the data reduction engine comprises a matching stage and a downstream summarization stage, wherein the matching stage is configured to search within the streaming financial information to find matching stocks of interest with respect to at least one data key, and wherein the summarization stage is configured to summarize the matching stocks in an aggregate form.

12. The system of claim 11 wherein the matching engine comprises:

a compare register, the compare register having a plurality of cells configured to store elements of the data key;

a data shift register, the data shift register having a plurality of cells configured to store elements of the streaming financial information, wherein each data shift register cell has a corresponding compare register cell;

a fine-grained comparison logic device, the fine-grained comparison logic device being configured to perform an element-by-element comparison as between the financial information elements and data key elements stored within corresponding ones of the data shift register cells and compare register cells;

word-level comparison logic, the word-level comparison logic being configured to find the matching stocks based on the element-by-element comparison from the fine-grained comparison logic device; and

wherein the data shift register is configured to continuously shift elements of the financial information from one data shift register cell to the next.

13. The system of claim 12 wherein the fine-grained comparison logic device comprises a plurality of cells, each fine-gained comparison logic cell being in communication with a compare register cell and at least one data shift register cell, and wherein the matching stage is further configured to route financial information elements from at least one of the data shift register cells to a plurality of different fine-grained comparison logic cells as the elements of the streaming financial information shift from one cell to the next within the data shift register to thereby support approximate matching.

14. The system of claim 11 wherein the summarization stage is configured to compute a minimum price for the stock prices of a matching stock found by the search stage.

15. The system of claim 14 wherein the summarization stage comprises:

a data shift register;

a data register configured to store a most recent minimum stock price; and

a comparator;

wherein the summarization stage is further configured to stream the stock prices for the matching stock through the data shift register;

wherein the comparator is configured to compare a current stock price in the data shift register with the stored most recent minimum stock price to determine which is lower; and

wherein the summarization stage is further configured to update the stored most recent minimum stock price in the data register with the current stock price in response to a determination by the comparator that the current stock price is lower.

16. The system of claim 11 wherein the summarization stage is configured to compute a maximum price for the stock prices of a matching stock found by the search stage.

17. The system of claim 16 wherein the summarization stage comprises:

a data shift register;

a data register configured to store a most recent maximum stock price; and

a comparator;

wherein the summarization stage is further configured to stream the stock prices for the matching stock through the data shift register;

wherein the comparator is configured to compare a current stock price in the data shift register with the stored most recent maximum stock price to determine which is higher; and

wherein the summarization stage is further configured to update the stored most recent maximum stock price in the data register with the current stock price in response to a determination by the comparator that the current stock price is higher.

18. The system of claim 11 wherein the streaming financial information further comprises data representative of an associated time for each stock price, and wherein the summarization stage is configured to compute a latest price for the stock prices of a matching stock found by the search stage.

19. The system of claim 18 wherein the summarization stage comprises:

a data shift register;

a first data register configured to store a most recent latest stock price;

a second data register configured to store a most recent time; and

a comparator;

wherein the summarization stage is further configured to stream the stock prices and associated times for the matching stock through the data shift register;

wherein the comparator is configured to compare a current time in the data shift register with the stored most recent time to determine which is later; and

wherein the summarization stage is further configured to update the stored most recent latest stock price in the data register with the current stock price in response to a determination by the comparator that the current time is later.

20. The system of claim 11 wherein the streaming financial information further comprises data representative of an associated time for each stock price, and wherein the summarization stage is configured to simultaneously compute a minimum price, a maximum price and a latest price for the stock prices of a matching stock found by the search stage.

21. The system of claim 20 wherein the summarization stage comprises:

a data shift register;

a first data register configured to store a most recent minimum stock price;

a second data register configured to store a most recent maximum stock price;

a third data register configured to store a most recent latest stock price;

a fourth data register configured to store a most recent time;

a first comparator;

a second comparator; and

a third comparator;

wherein the summarization stage is further configured to stream the stock prices and times for the matching stock through the data shift register;

wherein the first comparator is configured to compare the current stock price in the data shift register with the stored most recent minimum stock price to determine which is lower; and

wherein the summarization stage is further configured to update the stored most recent minimum stock price in the first data register with the current stock price in response to a determination by the first comparator that the current stock price is lower;

wherein the second comparator is configured to compare the current stock price in the data shift register with the stored most recent maximum stock price to determine which is higher;

wherein the summarization stage is further configured to update the stored most recent maximum stock price in the second data register with the current stock price in response to a determination by the second comparator that the current stock price is higher;

wherein the third comparator is configured to compare the current time in the data shift register with the stored most recent time to determine which is later; and

wherein the summarization stage is further configured to update the stored most recent latest stock price in the third data register with the current stock price in the data shift register in response to a determination by the third comparator that the current time is later.

22. The system of claim 11 further comprising:

a data storage medium in communication with the re-configurable logic device, the data storage medium being configured to at least temporarily store the financial information; and

wherein the re-configurable logic device is further configured to read the financial information from the data storage medium to thereby receive the financial information stream.

23. The system of claim 22 wherein the data storage medium comprises a mass storage medium.

24. The system of claim 23 wherein the mass storage medium comprises a magnetic storage device.

25. The system of claim 11 wherein the pipeline is configured to process the streaming financial information on a frame-by-frame basis.

26. The system of claim 11 wherein the pipeline is configured to process the streaming financial information on a frameless basis.

27. The system of claim 11 wherein the matching stage and the summarization stage are configured to perform their respective operations on different fields of the streaming financial information.

28. The system of claim 11 wherein the matching stage is configured return, in response to finding a match with respect to the data key, a portion of the streaming financial information within a bounding field that encompasses the matching stock for processing by the summarization stage.

29. The system of claim 11 wherein the control processor is further configured to controllably activate the data reduction engine and controllably deactivate another pipelined data processing engine in the pipeline.

30. The system of claim 29 wherein the deactivated another pipelined data processing engine in the pipeline is configured to process the streaming financial information as a pass through.

31. The system of claim 1 wherein at least one of the pipelined data processing engines in the pipeline comprises a data classification engine.

32. The system of claim 1 wherein the control processor is further configured to controllably deactivate at least one of the pipelined data processing engines in the pipeline, whereby the at least one deactivated pipelined data processing engine in the pipeline acts as a pass through for the data it receives.

33. The system of claim 32 wherein the pipelined data processing engines have an associated order within the pipeline, the order of the pipelined data processing engines in the pipeline remaining the same whether any of the pipelined data processing engines in the pipeline are deactivated.

34. The system of claim 1 wherein re-configurable logic device is further configured to receive streaming data flowing from a data storage medium and streaming data flowing to a data storage medium, wherein the control processor is further configured to controllably activate and deactivate the pipelined data processing engines in the pipeline differently based on a flow direction for the streaming data to define different pipeline functions for streaming data flowing from the data storage medium and for streaming data flowing to the data storage medium.

35. The system of claim 1 further comprising a data storage medium and a processor, both in communication with the re-configurable logic device, wherein the pipelined data processing engines in the pipeline comprise an encryption engine, a decryption engine, a decompression engine, a compression engine and a search engine;

wherein the control processor is further configured to transform the pipeline function for a data stream flowing to the data storage medium by (1) controllably activating the encryption engine and the compression engine, and (2) controllably deactivating the decryption engine, the decompression engine and the search engine to render the decryption engine, the decompression engine and the search engine as pass through stages within the pipeline such that the pipeline will be configured to create an output data stream of encrypted and compressed data from an input data stream; and

wherein the control processor is further configured to transform the pipeline function for a data stream of encrypted and compressed data flowing from the data storage medium by (1) controllably activating the decryption engine, the decompression engine and the search engine, and (2) controllably deactivating the encryption engine and the compression engine to render the encryption engine and the compression engine as pass through stages within the pipeline such that the pipeline will be configured to perform a search operation in response to a search inquiry from the processor within decrypted and decompressed data.

36. The system of claim 35 wherein the pipeline comprises the (1) decryption engine positioned downstream from the encryption engine, (2) the decompression engine positioned downstream from the decryption engine and the encryption engine, (3) the compression engine positioned downstream from the decompression engine, the decryption engine and the encryption engine, and (4) the search engine positioned downstream from the compression engine, the decompression engine, the decryption engine and the encryption engine, all with reference to a stream of data flowing from the data storage medium.

37. The system of claim 1 wherein the pipeline is configured to process the streaming data on a frame-by-frame basis.

38. The system of claim 1 wherein the pipeline is configured to process the streaming data on a frameless basis.

39. The system of claim 1 wherein the re-configurable logic device comprises a Field Programmable Gate Array (FPGA), wherein the pipeline is resident on the FPGA, and wherein the control processor comprises a firmware socket module resident on the FPGA.

40. The system of claim 1 wherein the computer system further comprises a processor, and wherein the control processor is configured to controllably activate or deactivate the pipelined data processing engines in the pipeline in response to instructions received from the processor.

41. The system of claim 40 wherein the processor comprises a Central Processing Unit (CPU) configured to provide the instructions to the control processor in response to software execution.

42. The system of claim 1 further comprising a data storage medium in communication with the re-configurable logic device for storing an output from the pipeline.

43. A data processing system of comprising:

a data storage medium, wherein the data storage medium comprises a disk drive system for magnetically storing data, the disk drive system comprising;

a rotatable disk upon which data is magnetically stored in a plurality of discontiguous arcs, wherein each arc possesses a substantially constant curvature, the plurality of discontiguous arcs together defining a generally helical pattern about a central origin;

a device for rotating the disk when data is to be read therefrom;

a read head positioned for reading the data stored on the disk as the disk rotates; and

a positioning system configured to position the read head over the disk such that, as the disk rotates, the read head follows the generally helical pattern of the discontiguous arcs;

a processing device in communication with the data storage medium; and

a computer system having a system bus, wherein the computer system is configured to communicate with the processing device over the system bus;

wherein the processing device comprises a programmable logic device configured to process data, as the data passes between the data storage medium and the computer system, through a plurality of stages implemented on the programmable logic device as a processing pipeline, each processing stage being dedicated to a different processing operation; and

wherein the processing operations comprise at least two selected from the group consisting of a search operation, a data reduction operation, a data classification operation, an encryption operation, a decryption operation, a compression operation, and a decompression operation.

44. A data processing system comprising:

a data storage medium, wherein a plurality of data files are stored in the data storage medium, each data file being stored as a sequence of segments, each segment having a size that is a power of 2;

a processing device in communication with the data storage medium; and

a computer system having a system bus, wherein the computer system is configured to communicate with the processing device over the system bus;

wherein the processing device comprises a programmable logic device configured to process data, as the data passes between the data storage medium and the computer system, through a plurality of stages implemented on the programmable logic device as a processing pipeline, each processing stage being dedicated to a different processing operation; and

wherein the processing operations comprise at least two selected from the group consisting of a search operation, a data reduction operation, a data classification operation, an encryption operation, a decryption operation, a compression operation, and a decompression operation.

45. A hard disk drive accelerator for connection between a hard disk drive and a processor, said accelerator comprising a reconfigurable logic device arranged such that data read from the hard disk drive streams through the reconfigurable logic device prior to being passed on to the processor, wherein the reconfigurable logic device is configured to process the data stream through a pipeline deployed thereon, the pipeline comprising a plurality of pipelined data processing engines, each pipelined data processing engine being configured to perform a data processing operation on received data, and wherein the pipeline comprises a multi-functional pipeline, and wherein the reconfigurable logic device further comprises a control processor, wherein the control processor is configured to controllably activate or deactivate each of the pipelined data processing engines in the pipeline and thereby define a function for the pipeline, the pipeline function being the combined functionality of each activated pipelined data processing engine in the pipeline.

46. The accelerator of claim 45 wherein the reconfigurable logic device comprises a programmable logic device, wherein the pipelined data processing engines in the pipeline comprise a decryption engine and a downstream search engine, wherein the control processor is further configured to controllably activate the decryption engine and the search engine, wherein the hard disk drive comprises data stored therein in an encrypted format, and wherein the programmable logic device is configured to (1) receive a stream of encrypted data from the hard disk drive, (2) decrypt the received encrypted data stream using the decryption engine to create a decrypted data stream, and (3) perform a search operation within the decrypted data stream using the search engine.

47. The accelerator of claim 46 wherein the search engine is configured to determine whether a pattern match exists between a search key that is representative of data desired to be retrieved from the hard disk drive and a data signal that is representative of the decrypted data stream.

48. The accelerator of claim 45 wherein the reconfigurable logic device comprises a programmable logic device, wherein the pipelined data processing engines in the pipeline comprise a decryption engine, a downstream search engine and a decompression engine positioned between the decryption engine and the search engine, wherein the hard disk drive comprises data stored therein in an encrypted compressed format, and wherein the programmable logic device is configured to (1) receive a stream of encrypted compressed data from the hard disk drive, (2) decrypt the received stream using the decryption engine to create a decrypted compressed data stream, (3) decompress the decrypted compressed data stream using the decompression engine to create a decompressed decrypted data stream, and (4) perform a search operation within the decompressed decrypted data stream using the search engine.

49. The accelerator of claim 48 wherein the search engine is configured to determine whether a pattern match exists between a search key that is representative of data desired to be retrieved from the hard disk drive and a data signal that is representative of the decompressed decrypted data stream.

50. The accelerator of claim 45 wherein the reconfigurable logic device comprises a Field Programmable Gate Array (FPGA).

51. The accelerator of claim 45 wherein at least one of the pipelined data processing engines in the pipeline comprises a search engine.

52. The accelerator of claim 45 wherein at least one of the pipelined data processing engines in the pipeline comprises a compression engine.

53. The accelerator of claim 45 wherein at least one of the pipelined data processing engines in the pipeline comprises a decompression engine.

54. The accelerator of claim 45 wherein at least one of the pipelined data processing engines in the pipeline comprises a data reduction engine.

55. The accelerator of claim 45 wherein at least one of the pipelined data processing engines in the pipeline comprises a data classification engine.

56. The accelerator of claim 45 wherein the pipelined data processing engines in the pipeline comprises at least two selected from the group consisting of a search engine, a data reduction engine, a data classification engine, an encryption engine, a decryption engine, a compression engine, and a decompression engine.

57. The accelerator of claim 45 wherein the control processor is further configured to controllably activate at least one of the pipelined data processing engines in the pipeline and controllably deactivate at least one of the other pipelined data processing engines in the pipeline.

58. The accelerator of claim 57 wherein the at least one deactivated pipelined data processing engine in the pipeline is configured to process the streaming data as a pass through.

59. The accelerator of claim 45 wherein reconfigurable logic device is further arranged such that data flowing to the hard disk drive streams through the reconfigurable logic device prior to being stored in the hard disk drive, wherein the control processor is further configured to controllably activate and deactivate the pipelined data processing engines in the pipeline differently based on a flow direction for the streaming data to define different pipeline functions for streaming data flowing from the hard disk drive and for streaming data flowing to the hard disk drive.

60. The accelerator of claim 45 wherein reconfigurable logic device is further arranged such that data flowing to the hard disk drive streams through the reconfigurable logic device prior to being stored in the hard disk drive, wherein the pipelined data processing engines in the pipeline comprise an encryption engine, a decryption engine, a decompression engine, a compression engine and a search engine;

wherein the control processor is further configured to transform the pipeline function for a data stream flowing to the hard disk drive by (1) controllably activating the encryption engine and the compression engine, and (2) controllably deactivating the decryption engine, the decompression engine and the search engine to render the decryption engine, the decompression engine and the search engine as pass through stages within the pipeline such that the pipeline will be configured to create an output data stream of encrypted and compressed data from an input data stream; and

wherein the control processor is further configured to transform the pipeline function for a data stream of encrypted and compressed data flowing from the hard disk drive by (1) controllably activating the decryption engine, the decompression engine and the search engine, and (2) controllably deactivating the encryption engine and the compression engine to render the encryption engine and the compression engine as pass through stages within the pipeline such that the pipeline will be configured to perform a search operation in response to a search inquiry from the processor within decrypted and decompressed data.

61. The accelerator of claim 45 wherein the pipeline is configured to process the streaming data on a frame-by-frame basis.

62. The accelerator of claim 45 wherein the pipeline is configured to process the streaming data on a frameless basis.

63. The accelerator of claim 45 wherein the control processor is configured to controllably activate or deactivate the pipelined data processing engines in the pipeline in response to instructions received from the processor.

64. A data processing system comprising:

a processing card for communication with a processor and a data source via a bus; and

wherein the processing card comprises a re-configurable logic device, the re-configurable logic device comprising a multi-functional pipeline for processing streaming data received by the data processing card from the data source, the pipeline comprising a plurality of different pipelined data processing engines;

wherein each of the pipelined data processing engines in the pipeline is configured to (1) be selectively activated and deactivated in response to control instructions to define a function for the pipeline, and (2) perform a processing operation on the streaming data in response to being activated, the pipeline function being the combined functionality of the activated pipelined data processing engines in the pipeline such that the pipeline is changeable to another function of a plurality of functions in response to control instructions that change which of the pipelined data processing engines in the pipeline are activated and deactivated.

65. The system of claim 64 wherein the re-configurable logic device comprises a Field Programmable Gate Array (FPGA).

66. The system of claim 64 wherein pipeline comprises a data reduction engine as an activated pipelined data processing engine in the pipeline, wherein the streaming data comprises streaming financial information, the streaming financial information comprising data representative of a plurality of stocks and their associated prices, wherein the data reduction engine comprises a matching stage and a downstream summarization stage, wherein the matching stage is configured to search within the streaming financial information to find matching stocks of interest with respect to at least one data key, and wherein the summarization stage is configured to summarize the matching stocks in an aggregate form.

67. The system of claim 66 wherein the matching engine comprises:

a compare register, the compare register having a plurality of cells configured to store elements of the data key;

a data shift register, the data shift register having a plurality of cells configured to store elements of the streaming financial information, wherein each data shift register cell has a corresponding compare register cell;

a fine-grained comparison logic device, the fine-grained comparison logic device being configured to perform an element-by-element comparison as between the financial information elements and data key elements stored within corresponding ones of the data shift register cells and compare register cells;

word-level comparison logic, the word-level comparison logic being configured to find the matching stocks based on the element-by-element comparison from the fine-grained comparison logic device; and

wherein the data shift register is configured to continuously shift elements of the financial information from one data shift register cell to the next.

68. The system of claim 67 wherein the fine-grained comparison logic device comprises a plurality of cells, each fine-grained comparison logic cell being in communication with a compare register cell and at least one data shift register cell, and wherein the matching stage is further configured to route financial information elements from at least one of the data shift register cells to a plurality of different fine-grained comparison logic cells as the elements of the streaming financial information shift from one cell to the next within the data shift register to thereby support approximate matching.

69. The system of claim 66 wherein the summarization stage is configured to compute a minimum price for the stock prices of a matching stock found by the search stage.

70. The system of claim 69 wherein the summarization stage comprises:

a data shift register;

a data register configured to store a most recent minimum stock price; and

a comparator;

wherein the summarization stage is further configured to stream the stock prices for the matching stock through the data shift register;

wherein the comparator is configured to compare a current stock price in the data shift register with the stored most recent minimum stock price to determine which is lower; and

wherein the summarization stage is further configured to update the stored most recent minimum stock price in the data register with the current stock price in response to a determination by the comparator that the current stock price is lower.

71. The system of claim 66 wherein the summarization stage is configured to compute a maximum price for the stock prices of a matching stock found by the search stage.

72. The system of claim 71 wherein the summarization stage comprises:

a data shift register;

a data register configured to store a most recent maximum stock price; and

a comparator;

wherein the summarization stage is further configured to stream the stock prices for the matching stock through the data shift register;

wherein the comparator is configured to compare a current stock price in the data shift register with the stored most recent maximum stock price to determine which is higher; and

wherein the summarization stage is further configured to update the stored most recent maximum stock price in the data register with the current stock price in response to a determination by the comparator that the current stock price is higher.

73. The system of claim 66 wherein the streaming financial information further comprises data representative of an associated time for each stock price, and wherein the summarization stage is configured to compute a latest price for the stock prices of a matching stock found by the search stage.

74. The system of claim 73 wherein the summarization stage comprises:

a data shift register;

a first data register configured to store a most recent latest stock price;

a second data register configured to store a most recent time; and

a comparator;

wherein the summarization stage is further configured to stream the stock prices and associated times for the matching stock through the data shift register;

wherein the comparator is configured to compare a current time in the data shift register with the stored most recent time to determine which is later; and

wherein the summarization stage is further configured to update the stored most recent latest stock price in the data register with the current stock price in response to a determination by the comparator that the current time is later.

75. The system of claim 66 wherein the streaming financial information further comprises data representative of an associated time for each stock price, and wherein the summarization stage is configured to simultaneously compute a minimum price, a maximum price and a latest price for the stock prices of a matching stock found by the search stage.

76. The system of claim 75 wherein the summarization stage comprises:

a data shift register;

a first data register configured to store a most recent minimum stock price;

a second data register configured to store a most recent maximum stock price;

a third data register configured to store a most recent latest stock price;

a fourth data register configured to store a most recent time;

a first comparator;

a second comparator; and

a third comparator;

wherein the summarization stage is further configured to stream the stock prices and times for the matching stock through the data shift register;

wherein the first comparator is configured to compare the current stock price in the data shift register with the stored most recent minimum stock price to determine which is lower; and

wherein the summarization stage is further configured to update the stored most recent minimum stock price in the first data register with the current stock price in response to a determination by the first comparator that the current stock price is lower;

wherein the second comparator is configured to compare the current stock price in the data shift register with the stored most recent maximum stock price to determine which is higher;

wherein the summarization stage is further configured to update the stored most recent maximum stock price in the second data register with the current stock price in response to a determination by the second comparator that the current stock price is higher;

wherein the third comparator is configured to compare the current time in the data shift register with the stored most recent time to determine which is later; and

wherein the summarization stage is further configured to update the stored most recent latest stock price in the third data register with the current stock price in the data shift register in response to a determination by the third comparator that the current time is later.

77. The system of claim 66 wherein the data source comprises a mass storage medium configured to at least temporarily store the financial information, and wherein the re-configurable logic device is further configured to read the financial information from the mass storage medium to thereby receive the financial information stream.

78. The system of claim 77 wherein the mass storage medium comprises a magnetic storage device.

79. The system of claim 66 wherein the pipeline is configured to process the streaming financial information on a frame-by-frame basis.

80. The system of claim 66 wherein the pipeline is configured to process the streaming financial information on a frameless basis.

81. The system of claim 66 wherein the matching stage and the summarization stage are configured to perform their respective operations on different fields of the streaming financial information.

82. The system of claim 66 wherein the matching stage is configured return, in response to finding a match with respect to the data key, a portion of the streaming financial information within a bounding field that encompasses the matching stock for processing by the summarization stage.

83. The system of claim 66 wherein the re-configurable logic device further comprises a control processor, wherein the control processor is configured to controllably activate the data reduction engine and controllably deactivate another pipelined data processing engine in the pipeline.

84. The system of claim 83 wherein the deactivated another pipelined data processing engine is configured to process the streaming financial information as a pass through.

85. The system of claim 64 wherein the re-configurable logic device further comprises a control processor, wherein the data source is also configured to receive data processed by the re-configurable logic device, wherein re-configurable logic device is further configured to receive streaming data flowing from the data source and streaming data flowing to the data source, wherein the control processor is configured to controllably activate and deactivate the pipelined data processing engines in the pipeline differently based on a flow direction for the streaming data to define different pipeline functions for streaming data flowing from the data source and for streaming data flowing to the data source.

86. The system of claim 64 wherein the re-configurable logic device further comprises a control processor, wherein the data source is also configured to receive data processed by the re-configurable logic device, wherein the pipelined data processing engines in the pipeline comprise an encryption engine, a decryption engine, a decompression engine, a compression engine and a search engine;

wherein the control processor is configured to transform the pipeline function for a data stream flowing to the data source by (1) controllably activating the encryption engine and the compression engine, and (2) controllably deactivating the decryption engine, the decompression engine and the search engine to render the decryption engine, the decompression engine and the search engine as pass through stages within the pipeline such that the pipeline will be configured to create an output data stream of encrypted and compressed data from an input data stream; and

wherein the control processor is further configured to transform the pipeline function for a data stream of encrypted and compressed data flowing from the data source by (1) controllably activating the decryption engine, the decompression engine and the search engine, and (2) controllably deactivating the encryption engine and the compression engine to render the encryption engine and the compression engine as pass through stages within the pipeline such that the pipeline will be configured to perform a search operation in response to a search inquiry from the processor within decrypted and decompressed data.

87. The system of claim 64 wherein the pipeline is configured to process the streaming data on a frame-by-frame basis.

88. The system of claim 64 wherein the pipeline is configured to process the streaming data on a frameless basis.

89. The system of claim 64 wherein the re-configurable logic device further comprises a control processor, wherein the control processor is configured to controllably activate or deactivate the pipelined data processing engines in the pipeline in response to instructions received from the processor.

90. A data processing method comprising:

within a computer system comprising a processor and a reconfigurable logic device operating under control of the processor, streaming data through the reconfigurable logic device for processing thereby, the reconfigurable logic device comprising a multi-functional pipeline, the multi-functional pipeline comprising a control processor and a plurality of pipelined data processing engines, each of the pipelined data processing engines in the pipeline being configured to (1) receive streaming data and (2) perform a data processing operation on the received streaming data;

the control processor selectively activating and deactivating the pipelined data processing engines in the pipeline to achieve a desired pipeline function, the pipeline function being the combined functionality of the activated pipelined data processing engines in the pipeline; and

the activated pipelined data processing engines in the pipeline performing their data processing operations on the streaming data at hardware speeds.

91. The method of claim 90 wherein the plurality of pipelined data processing engines have an associated order within the pipeline, the order of the pipelined data processing engines in the pipeline remaining the same regardless of whether any of the pipelined data processing engines in the pipeline are deactivated.

92. The method of claim 91 wherein each deactivated pipelined data processing engine within the pipeline is configured to process any received streaming data as a pass through.

93. The method of claim 90 wherein pipeline comprises a data reduction engine, wherein the control processor activating step comprises the control processor activating the data reduction engine, wherein the streaming data comprises streaming financial information, the streaming financial information comprising data representative of a plurality of stocks and their associated prices, wherein the data reduction engine comprises a matching stage and a downstream summarization stage, and wherein the performing step comprises (1) the matching stage searching within the streaming financial information to find matching stocks of interest with respect to at least one data key, and (2) the summarization stage summarizing the matching stocks in an aggregate form.

94. The method of claim 93 wherein the summarizing step comprises the summarization stage computing a minimum price for the stock prices of a matching stock found by the searching step.

95. The method of claim 93 wherein the summarizing step comprises the summarization stage computing a maximum price for the stock prices of a matching stock found by the searching step.

96. The method of claim 93 wherein the streaming financial information further comprises data representative of an associated time for each stock price, and wherein the summarizing step comprises the summarization stage computing a latest price for the stock prices of a matching stock found by the searching step.

97. The method of claim 93 wherein the streaming financial information further comprises data representative of an associated time for each stock price, and wherein the summarizing step comprises the summarization stage simultaneously computing a minimum price, a maximum price and a latest price for the stock prices of a matching stock found by the searching step.

98. The method of claim 93 wherein the method further comprises receiving the streaming financial information from a data source.

99. The method of claim 98 wherein the data source comprises a magnetic storage device, and wherein the receiving step comprises reading the financial information from the magnetic storage device to thereby receive the financial information stream.

100. The method of claim 93 wherein the performing step comprises the data reduction engine processing the streaming financial information on a frame-by-frame basis.

101. The method of claim 93 wherein the performing step comprises the data reduction engine processing the streaming financial information on a frameless basis.

102. The method of claim 93 wherein the searching step and the summarizing step operate on different fields of the streaming financial information.