Method and System for High Performance Integration, Processing and Searching of Structured and Unstructured Data Using Coprocessors

• US 20080114724A1
• Filed: 11/12/2007
• Published: 05/15/2008
• Est. Priority Date: 11/13/2006
• Status: Active Grant

• Current Assignee: Ip Reservoir, LLC
• Original Assignee: Exegy Incorporated
• Inventors: Indeck, Ronald S., Indeck, David Mark
• Granted Patent: US 7,660,793 B2
• Time in Patent Office: Days
• US Class Current: 707/2
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Disclosed herein is a method and system for integrating an enterprise'"'"'s structured and unstructured data to provide users and enterprise applications with efficient and intelligent access to that data. Queries can be directed toward both an enterprise'"'"'s structured and unstructured data using standardized database query formats such as SQL commands. A coprocessor can be used to hardware-accelerate data processing tasks (such as full-text searching) on unstructured data as necessary to handle a query. Furthermore, traditional relational database techniques can be used to access structured data stored by a relational database to determine which portions of the enterprise'"'"'s unstructured data should be delivered to the coprocessor for hardware-accelerated data processing.

Claims

1. A method of providing integrated access to both structured and unstructured data, the method comprising:

receiving a query directed toward a combination of structured and unstructured data;

retrieving structured data from a database of structured data in accordance with the received query;

retrieving unstructured data from a data store of unstructured data based on the retrieved structured data;

streaming the retrieved unstructured data through a processing device in a computing system other than a main processor for the system; and

performing a query-specified data processing operation on the streaming unstructured data using the processing device.

2. The method of claim 1 wherein the processing device comprises a coprocessor.

3. The method of claim 2 wherein the coprocessor comprises a reconfigurable logic device.

4. The method of claim 3 wherein the reconfigurable logic device has firmware deployed thereon, the firmware being configured to perform the query-specified data processing operation.

5. The method of claim 1 wherein the query comprises a standardized database query.

6. The method of claim 5 wherein the standardized database query comprises a SQL command.

7. The method of claim 6 wherein the query-specified data processing operation comprises a text searching operation based on at least one key found in the query.

8. The method of claim 1 wherein the structured data comprises metadata about the unstructured data in the data store of unstructured data.

9. The method of claim 8 further comprising:

streaming new unstructured data through the processing device;

performing a metadata generation operation on the streaming unstructured data using the processing device to thereby generate metadata about the new unstructured data;

storing the new unstructured data in the data store of unstructured data; and

storing the metadata about the new unstructured data in the database of structured data.

10. The method of claim 9 wherein the processing device comprises a coprocessor.

11. The method of claim 10 wherein the coprocessor comprises a reconfigurable logic device.

12. The method of claim 11 wherein the reconfigurable logic device has firmware deployed thereon, the firmware being configured to perform the metadata generation operation.

13. The method of claim 8 wherein the database of structured data comprises a RDBMS.

14. The method of claim 1 wherein the structured data retrieving step comprises retrieving structured data from an external database of structured data in accordance with the received query.

15. The method of claim 14 wherein the structured data retrieving step further comprises also retrieving structured data from an internal database of structured data in accordance with the received query.

16. The method of claim 1 wherein the structured data retrieving step comprises retrieving structured data from an internal database of structured data in accordance with the received query.

19. The system of claim 16 wherein the coprocessor comprises a reconfigurable logic device.

20. The system of claim 19 wherein the reconfigurable logic device has firmware deployed thereon, the firmware being configured to perform the query-specified data processing operation.

17. A system for processing data, the system comprising:

a main processor;

a processing device other than the main processor;

a data store of unstructured data in communication with the main processor and the processing device;

a data store of structured data in communication with the main processor and the processing device;

wherein the main processor is configured to (1) receive a query, (2) process at least a portion of the query against the structured data in the data store of structured data to identify a subset of unstructured data in the data store of unstructured data that satisfies at least a portion of the query, and (3) request that the subset of unstructured data be delivered to the processing device; and

wherein the processing device is configured to perform a query-specified data processing operation on the subset of unstructured data.

18. The system of claim 17 wherein the processing device comprises a coprocessor.

21. The system of claim 17 wherein the database of structured data comprises a RDBMS, and wherein the query comprises a standardized database query.

22. The system of claim 21 wherein the structured data within the RDBMS comprises metadata about the unstructured data within the data store of unstructured data.

23. The system of claim 22 wherein the standardized database query comprises a SQL command.

24. The system of claim 23 wherein the query-specified data processing operation comprises a query-specified text search operation for at least one key defined in the query.

25. The system of claim 24 wherein the main processor is configured with a SQL-enabled API for receiving and processing the query.

26. The system of claim 25 further comprising a bus that interconnects the main processor, the processing device, the data store of unstructured data, and the RDBMS.

27. The system of claim 26 further comprising a network interface connected to the bus, the network interface being in communication with a computer network in which another data store of unstructured data is located, and wherein the SQL-enabled API is configured to also process queries directed toward unstructured data stored in the another data store such that unstructured data within the another data store is streamed through the processing device in response to a query.

28. The system of claim 26 further comprising a network interface connected to the bus, the network interface being in communication with a computer network in which a SQL-enabled client application resides, the SQL-enabled client application being configured to formulate the query in response to user input and communicate the query to the SQL-enabled API via the network interface.

29. The system of claim 28 wherein the SQL-enabled client application comprises a business intelligence reporting tool.

30. The system of claim 25 further comprising:

a bus that interconnects the main processor, the processing device, and the RDBMS; and

a network interface that interconnects the data store of unstructured data and the processing device by way of the bus.

31. The system of claim 25 wherein the main processor is further configured to direct a stream of unstructured data through the processing device, wherein the processing device is further configured to generate metadata about the unstructured data streamed therethrough, and wherein the main processor is further configured to store the generated metadata in the RDBMS for use during a subsequent query-handling operation.

35. The method of claim 31 wherein the coprocessor comprises a reconfigurable logic device.

36. The method of claim 35 wherein the reconfigurable logic device has firmware deployed thereon, and wherein the firmware is configured to perform the data analysis operation.

32. A method of providing integrated access to both structured and unstructured data, the method comprising:

receiving a query directed toward a combination of structured and unstructured data;

retrieving structured data from a database of structured data in accordance with the received query;

retrieving unstructured data from a data store of unstructured data based on the retrieved structured data;

streaming the retrieved unstructured data through firmware deployed on a reconfigurable logic device, wherein the firmware is configured to perform a query-specified data processing operation on the unstructured data streamed therethrough.

33. A system for processing data, the system comprising:

a reconfigurable logic device having firmware deployed thereon;

a data store of unstructured data in communication with the reconfigurable logic device;

a data store of structured data in communication with the reconfigurable logic device;

an interface configured to (1) receive a query, (2) process at least a portion of the query against the structured data in the data store of structured data to identify a subset of unstructured data in the data store of unstructured data that satisfies at least a portion of the query, and (3) request that the subset of unstructured data be delivered to the firmware on the reconfigurable logic device; and

wherein the firmware is configured to perform a query-specified data processing operation on the subset of unstructured data.

34. A method for intelligently accessing unstructured data to perform a data analysis operation thereon, the method comprising:

maintaining metadata corresponding to the unstructured data within a data store of structured data;

identifying a subset of the unstructured data by applying a query to the data store of structured data that is directed at least partially toward the metadata; and

retrieving the identified subset of unstructured data from a data store of unstructured data; and

performing a data analysis operation on the retrieved unstructured data using a coprocessor to thereby generate data for a response to the query.

37. The method of claim 34 wherein the data analysis operation performing step comprises full-text searching the retrieved unstructured data using the coprocessor.

38. The method of claim 37 wherein the full-text searching include proximity searching.

39. The method of claim 34 wherein the query comprises a SQL command.

40. The method of claim 34 further comprising streaming the unstructured data through a coprocessor to thereby generate the metadata, wherein the coprocessor is configured to perform a metadata generation operation on the streaming data.

41. A processor-readable medium for interfacing a coprocessor with a software application, the computer-readable medium comprising:

processor-executable code for (1) receiving a query from a software application, (2) determining a portion of the query to be directed against structured data stored by a data store of structured data, (3) determining portion of the query to be directed toward unstructured data stored by a data store of unstructured data, (4) applying the determined query portion for structured data against the data store of structured data, (5) in response to the application of the determined query portion for structured data, receiving from the data store of structured data an identification of a subset of the stored unstructured data, (6) delivering a command to the coprocessor to thereby configure the coprocessor to perform a data analysis operation specified by the determined query portion for unstructured data, and (7) directing a delivery of the subset of unstructured data from the data store of unstructured data to the coprocessor for performance of the data analysis operation on the subset of unstructured data, wherein the code is resident on the processor-readable medium.

42. The processor-readable medium of claim 41 wherein the query comprises a SQL command.

43. The processor-readable medium of claim 41 further comprising processor-executable code resident on the processor-readable medium for (1) receiving data from the coprocessor in response to the data analysis operation, and (2) formatting the received data for delivery to the software application.

44. A method of processing a query that is directed toward structured and unstructured data, the method comprising:

performing a structured data retrieval operation for a portion of the query that is directed toward structured data; and

hardware-accelerating a query-specified data processing operation for a portion of the query that is directed toward unstructured data.

45. A method of performing queries on unstructured data, the method comprising:

receiving a query;

accessing structured data in response to the query to identify a subset of unstructured data that is to be analyzed against the query; and

performing a query-specified data analysis operation on the identified subset of unstructured data to thereby generate data for a response to the query;

wherein the accessing step is conducted by a processor; and

wherein the performing step is conducted by a coprocessor.

46. The method of claim 45 wherein the coprocessor comprises a reconfigurable logic device.

47. The method of claim 46 wherein the reconfigurable logic device has firmware deployed thereon, the firmware being configured to perform the query-specified data analysis operation on the subset of unstructured data.

48. The method of claim 47 wherein the data analysis operation comprises a full-text search operation.

49. The method of claim 45 wherein the structured data is stored in a relational database.

50. The method of claim 49 further comprising retrieving the identified subset of unstructured data prior to the performing step, wherein the retrieving step comprises retrieving the identified subset of unstructured data from the relational database.

51. The method of claim 45 wherein the structured data comprises a metadata index corresponding to unstructured data.

52. An enterprise computing system, the system comprising:

a computer;

an appliance in communication with the computer over a network, the appliance comprising a processor, a coprocessor, and a data store;

wherein the computer is configured to execute a software application to deliver a query to the appliance;

wherein the appliance is configured to receive the query;

wherein the processor is configured to selectively apply a portion of the received query to structured data stored by the data store and identify a subset of unstructured data in response to the applied query;

wherein the processor is further configured to define a data processing operation for the coprocessor based on another portion of the received query;

wherein the appliance is further configured to stream the identified subset of unstructured data through the coprocessor;

wherein the coprocessor is configured to perform the defined data processing operation on the identified subset of unstructured data streamed therethrough to thereby generate a result set corresponding to the received query; and

wherein the processor is further configured to (1) formulate a response to the query based on the generated result set and (2) communicate the query response to the computer.

53. The system of claim 52 wherein the data store comprises a relational database in which at least a portion of the structured data is stored, and wherein the appliance further comprises a second data store in which at least a portion of the unstructured data is stored.

54. The system of claim 53 wherein the structured data comprises at least in part a metadata index corresponding to the unstructured data.

55. The system of claim 54 wherein the software application comprises a SQL-enabled software application, and wherein the query comprises a SQL command.

56. The system of claim 54 further comprising a second relational database in which structured data is stored, wherein the appliance is in communication with the second relational database over a network, and wherein the processor is further configured to (1) selectively apply yet another portion of the received query to the second relational database and (2) identify the subset of unstructured data based on the portions of the query applied to the two relational databases.

57. The system of claim 54 further comprising a third data store in which unstructured data is stored, wherein the appliance is in communication with the second data store over a network, and wherein the identified subset of unstructured data comprises unstructured data stored by both of the second and third data stores.

58. The system of claim 54 wherein the coprocessor comprises a reconfigurable logic device.

59. The system of claim 58 wherein the reconfigurable logic device has firmware deployed thereon, the firmware configured to perform the defined data processing operation.

60. The system of claim 59 wherein the defined data processing operation comprises a full-text search operation.

61. The system of claim 59 wherein the defined data processing operation comprises a text mining operation.

62. The system of claim 59 wherein the defined data processing operation comprises a regular expression matching operation.

63. The system of claim 59 wherein the firmware comprises a firmware pipeline, the firmware pipeline being configured to perform a decryption operation prior to the defined data processing operation.

64. The system of claim 59 wherein the firmware comprises a firmware pipeline, the firmware pipeline being configured to perform a decompression operation prior to the defined data processing operation.

65. The system of claim 54 wherein the processor is configured to execute API software to handle the received query.

66. The system of claim 65 wherein the API software comprises relational engine software and coprocessor interface software, wherein the relational engine software is configured to invoke the coprocessor interface software upon encountering the received another query portion, and wherein the coprocessor interface software is configured to invoke the coprocessor to handle the received another query portion.

67. An apparatus for handling a query directed toward structured and unstructured data, the apparatus comprising:

a processor configured to (1) execute relational engine software and (2) execute coprocessor interface software, wherein the relational engine software is configured to (1) apply a portion of a query that is directed toward structured data to a relational database to thereby identify a subset of unstructured data, and (2) invoke the coprocessor interface software upon encountering a portion of the query that is directed toward unstructured data, and wherein the coprocessor interface software is configured to invoke a coprocessor to perform a query-specified data processing operation on the identified subset of unstructured data.

68. The apparatus of claim 67 wherein the coprocessor interface software is further configured to pass data to the coprocessor to thereby enable the coprocessor to perform the query-specified data processing operation on the identified subset of unstructured data.

69. The apparatus of claim 68 wherein the relational engine software comprises SQL relational engine software.