Method and system for high performance integration, processing and searching of structured and unstructured data using coprocessors

• US 7,660,793 B2
• Filed: 11/12/2007
• Issued: 02/09/2010
• Est. Priority Date: 11/13/2006
• Status: Active Grant

• Current Assignee: Ip Reservoir, LLC
• Original Assignee: Exegy Incorporated
• Inventors: Indeck, David Mark, Indeck, Ronald S.
• Primary Examiner(s): Le; Debbie M
• Application Number:
  US11/938,709
• Publication Number:
  US 20080114724A1
• Time in Patent Office: 820 Days
• Field of Search: 707 1- 5, 707/10, 707101-102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Disclosed herein is a method and system for integrating an enterprise'"'"'s structured and unstructured data to provide users and enterprise applications with efficient and intelligent access to that data. Queries can be directed toward both an enterprise'"'"'s structured and unstructured data using standardized database query formats such as SQL commands. A coprocessor can be used to hardware-accelerate data processing tasks (such as full-text searching) on unstructured data as necessary to handle a query. Furthermore, traditional relational database techniques can be used to access structured data stored by a relational database to determine which portions of the enterprise'"'"'s unstructured data should be delivered to the coprocessor for hardware-accelerated data processing.

Claims

1. A method of providing integrated access to both structured and unstructured data, the method comprising:

maintaining metadata in a database of structured data, the metadata comprising data about unstructured data;

receiving a query directed toward a combination of structured and unstructured data, the query having a portion directed toward unstructured data, wherein the portion comprises at least one term for searching within unstructured data;

retrieving structured data from a database of structured data in accordance with the received query, the retrieved structured data comprising at least a portion of the metadata;

processing the retrieved structured data to identify a scope of unstructured data for retrieval from a data store of unstructured data;

retrieving unstructured data from the data store of unstructured data corresponding to the identified scope of unstructured data;

streaming the retrieved unstructured data through a processing device in a computing system other than a main processor for the system; and

searching the retrieved streaming unstructured data based on the at least one query term using the processing device to find unstructured data within the retrieved streaming unstructured data that is responsive to the query.

2. The method of claim 1 wherein the processing device comprises a coprocessor.

3. The method of claim 2 wherein the coprocessor comprises a reconfigurable logic device.

4. The method of claim 3 wherein the reconfigurable logic device has firmware deployed thereon, the firmware being configured to perform the searching operation.

5. The method of claim 1 wherein the query comprises a standardized database query.

6. The method of claim 5 wherein the standardized database query comprises a SQL command.

7. The method of claim 6 wherein the searching operation comprises a text searching operation based on the at least one query term.

8. The method of claim 1 further comprising:

streaming new unstructured data through the processing device;

performing a metadata generation operation on the streaming unstructured data using the processing device to thereby generate metadata about the new unstructured data;

storing the new unstructured data in the data store of unstructured data; and

storing the metadata about the new unstructured data in the database of structured data.

9. The method of claim 8 wherein the processing device comprises a coprocessor.

10. The method of claim 9 wherein the coprocessor comprises a reconfigurable logic device.

11. The method of claim 10 wherein the reconfigurable logic device has firmware deployed thereon, the firmware being configured to perform the metadata generation operation.

12. The method of claim 1 wherein the database of structured data comprises a RDBMS.

13. The method of claim 1 wherein the structured data retrieving step comprises retrieving structured data from an external database of structured data in accordance with the received query.

14. The method of claim 13 wherein the structured data retrieving step further comprises also retrieving structured data from an internal database of structured data in accordance with the received query.

15. The method of claim 1 wherein the structured data retrieving step comprises retrieving structured data from an internal database of structured data in accordance with the received query.

50. The method of claim 1 wherein the identified scope comprises a list of files in the data store of unstructured data.

53. The system of claim 1 wherein the database of structured data and the data store of unstructured data are housed within a common data store.

54. The method of claim 1 wherein the searching step comprises performing at least one member of the group consisting of an exact matching operation, an approximate matching operation, and a regular expression pattern matching operation using the at least one query term.

16. A system for processing data, the system comprising:

a main processor;

a processing device other than the main processor;

a data store of unstructured data in communication with the main processor and the processing device;

a data store of structured data in communication with the main processor and the processing device, the structured data comprising metadata about at least a portion of the unstructured data;

wherein the main processor is configured to (1) receive a query, the query comprising at least one term for searching within unstructured data, (2) process at least a portion of the query against the structured data in the data store of structured data to identify a subset of unstructured data in the data store of unstructured data, and (3) request that the subset of unstructured data be delivered to the processing device; and

wherein the processing device is configured to (1) receive the subset of unstructured data and (2) search the received subset of unstructured data based on the at least one query term to determine whether any of the unstructured data within the received subset of unstructured data matches the at least one query term.

17. The system of claim 16 wherein the processing device comprises a coprocessor.

18. The system of claim 17 wherein the coprocessor comprises a reconfigurable logic device.

19. The system of claim 18 wherein the reconfigurable logic device has firmware deployed thereon, the firmware being configured to perform the searching operation.

20. The system of claim 16 wherein the database of structured data comprises a RDBMS, and wherein the query comprises a standardized database query.

21. The system of claim 20 wherein the structured data within the RDBMS comprises metadata about the unstructured data within the data store of unstructured data.

22. The system of claim 21 wherein the standardized database query comprises a SQL command.

23. The system of claim 22 wherein the searching operation comprises a query-specified text search operation for the at least one query term.

24. The system of claim 23 wherein the main processor is configured with a SQL-enabled API for receiving and processing the query.

25. The system of claim 24 further comprising a bus that interconnects the main processor, the processing device, the data store of unstructured data, and the RDBMS.

26. The system of claim 25 further comprising a network interface connected to the bus, the network interface being in communication with a computer network in which another data store of unstructured data is located, and wherein the SQL-enabled API is configured to also process queries directed toward unstructured data stored in the another data store such that unstructured data within the another data store is streamed through the processing device in response to a query.

27. The system of claim 25 further comprising a network interface connected to the bus, the network interface being in communication with a computer network in which a SQL-enabled client application resides, the SQL-enabled client application being configured to formulate the query in response to user input and communicate the query to the SQL-enabled API via the network interface.

28. The system of claim 27 wherein the SQL-enabled client application comprises a business intelligence reporting tool.

29. The system of claim 24 further comprising:

a bus that interconnects the main processor, the processing device, and the RDBMS; and

a network interface that interconnects the data store of unstructured data and the processing device by way of the bus.

30. The system of claim 24 wherein the main processor is further configured to direct a stream of unstructured data through the processing device, wherein the processing device is further configured to generate metadata about the unstructured data streamed therethrough, and wherein the main processor is further configured to store the generated metadata in the RDBMS for use during a subsequent query-handling operation.

51. The system of claim 16 wherein the identified subset comprises a list of files in the data store of unstructured data.

52. The system of claim 16 wherein the data store of structured data and the data store of unstructured data are housed within a common data store.

55. The system of claim 16 wherein the processing device is further configured to perform the searching operation via at least one member of the group consisting of an exact matching operation, an approximate matching operation, and a regular expression pattern matching operation based on the at least one query term.

31. A method of providing integrated access to both structured and unstructured data, the method comprising:

maintaining metadata in a database of structured data, the metadata comprising data about unstructured data;

receiving a query directed toward a combination of structured and unstructured data, the query having a portion directed toward unstructured data wherein the portion comprises at least one term for processing against the unstructured data;

retrieving structured data from a database of structured data in accordance with the received query, the structured data comprising at least a portion of the metadata;

identifying which unstructured data in a data store of unstructured data is to be retrieved based on the retrieved structured data;

retrieving the identified unstructured data from the data store of unstructured data;

streaming the retrieved unstructured data through firmware deployed on a reconfigurable logic device; and

performing a query-specified data processing operation on the streaming retrieved unstructured data using the firmware to find a set of unstructured data within the streaming retrieved unstructured data that is relevant to the query in view of the at least one query term.

56. The method of claim 31 wherein the database of structured data and the data store of unstructured data are housed within a common data store.

57. The method of claim 31 wherein the step of performing the query-specified data processing operation comprises performing at least one member of the group consisting of an exact matching operation, an approximate matching operation, and a regular expression pattern matching operation using the at least one query term.

32. A system for processing data, the system comprising:

a reconfigurable logic device having firmware deployed thereon;

a data store of unstructured data in communication with the reconfigurable logic device;

a data store of structured data in communication with the reconfigurable logic device, the structured data comprising metadata about at least a portion of the unstructured data;

a processor configured to execute an interface, the interface configured to (1) receive a query, the query comprising at least one term for processing against unstructured data, (2) process at least a portion of the query against the structured data in the data store of structured data to identify a subset of unstructured data in the data store of unstructured data, and (3) request that the subset of unstructured data be delivered to the firmware on the reconfigurable logic device; and

wherein the firmware is configured to perform a query-specified data processing operation on the subset of unstructured data using the at least one query term.

58. The system of claim 32 wherein the data store of structured data and the data store of unstructured data are housed within a common data store.

59. The system of claim 32 wherein the firmware is further configured to perform the query-specified data processing operation via at least one member of the group consisting of an exact matching operation, an approximate matching operation, and a regular expression pattern matching operation based on the at least one query term.

33. A processor-readable storage medium for interfacing a coprocessor with a software application, the processor-readable storage medium comprising:

processor-executable code for (1) receiving a query from a software application, (2) determining a portion of the query to be directed against structured data stored by a data store of structured data, (3) determining portion of the query to be directed toward unstructured data stored by a data store of unstructured data, the determined query portion for unstructured data comprising at least one term for searching within unstructured data, (4) applying the determined query portion for structured data against the data store of structured data, the structured data comprising metadata about at least a portion of the unstructured data, (5) in response to the application of the determined query portion for structured data, receiving metadata from the data store of structured data that is representative of a subset of the stored unstructured data, (6) delivering a command to the coprocessor to thereby configure the coprocessor to perform a data analysis operation specified by the determined query portion for unstructured data, the data analysis operation comprising a searching operation based on the at least one query term, and (7) directing a delivery of the subset of unstructured data from the data store of unstructured data to the coprocessor for performance of the data analysis operation on the subset of unstructured data, wherein the code is resident on the processor-readable storage medium.

34. The processor-readable storage medium of claim 33 wherein the query comprises a SQL command.

35. The processor-readable storage medium of claim 33 further comprising processor-executable code resident on the processor-readable medium for (1) receiving data from the coprocessor in response to the data analysis operation, and (2) formatting the received data for delivery to the software application.

36. An enterprise computing system, the system comprising:

a computer;

an appliance in communication with the computer over a network, the appliance comprising a processor, a coprocessor, and a data store;

wherein the computer is configured to execute a software application to deliver a query to the appliance;

wherein the appliance is configured to receive the query;

wherein the processor is configured to selectively apply a portion of the received query to structured data stored by the data store and identify a subset of unstructured data in response to the applied query, wherein the structured data comprises metadata about at least a portion of the unstructured data;

wherein the processor is further configured to define a searching operation for the coprocessor based on another portion of the received query, the another portion including at least one query term for searching unstructured data;

wherein the appliance is further configured to stream the identified subset of unstructured data through the coprocessor;

wherein the coprocessor is configured to perform the defined searching operation on the identified subset of unstructured data streamed therethrough based on the at least one query term to thereby generate a result set corresponding to the received query; and

wherein the processor is further configured to (1) formulate a response to the query based on the generated result set and (2) communicate the query response to the computer.

37. The system of claim 36 wherein the data store comprises a relational database in which at least a portion of the structured data is stored, and wherein the appliance further comprises a second data store in which at least a portion of the unstructured data is stored.

38. The system of claim 37 wherein the software application comprises a SQL-enabled software application, and wherein the query comprises a SQL command.

39. The system of claim 37 further comprising a second relational database in which structured data is stored, wherein the appliance is in communication with the second relational database over a network, and wherein the processor is further configured to (1) selectively apply yet another portion of the received query to the second relational database and (2) identify the subset of unstructured data based on the portions of the query applied to the two relational databases.

40. The system of claim 37 further comprising a third data store in which unstructured data is stored, wherein the appliance is in communication with the second data store over a network, and wherein the identified subset of unstructured data comprises unstructured data stored by both of the second and third data stores.

41. The system of claim 37 wherein the coprocessor comprises a reconfigurable logic device.

42. The system of claim 41 wherein the reconfigurable logic device has firmware deployed thereon, the firmware configured to perform the defined searching operation.

43. The system of claim 42 wherein the defined searching operation comprises a full-text search operation.

44. The system of claim 42 wherein the defined searching operation comprises a text mining operation.

45. The system of claim 42 wherein the defined searching operation comprises a regular expression matching operation.

46. The system of claim 42 wherein the firmware comprises a firmware pipeline, the firmware pipeline being configured to perform a decryption operation prior to the defined searching operation.

47. The system of claim 42 wherein the firmware comprises a firmware pipeline, the firmware pipeline being configured to perform a decompression operation prior to the defined searching operation.

48. The system of claim 37 wherein the processor is configured to execute API software to handle the received query.

49. The system of claim 48 wherein the API software comprises relational engine software and coprocessor interface software, wherein the relational engine software is configured to invoke the coprocessor interface software upon encountering the received another query portion, and wherein the coprocessor interface software is configured to invoke the coprocessor to handle the received another query portion.

60. The system of claim 36 wherein the coprocessor is further configured to perform the defined searching operation via at least one member of the group consisting of an exact matching operation, an approximate matching operation, and a regular expression pattern matching operation based on the at least one query term.