Method and system for high performance integration, processing and searching of structured and unstructured data using coprocessors

• US 9,396,222 B2
• Filed: 11/03/2014
• Issued: 07/19/2016
• Est. Priority Date: 11/13/2006
• Status: Active Grant

• Current Assignee: Ip Reservoir, LLC
• Original Assignee: Ip Reservoir, LLC
• Inventors: Indeck, Ronald S., Indeck, David Mark
• Primary Examiner(s): LE, DEBBIE M
• Application Number:
  US14/531,255
• Publication Number:
  US 20150052148A1
• Time in Patent Office: 624 Days
• Field of Search: 707/741
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Disclosed herein is a method and system for integrating an enterprise'"'"'s structured and unstructured data to provide users and enterprise applications with efficient and intelligent access to that data. In accordance with exemplary embodiments, the generation of metadata indexes about unstructured data can be hardware-accelerated by processing streaming unstructured data through a reconfigurable logic device to generate the metadata about the unstructured data for the index.

Claims

1. A method for building a metadata index for unstructured data for a plurality of different data sources, the method comprising:

receiving streaming unstructured data into a reconfigurable logic device, the streaming unstructured data comprising a plurality of data items for a plurality of different sources, wherein the reconfigurable logic device has a plurality of pipelined firmware application modules deployed thereon;

the pipelined firmware application modules analyzing the streaming unstructured data to generate metadata about the streaming unstructured data at hardware processing speeds, the analyzing including detecting whether a term relating to a name is found in any of the data items, the generated metadata comprising data associated with the data item that is indicative of where a data item having the detected term can be located; and

generating an index about the streaming unstructured data from the generated metadata, the index for subsequent querying to locate data items of interest based on associations between the metadata and the data items.

2. The method of claim 1 wherein the data items comprise at least two members of the group consisting of (1) a plurality of news reports, (2) a plurality of web pages, (3) a plurality of market analyses, (4) a plurality of emails, (5) a plurality of social network communications, and (6) a plurality of corporate documents.

3. The method of claim 2 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the FPGA having the pipelined firmware application modules deployed thereon.

4. The method of claim 3 wherein the analyzing step comprises the pipelined firmware application modules performing a classification operation on the streaming unstructured data to determine classification information for the data items, the generated metadata including the determined classification information;

and

wherein the index generating step further comprises generating an index about the streaming unstructured data based on the determined classification information.

5. The method of claim 4 wherein the classification operation performing step includes the pipelined firmware application modules generating word counts for the data items, the determined classification information being based on the generated word counts.

6. The method of claim 2 further comprising storing the generated index in a database for subsequent querying.

7. The method of claim 2 further comprising:

streaming a plurality of the data items into the reconfigurable logic device from a plurality of remote data sources via a network interface.

8. The method of claim 2 further comprising:

performing a lookup using the generated index as part of a reputation analysis operation for an enterprise.

9. The method of claim 2 wherein the detecting comprises:

the pipelined firmware application modules identifying a plurality of names that are found within the data items, the index indexing the data items by the found names;

wherein the method further comprises;

performing a plurality of lookups relating to a plurality of the names using the generated index; and

determining a connectedness for a plurality of individuals based on the lookups.

10. The method of claim 9 wherein the at least two members comprise the emails and the social network communications.

11. The method of claim 9 the data items comprise at least three members of the group consisting of (1) a plurality of news reports, (2) a plurality of web pages, (3) a plurality of market analyses, (4) a plurality of emails, (5) a plurality of social network communications, and (6) a plurality of corporate documents.

12. The method of claim 2 wherein the detecting comprises:

the pipelined firmware application modules identifying a plurality of names that are found within the data items, the index indexing the data items by the found names;

wherein the method further comprises;

performing a plurality of lookups relating to a plurality of the names using the generated index; and

determining a connectedness for a plurality of organizations based on the lookups.

13. The method of claim 12 wherein the at least two members comprise the emails and the corporate documents.

14. The method of claim 12 the data items comprise at least three members of the group consisting of (1) a plurality of news reports, (2) a plurality of web pages, (3) a plurality of market analyses, (4) a plurality of emails, (5) a plurality of social network communications, and (6) a plurality of corporate documents.

15. The method of claim 2 wherein the detecting comprises:

the pipelined firmware application modules identifying a plurality of names that are found within the data items, the index indexing the data items by the found names;

wherein the method further comprises;

performing a plurality of lookups relating to a plurality of the names using the generated index; and

determining a connectedness for a plurality of individuals and organizations based on the lookups.

16. The method of claim 2 further comprising:

integrating the index with structured data relating to the name in a structured database.

17. An apparatus for building a metadata index for unstructured data for a plurality of different data sources, the apparatus comprising:

a reconfigurable logic device; and

a memory;

wherein the reconfigurable logic device is configured to receive streaming unstructured data, the streaming unstructured data comprising a plurality of data items for a plurality of different sources, wherein the reconfigurable logic device has a plurality of pipelined firmware application modules deployed thereon;

the pipelined firmware application modules configured to perform analysis of the streaming unstructured data to generate metadata about the streaming unstructured data at hardware processing speeds, the analysis including a detection by the pipelined firmware application modules whether a term relating to a name is found in any of the data items, the generated metadata comprising data associated with the data item that is indicative of where a data item having the detected term can be located; and

the memory configured to store an index about the streaming unstructured data from the generated metadata, the index for querying to locate data items of interest based on associations between the metadata and the data items.

18. A method for integrating unstructured data for a plurality of different data sources, the method comprising:

streaming unstructured data through a field programmable gate array (FPGA), the unstructured data comprising at least two members of the group consisting of (1) a plurality of emails, (2) a plurality of social network communications, (3) a plurality of corporate documents, and (4) a plurality of news reports;

the FPGA performing a metadata generation operation on the unstructured data streamed therethrough to thereby generate metadata about the unstructured data;

storing the unstructured data in a data store of unstructured data;

storing the metadata about the unstructured data in a database of structured data; and

determining a connectedness of a plurality of subjects based on an analysis of the stored unstructured data and the stored metadata.

19. The method of claim 18 wherein the metadata includes an identification of where the unstructured data is stored in the data store of unstructured data.

20. The method of claim 19 wherein the performing step further comprises:

the FPGA identifying a plurality of names that are found in the streaming unstructured data, and wherein the generated metadata identifies where the identified names are present in the unstructured data.

21. The method of claim 20 wherein the subjects comprise a plurality of individuals.

22. The method of claim 20 wherein the subjects comprise a plurality of organizations.

23. A method for building a metadata index for unstructured data for a plurality of different data sources, the method comprising:

receiving streaming unstructured data into a reconfigurable logic device, the streaming unstructured data comprising a plurality of data items for a plurality of different sources, wherein the reconfigurable logic device has a plurality of pipelined firmware application modules deployed thereon;

the pipelined firmware application modules analyzing the streaming unstructured data to generate metadata about the streaming unstructured data at hardware processing speeds, the analyzing including detecting whether a term relating to a security instrument is found in any of the data items, the generated metadata comprising data associated with the data item that is indicative of where a data item having the detected term can be located; and

generating an index about the streaming unstructured data from the generated metadata, the index for subsequent querying to locate data items of interest based on associations between the metadata and the data items.

24. The method of claim 23 wherein the data items comprise at least two members of the group consisting of (1) a plurality of news reports, (2) a plurality of corporate filings, (3) a plurality of corporate documents, (4) a plurality of trading records, and (5) a plurality of transactional records.

25. The method of claim 24 wherein the reconfigurable logic device comprises a field programmable gate array (FPGA), the FPGA having the pipelined firmware application modules deployed thereon.

26. The method of claim 25 wherein the analyzing step comprises the pipelined firmware application modules performing a classification operation on the streaming unstructured data to determine classification information relating to the security instrument for the data items, the generated metadata including the determined classification information;

and

wherein the index generating step further comprises generating an index about the streaming unstructured data based on the determined classification information.

27. The method of claim 24 further comprising storing the generated index in a database for subsequent querying.

28. The method of claim 24 further comprising:

streaming a plurality of the data items into the reconfigurable logic device from a plurality of remote data sources via a network interface.

29. The method of claim 24 further comprising:

performing a lookup using the generated index as part of an insider trading detection operation relating to the security instrument.

30. The method of claim 24 further comprising:

performing a lookup using the generated index as part of an reporting irregularity detection operation.

31. The method of claim 24 further comprising:

performing a lookup using the generated index as part of an illegal transaction detection operation relating to the security instrument.

32. The method of claim 24 further comprising:

performing a lookup using the generated index as part of a pricing anomaly detection operation relating to the security instrument.

33. The method of claim 24 further comprising:

performing a lookup using the generated index as part of a money laundering detection operation relating to the security instrument.

34. The method of claim 24 wherein the at least two members comprise the corporate filings and the trading records.

35. The method of claim 24 wherein the at least two members comprise the news reports and the trading records.

36. The method of claim 24 the data items comprise at least three members of the group consisting of (1) a plurality of news reports, (2) a plurality of corporate filings, (3) a plurality of corporate documents, (4) a plurality of trading records, and (5) a plurality of transactional records.

37. The method of claim 24 wherein the data items comprise (1) a plurality of news reports, (2) a plurality of corporate filings, (3) a plurality of corporate documents, (4) a plurality of trading records, and (5) a plurality of transactional records.

38. The method of claim 24 further comprising:

integrating the index with structured data relating to the security instrument in a structured database.

39. An apparatus for building a metadata index for unstructured data for a plurality of different data sources, the apparatus comprising:

a reconfigurable logic device; and

a memory;

wherein the reconfigurable logic device is configured to receive streaming unstructured data, the streaming unstructured data comprising a plurality of data items for a plurality of different sources, wherein the reconfigurable logic device has a plurality of pipelined firmware application modules deployed thereon;

the pipelined firmware application modules configured to perform analysis of the streaming unstructured data to generate metadata about the streaming unstructured data at hardware processing speeds, the analysis including a detection by the pipelined firmware application modules whether a term relating to a security instrument is found in any of the data items, the generated metadata comprising data associated with the data item that is indicative of where a data item having the detected term can be located; and

the memory configured to store an index about the streaming unstructured data from the generated metadata, the index for querying to locate data items of interest based on associations between the metadata and the data items.