Systems and methods for intelligent parallel searching

• US 8,862,566 B2
• Filed: 10/26/2012
• Issued: 10/14/2014
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Equifax, Inc.
• Original Assignee: Equifax, Inc.
• Inventors: Leitner, Stephen, Manthey, Kevin W., Burgess, Mark, Canfield, Samuel
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US13/661,485
• Publication Number:
  US 20140122455A1
• Time in Patent Office: 718 Days
• Field of Search: 707/3, 707/692, 707/706, 709204-206
• US Class Current: 707/706
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods are provided for conducting intelligent parallel searches of data sources. A search engine receives a request to access target data that is stored in at least one of multiple data sources. Each data source has a candidate index. The search engine extracts inquiry parameters from the request. The inquiry parameters corresponds to sub-indices of respective general indices. Each general index includes an index of relationships between data from at least two of the data sources. Each sub-index includes a subset of the respective general index. The search engine performs parallel searches of the general indices common to the data sources. Each parallel search includes searching the sub-indices based on corresponding inquiry parameters for the sub-indices. The search engine performs additional parallel searches of the candidate indices based on results of the parallel searches. The search engine extracts an output from results of the additional parallel searches.

Claims

1. A method comprising:

receiving, by a search engine executed by a processor device, a request to access target data, wherein the target data is stored in at least one of a plurality of data sources, each of the plurality of data sources having a respective candidate index;

accessing, by the search engine, a plurality of general indices, wherein each of the plurality of general indices comprises a respective index of relationships between data from at least two of the plurality of data sources;

extracting, by the search engine, a plurality of inquiry parameters from the request, wherein each of the extracted plurality of inquiry parameters corresponds to a respective sub-index from the plurality of general indices, wherein each sub-index from the plurality of general indices comprises a respective subset of at least one respective general index from the plurality of general indices;

performing, by the search engine, parallel searches of the plurality of general indices, wherein each parallel search comprises searching a respective sub-index from the plurality of general indices based on a respective inquiry parameter from the extracted plurality of inquiry parameters that corresponds to the respective sub-index;

performing, by the search engine, additional parallel searches of the candidate indices for the plurality of data sources based on the extracted plurality of inquiry parameters and at least one additional inquiry parameter obtained from the parallel searches of the plurality of general indices; and

extracting, by the search engine, an output based on results returned from the additional parallel searches of the candidate indices.

2. The method of claim 1, wherein a first data source of the plurality of data sources comprises structured data and wherein a second data source of the plurality of data sources comprises unstructured data, wherein performing the additional parallel searches comprises:

formatting a first one of the additional parallel searches for accessing the structured data using a first one of the extracted plurality of inquiry parameters or the at least one additional inquiry parameter; and

formatting a second one of the additional parallel searches for accessing the structured data using a second one of the extracted plurality of inquiry parameters or the at least one additional inquiry parameter.

3. The method of claim 2, wherein the search engine executes the parallel searches via a data service layer.

4. The method of claim 1, wherein returning the output comprises returning the target data from at least two data sources of the plurality of data sources and a relationship between the target data from the at least two data sources.

5. The method of claim 4, wherein the target data and the relationship between the target data are usable for identity resolution, wherein the identity resolution comprises determining that a first entity or individual identified in a first one of the plurality of data sources is the same as or associated with a second entity or individual identified in a second one of the plurality of data sources.

6. The method of claim 1, wherein executing the parallel searches of the plurality of data sources comprises executing a first search of a first data source having a first type of storage medium and executing a second search of a second data source having a second type of storage medium different from the first type of storage medium.

7. The method of claim 1, wherein executing the additional parallel searches of the candidate indices further comprises excluding duplicate candidate data returned from the parallel searches.

8. The method of claim 1, wherein the at least one additional inquiry parameter is non-duplicative of the extracted plurality of inquiry parameters and wherein at least one of the additional parallel searches comprises searching a respective one of the candidate indices to which the at least one additional inquiry parameter corresponds and to which none of the extracted plurality of inquiry parameters corresponds.

9. The method of claim 8, further comprising obtaining the at least one additional inquiry parameter from the parallel searches by performing operations comprising:

determining from at least one of the plurality of general indices that first data from a one of the plurality of data sources is related to second data from a second one of the plurality of data sources; and

obtaining the at least one additional inquiry parameter from the second data based on (i) the second data being related to the first data and (ii) the first data corresponding to at least some of the extracted plurality of inquiry parameters.

10. The method of claim 9, wherein obtaining the at least one additional inquiry parameter from the second data comprises selecting the second data as the at least one additional inquiry parameter.

11. The method of claim 1, wherein a first set of results returned by searching the plurality of data sources with the extracted plurality of inquiry parameters is smaller than a second set of results returned by searching the plurality of data sources with the extracted plurality of inquiry parameters unioned with the at least one additional inquiry parameter.

12. A non-transitory computer-readable medium embodying program code executable by a computer system, the non-transitory computer-readable medium comprising:

program code for receiving a request to access target data, wherein the target data is stored in at least one of a plurality of data sources, each of the plurality of data sources having a respective candidate index;

program code for accessing a plurality of general indices, wherein each of the plurality of general indices comprises a respective index of relationships between data from at least two of the plurality of data sources;

program code for extracting a plurality of inquiry parameters from the request, wherein each of the extracted plurality of inquiry parameters corresponds to a respective sub-index from the plurality of general indices, wherein each sub-index from the plurality of general indices comprises a respective subset of at least one respective general index from the plurality of general indices;

program code for performing parallel searches of the plurality of general indices, wherein each parallel search comprises searching a respective sub-index from the plurality of general indices based on a respective inquiry parameter from the extracted plurality of inquiry parameters that corresponds to the respective sub-index;

program code for performing additional parallel searches of the candidate indices for the plurality of data sources based on the extracted plurality of inquiry parameters and at least one additional inquiry parameter obtained from the parallel searches of the plurality of general indices; and

program code for extracting an output based on results returned from the additional parallel searches of the candidate indices.

13. The non-transitory computer-readable medium of claim 12, wherein a first data source of the plurality of data sources comprises structured data and wherein a second data source of the plurality of data sources comprises unstructured data.

14. The non-transitory computer-readable medium of claim 12, wherein the program code for returning the output comprise program code for returning the target data from at least two data sources of the plurality of data sources and a relationship between the target data from the at least two data sources.

15. The non-transitory computer-readable medium of claim 12, wherein the program code for executing the parallel searches of the plurality of data sources comprises program code for executing a first search of a first data source having a first type of storage medium and executing a second search of a second data source having a second type of storage medium different from the first type of storage medium.

16. A system comprising:

a non-transitory computer-readable medium configured to store instructions providing a search engine;

a processor configured to execute the instructions stored in the non-transitory computer-readable medium to execute the search engine by performing operations comprising;

receiving a request to access target data, wherein the target data is stored in at least one of a plurality of data sources, each of the plurality of data sources having a respective candidate index;

accessing, by the search engine, a plurality of general indices, wherein each of the plurality of general indices comprises a respective index of relationships between data from at least two of the plurality of data sources;

extracting a plurality of inquiry parameters from the request, wherein each of the extracted plurality of inquiry parameters corresponds to a respective sub-index from the plurality of general indices, wherein each sub-index from the plurality of general indices comprises a respective subset of at least one respective general index from the plurality of general indices;

performing parallel searches of the plurality of general indices, wherein each parallel search comprises searching a respective sub-index from the plurality of general indices based on a respective inquiry parameter from the extracted plurality of inquiry parameters that corresponds to the respective sub-index;

performing additional parallel searches of the candidate indices for the plurality of data sources based on the extracted plurality of inquiry parameters and at least one additional inquiry parameter obtained from the parallel searches of the plurality of general indices; and

extracting an output based on results returned from the additional parallel searches of the candidate indices.

17. The system of claim 16, wherein a first data source of the plurality of data sources comprises structured data and wherein a second data source of the plurality of data sources comprises unstructured data.

18. The system of claim 16, wherein returning the output comprises returning the target data from at least two data sources of the plurality of data sources and a relationship between the target data from the at least two data sources.

19. The system of claim 18, wherein the target data and the relationship between the target data are usable for identity resolution.

20. The system of claim 16, wherein executing the parallel searches of the plurality of data sources comprises executing a first search of a first data source having a first type of storage medium and executing a second search of a second data source having a second type of storage medium different from the first type of storage medium.