Federated search of multiple sources with conflict resolution

• US 10,545,982 B1
• Filed: 07/23/2015
• Issued: 01/28/2020
• Est. Priority Date: 04/01/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Kramer, Danielle, Israel, Andrew, Chen, Jeffrey, Cohen, David, Freiberg, Stephen, Offutt, Bryan, Avant, Matt, Wilczynski, Peter, Hoch, Jason, Liu, Robert, Waldrep, William, Zhang, Kevin, Landau, Alexander, Tobin, David
• Primary Examiner(s): Pham, Khanh B
• Application Number:
  US14/807,658
• Time in Patent Office: 1,650 Days
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Methods and apparatuses related to federated search of multiple sources with conflict resolution are disclosed. A method may comprise obtaining a set of data ontologies (e.g., types, properties, and links) associated with a plurality of heterogeneous data sources; receiving a selection of a graph comprising a plurality of graph nodes connected by one or more graph edges; and transforming the graph into one or more search queries across the plurality of heterogeneous data sources. A method may comprise obtaining a first data object as a result of executing a first search query across a plurality of heterogeneous data sources; resolving, based on one or more resolution rules, at least the first data object with a repository data object; deduplicating data associated with at least the first data object and the repository data object prior to storing the deduplicated data in a repository that has a particular data model.

Claims

1. A method comprising:

obtaining a set of data ontologies associated with a plurality of heterogeneous data sources;

wherein each heterogeneous data source of the plurality of heterogeneous data sources uses a different data model;

receiving, as input, a selection of a graph comprising a plurality of graph nodes connected by one or more graph edges, wherein the graph has an object-centric data model in which each graph node represents a data object type or a data object property that is described in at least one data ontology of the set of data ontologies and each graph edge represents a data object link that represents a relationship between a pair of graph nodes and that is described in at least one data ontology of the set of data ontologies, wherein each graph edge of the one or more graph edges is selected from a set of available data object links;

transforming the graph into one or more search queries that are executed across the plurality of heterogeneous data sources;

obtaining a first data object and a second data object based on executing the one or more search queries across the plurality of heterogeneous data sources;

generating an intermediate data object based on grouping the first data object with the second data object;

generating a unique identifier for the intermediate data object based on hashing one or more data object properties that uniquely identify the intermediate data object;

determining whether a repository data object that shares the unique identifier is stored in a repository that has a particular data model;

in response to determining that the repository data object is not stored in a repository that has a particular data model,

generating a stub data object that is referenced by the unique identifier and is stored in the repository;

resolving the intermediate data object with the stub data object;

deduplicating data associated with the intermediate data object and the stub data object;

storing the deduplicated data in the repository that has the particular data model;

causing display of the first data object and the second data object as a single graph node;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the graph is transformed to display one or more results of the one or more search queries.

3. The method of claim 1, wherein the transforming the graph into one or more search queries across the plurality of heterogeneous data sources comprises:

transforming the graph into a serializable hierarchy of elements that correspond to the plurality of graph nodes and the one or more graph edges.

4. The method of claim 1, wherein the second data object is a repository data object.

5. The method of claim 1, wherein one or more results of the one or more search queries are filtered based on one or more access control identifiers associated with one or more groups of users and one or more permissions.

6. The method of claim 1, wherein a change to data in one of the plurality of heterogeneous data sources is automatically imported based on a push or pull technology.

7. The method of claim 1, wherein the plurality of heterogeneous data sources comprises one or more caches associated with one or more repositories.

8. A system comprising:

one or more processors; and

one or more non-transitory storage media storing instructions which, when executed by the one or more processors, cause;

obtaining a set of data ontologies associated with a plurality of heterogeneous data sources;

wherein each heterogeneous data source of the plurality of heterogeneous data sources uses a different data model;

receiving, as input, a selection of a graph comprising a plurality of graph nodes connected by one or more graph edges, wherein the graph has an object-centric data model in which each graph node represents a data object type or a data object property that is described in at least one data ontology of the set of data ontologies and each graph edge represents a data object link that represents a relationship between a pair of graph nodes and that is described in at least one data ontology of the set of data ontologies, wherein each graph edge of the one or more graph edges is selected from a set of available data object links;

transforming the graph into one or more search queries that are executed across the plurality of heterogeneous data sources;

obtaining a first data object and a second data object based on executing the one or more search queries across the plurality of heterogeneous data sources;

generating an intermediate data object based on grouping the first data object with the second data object;

generating a unique identifier for the intermediate data object based on hashing one or more data object properties that uniquely identify the intermediate data object;

determining whether a repository data object that shares the unique identifier is stored in a repository that has a particular data model;

in response to determining that the repository data object is not stored in a repository that has a particular data model,

generating a stub data object that is referenced by the unique identifier and is stored in the repository;

resolving the intermediate data object with the stub data object;

deduplicating data associated with the intermediate data object and the stub data object;

storing the deduplicated data in the repository that has the particular data model;

causing display of the first data object and the second data object as a single graph node.

9. The system of claim 8, wherein the graph is transformed to display one or more results of the one or more search queries.

10. The system of claim 8, wherein the transforming the graph into one or more search queries across the plurality of heterogeneous data sources comprises:

transforming the graph into a serializable hierarchy of elements that correspond to the plurality of graph nodes and the one or more graph edges.

11. The system of claim 8, wherein the second data object is a repository data object.

12. The system of claim 8, wherein one or more results of the one or more search queries are filtered based on one or more access control identifiers associated with one or more groups of users and one or more permissions.

13. The system of claim 8, wherein a change to data in one of the plurality of heterogeneous data sources is automatically imported based on a push or pull technology.

14. The system of claim 8, wherein the plurality of heterogeneous data sources comprises one or more caches associated with one or more repositories.