Entity integration using high-level scripting languages

• US 9,535,951 B2
• Filed: 05/31/2013
• Issued: 01/03/2017
• Est. Priority Date: 05/31/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Brown, Armageddon Rhabdizo, Popa, Lucian, Hernandez-Sherrington, Mauricio Antonio, Thalamati, Suresh, Krishnamurthy, Rajasekar, Koutrika, Georgia, Wisnesky, Ryan
• Primary Examiner(s): NGUYEN, THU N
• Application Number:
  US13/907,532
• Publication Number:
  US 20140358932A1
• Time in Patent Office: 1,313 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24532 of parallel queries

Abstract

Embodiments of the present invention relate to a new method of entity integration using high-level scripting languages. In one embodiment, a method of and computer product for entity integration is provided. An entity declaration is read from a machine readable medium. The entity declaration describes an entity including at least one nested entity. An index declaration is read from a machine readable medium. The index declaration describes an index of nested entities. An entity population rule is read from a machine readable medium. The entity population rule describes a mapping from an input schema to an output schema. The output schema conforms to the entity declaration. A plurality of input records is read from a first data store. The input records conform to the input schema. The entity population rule applies to the plurality of records to create a plurality of output records complying with the output schema. An index of nested entities is populated. The index complies with the index declaration.

Claims

1. A method comprising:

reading an entity declaration from a machine readable medium, the entity declaration describing an entity including at least one nested entity;

reading an index declaration from a machine readable medium, the index declaration describing an index of nested entities;

reading an entity population rule from a machine readable medium, the entity population rule describing a mapping from an input schema to an output schema, the output schema conforming to the entity declaration;

reading at least one additional entity population rule from the machine readable medium;

compiling the entity population rule and the at least one additional entity population rule into at least one executable query, wherein compiling the entity population rule comprises topologically sorting the entities defined by the entity population rule and the at least one additional entity population rule based on at least one dependency between the entity population rule and the at least one additional entity population rule and generating a query by traversing from a leaf to a root of the topologically sorted entities;

reading a plurality of input records from a first data store, the input records conforming to the input schema;

reading an entity resolution rule from a machine readable medium, the entity resolution rule defining criteria for a link between entities;

compiling the entity resolution rule;

applying the entity resolution rule to the plurality of input records to determine a link between members of the plurality of input records;

applying the entity population rule to the plurality of input records to create a plurality of output records complying with the output schema;

subsequent to applying the entity population rule, recompiling the entity population rule and the at least one additional entity population rule; and

populating the index of nested entities using the at least one executable query, the index of nested entities complying with the index declaration, and the index including the link.

2. The method of claim 1, further comprising:

optimizing the compiled rules, wherein optimizing the compiled rules comprises;

fusing multiple indexes in parallel.

3. The method of claim 1, wherein applying the entity population rule comprises:

applying a user defined function to the plurality of records.

4. The method of claim 1, wherein the at least one additional entity population rule includes a reference to the entity declaration, wherein:

the index of nested entities is shared between the entity population rule and the at least one additional entity population rule.

5. The method of claim 1, wherein the entity population rule defines an aggregation of the plurality of input records.

6. The method of claim 5, wherein the aggregation is computed by populating an index separate from the entity.

7. The method of claim 1, wherein the at least one nested entity comprises a nested set and the nested entity is populated separately from the entity.

8. The method of claim 1, wherein the plurality of output records comprise a reference to a nested entity by the nested entity referred to by a key to the index.

9. The method of claim 1, further comprising:

reading a second plurality of input records; and

applying the at least one additional entity population rule to populate the nested entity without modifying the output records.

10. The method of claim 1, wherein compiling the entity population rule comprises:

implementing the index as a function; and

implementing at least one index lookup as a call to the function.

11. The method of claim 1, wherein compiling the entity population rule comprises:

implementing the index as a binary table; and

implementing at least one index lookup as a join.

12. The method of claim 1, wherein compiling the entity population rule comprises:

coalescing the index and an additional index into a super-index.

13. The method of claim 1, wherein determining the link comprises determining a plurality of candidate links and selecting the strongest link from the plurality of candidate links.

14. The method of claim 1, further comprising:

applying the at least one additional entity population rule to create a second plurality of output records based on the link.

15. The method of claim 1, wherein compiling the entity resolution rule comprises:

separating where and check clauses of the entity resolution rule from group and cardinality clauses of the entity resolution rule;

using matching conditions of the where clauses to join and select from the output records;

incorporating conditions of the check clauses within each matching condition;

translating the group and cardinality clauses into a constraint query; and

executing the constraint queries.

16. A computer program product for entity population, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by a processor to:

read an entity declaration from a machine readable medium, the entity declaration describing an entity including at least one nested entity;

read an index declaration from a machine readable medium, the index declaration describing an index of nested entities;

read an entity population rule from a machine readable medium, the entity population rule describing a mapping from an input schema to an output schema, the output schema conforming to the entity declaration;

reading at least one additional entity population rule from the machine readable medium;

compiling the entity population rule and the at least one additional entity population rule into at least one executable query, wherein compiling the entity population rule comprises topologically sorting the entities defined by the entity population rule and the at least one additional entity population rule based on at least one dependency between the entity population rule and the at least one additional entity population rule and generating a query by traversing from a leaf to a root of the topologically sorted entities;

read a plurality of input records from a first data store, the input records conforming to the input schema;

read an entity resolution rule from a machine readable medium, the entity resolution rule defining criteria for a link between entities;

compile the entity resolution rule;

apply the entity resolution rule to the plurality of input records to determine a link between members of the plurality of input records;

apply the entity population rule to the plurality of input records to create a plurality of output records complying with the output schema;

subsequent to applying the entity population rule, recompile the entity population rule and the at least one additional entity population rule; and

populate the index of nested entities using the at least one executable query, the index of nested entities complying with the index declaration, and the index including the link.

17. The computer program product of claim 16, wherein the program code is further executable by a processor to:

optimize the compiled rules, wherein optimizing the compiled rules comprises;

fusing multiple indexes in parallel.

18. The computer program product of claim 16, wherein compiling the entity resolution rule comprises:

separating where and check clauses of the entity resolution rule from group and cardinality clauses of the entity resolution rule;

using matching conditions of the where clauses to join and select from the output records;

incorporating conditions of the check clauses within each matching condition;

translating the group and cardinality clauses into a constraint query; and

executing the constraint queries.

19. The computer program product of claim 16, wherein the at least one additional entity population rule includes a reference to the entity declaration, wherein:

the index of nested entities is shared between the entity population rule and the at least one additional entity population rule.

20. The computer program product of claim 16, wherein determining the link comprises determining a plurality of candidate links and selecting the strongest link from the plurality of candidate links.