MAP INTUITION SYSTEM AND METHOD

• US 20120023261A1
• Filed: 05/10/2011
• Published: 01/26/2012
• Est. Priority Date: 05/10/2010
• Status: Active Grant

• Current Assignee: Open Text Holdings, Inc. (Open Text Corporation)
• Original Assignee: Liaison Technologies Incorporated (Open Text Corporation)
• Inventors: Lindsay, Walter Hughes, Shellman, Joel Timothy
• Granted Patent: US 8,850,071 B2
• Time in Patent Office: Days
• US Class Current: 709/246
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A map intuition system and method that involves machine learning techniques to analyze data sets and identify mappings and transformation rules as well as machine-human interactions to leverage human intuition and intelligence to rapidly complete a map.

Claims

1. A map intuition method, comprising:

a) providing at least one data flow comprising a first transformation, the first transformation having data inputs and data outputs;

b) capturing a plurality of first input samples from the data flow prior to the first transformation and a corresponding plurality of first output samples from the data flow subsequent the first transformation;

c) providing an inference engine;

d) feeding the first input samples and the corresponding first output samples into the inference engine, wherein the inference engine produces at least one candidate list comprising a plurality of candidates from which to choose and form at least a partial second transformation definition;

e) reviewing the at least one candidate list;

f) selecting a candidate from the at least one candidate list to at least partially complete the second transformation definition; and

g) saving the second transformation definition.

2. The map intuition method of claim 1, wherein the inference engine ranks at least a portion of the plurality of candidates and visually highlights the candidates with higher rankings.

3. The map intuition method of claim 1, wherein each data input and each data output comprises at least one file.

4. The map intuition method of claim 3, wherein each at least one file comprises a plurality of data fields, wherein each data field has a field value and a field type.

5. The map intuition method of claim 4, wherein steps (b) through (g) are repeated to further define the second transformation.

6. The map intuition method of claim 1, wherein the second transformation produces a second output sample.

7. The map intuition method of claim 5, wherein the first and second output samples are substantially equal given the same data input samples.

8. The map intuition method of claim 5, wherein the at least one candidate list comprises a plurality of candidate lists.

9. The map intuition method of claim 7, wherein the steps of reviewing the at least partial second transformation definition and at least one candidate list and selecting a candidate from the at least one candidate list to complete the second transformation definition comprises:

a) specifying selection criteria for candidates;

b) selecting at least one candidate from at least one of the candidate lists;

c) comparing a second data output sample from the second transformation definition to the first data output sample to produce a relative difference;

d) determining if the relative difference is within a predetermined criteria; and

e) if the relative difference is not within the predetermined criteria, modifying the second transformation definition and returning to step (c).

10. The map intuition method of claim 5, wherein the step of using the inference engine to produce at least one candidate list comprising a plurality of candidates from which to choose and form at least a partial second transformation definition comprises:

identifying value spaces relevant to the field type and field value of data in each of the first input sample and second output sample;

comparing field values of the first input sample to field values of the second output sample to provide a numerical degree of correspondence for each; and

visually highlighting the candidates with a higher numerical degree of correspondence.

11. An inference engine method for ranking and highlighting candidates in a map intuition system, comprising:

providing a source data set having data with input field values, each source data set having a source data set definition describing the data in the source data set;

providing a target data set having data with output field values Vo, each target data set having a target data set definition describing the data in the target data set;

comparing each input field value with each output field value;

identifying a degree of correspondence between each pair of values;

ranking the source and output field values into clusters; and

merging the rankings from a data level to a data definition level.

12. The inference engine method of claim 11, wherein the step of comparing each input field value with each output field value comprises:

determining whether the output field value is equivalent to the input field value;

determining whether the output field value is a subset of the input field value;

determining whether the output field value is able to be partially constructed from the input field value; and

determining how much of the output field value can be constructed.

13. The inference engine of claim 11, wherein the step of ranking the source and output field values into clusters comprises:

identifying, for each output field value Vo, a set No of other output field values within a predetermined distance from Vo;

identifying a set S of input items Vs related to No;

identifying, for each item Vs in S, a set Ni of other input field values within a second predetermined distance from Vs;

ranking at least one of the size, quality of match, and distinctiveness of match of the set Ni compared to Ni and No; and

determining, based upon a predetermined threshold, whether to increase the value of the degree of correspondence between items in Ni to items in No.

14. The inference engine of claim 13, wherein the step of merging the rankings from a data level to a data definition level comprises:

examining each target data format definition in No;

identifying a set Co of correlations directly involving No, any descendants in the data format definition of No, or all descendants Do of an ancestor of No;

examining each source data format definition in Ni that appear in Co;

identifying the set Ci of correlations directly involving Ni, all descendants in the data format definition of Ni, and all descendants Di of an ancestor of Ni;

evaluating the quality of matches between Co and Ci;

identifying a numerical rating for the correlation; and

recording the match between Co and Ci as part of an output of the inference engine.

15. The inference engine of claim 14, wherein the step of examining the target data format definition comprises performing a post-order traversal.

16. The inference engine of claim 14, wherein the step of examining the source data format definition comprises performing a post-order traversal.

17. A map intuition method for use with a map intuition system utilized by at least one user, comprising:

a) providing at least one data flow comprising a first transformation, the first transformation having data inputs and data outputs;

b) capturing a plurality of first input samples from the data flow prior to the first transformation and a corresponding plurality of first output samples from the data flow subsequent the first transformation;

c) providing an inference engine;

d) feeding the first input samples and the corresponding first output samples into the inference engine, wherein the inference engine attempts to produce at least one candidate list comprising a plurality of candidates from which to choose and form at least a partial second transformation definition, and wherein if the inference engine fails to provide a candidate, the user produces the at least one candidate list comprising a plurality of candidates from which to choose and form at least a partial second transformation definition;

e) reviewing the at least one candidate list;

f) selecting a candidate from the at least one candidate list to at least partially complete the second transformation definition; and

g) saving the second transformation definition.

18. The map intuition method of claim 17, wherein the inference engine ranks at least a portion of the plurality of candidates and visually highlights the candidates with higher rankings.

19. The map intuition method of claim 17, wherein each data input and each data output comprises at least one file.

20. The map intuition method of claim 19, wherein each at least one file comprises a plurality of data fields, wherein each data field has a field value and a field type.

21. The map intuition method of claim 20, wherein steps (b) through (g) are repeated to further define the second transformation.

22. The map intuition method of claim 21, wherein the second transformation produces a second output sample.

23. The map intuition method of claim 21, wherein the first and second output samples are substantially equal given the same data input samples.

24. The map intuition method of claim 21, wherein the at least one candidate list comprises a plurality of candidate lists.

25. The map intuition method of claim 17, wherein when the user produces the at least one candidate list comprising a plurality of candidates, the map intuition system comprises a graphical user interface that displays input and output data organized according to locality.

26. The map intuition method of claim 17, wherein when the user produces the at least one candidate list comprising a plurality of candidates, the map intuition system comprises search function to permit the user to search previous mappings of the data format.

27. The map intuition method of claim 17, wherein when the user produces the at least one candidate list comprising a plurality of candidates, the map intuition system comprises a library of common functions for which the user use to search for common data conversions.