MAPPING INSTANCES OF A DATASET WITHIN A DATA MANAGEMENT SYSTEM

• US 20100138388A1
• Filed: 12/01/2009
• Published: 06/03/2010
• Est. Priority Date: 12/02/2008
• Status: Abandoned Application

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Software Corporation
• Inventors: Weiss, Adam, Wakeling, Tim
• Application Number:
  US12/628,521
• Publication Number:
  US 20100138388A1
• Time in Patent Office: Days
• US Class Current: 707/634
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/84 Mapping; Conversion
• CPC Class Codes: G06F 40/197 Version control for software G06F8/71

Abstract

Mapping data stored in a data storage system for use by a computer system includes processing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data. At least one of the dataflow graphs receives a flow of data from at least one input dataset and at least one of the dataflow graphs provides a flow of data to at least one output dataset. A mapper identifies one or more sets of datasets. Each dataset in a given set matches one or more criteria for identifying different versions of a single dataset. A user interface is provided to receive a mapping between at least two datasets in a given set. The mapping received over the user interface is stored in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.

Claims

1. A method for mapping data stored in a data storage system for use by a computer system, the method including:

processing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;

identifying one or more sets of datasets, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset, each version of the single dataset representing data received or provided by a different one of the dataflow graphs;

providing a user interface to receive a mapping between at least two datasets in a given set; and

storing the mapping received over the user interface in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.

2. The method of claim 1, including presenting the set over the user interface.

3. The method of claim 1, including presenting over the user interface a list of possible mappings ordered according to a quantification of a match to the one or more criteria.

4. The method of claim 3, wherein the list of possible mappings includes candidates that are more likely to be an instance of a given dataset ordered higher in the list.

5. The method of claim 3, wherein one of the criteria is built into a mapper that identifies the one or more sets of datasets.

6. The method of claim 3, wherein one of the criteria is received from the user interface.

7. The method of claim 3, wherein at least one of the possible mappings indicates a component of a dataflow graph that represents a dataset, and at least one of the possible mappings indicates a component of a dataflow graph that does not represent a dataset.

8. The method of claim 1, wherein a sub-graph of a dataflow graph including multiple components represents a dataset.

9. The method of claim 8, wherein the sub-graph includes a data component.

10. The method of claim 8, wherein the sub-graph includes an executable component.

11. The method of claim 1, wherein identifying one or more sets of datasets includes using heuristics for determining if a dataset in a given set has one or more characteristics in common with another dataset.

12. The method of claim 11, wherein the characteristics include the quantity of bytes and records in a representation of the dataset.

13. The method of claim 11, wherein the characteristics include the name of a representation of the dataset.

14. The method of claim 11, wherein the characteristics include the date of creation of a representation of the dataset.

15. The method of claim 11, wherein the characteristics include the data format of a representation of the dataset.

16. The method of claim 1, wherein at least one of the datasets of the mapping belongs to a group of datasets known to a data management system.

17. The method of claim 1, further including providing a format mapping between datasets in a given set.

18. The method of claim 1, wherein the mapping includes an identifier that points to a record in the data management system that keeps track of the dataset.

19. The method of claim 1, further including updating the mapping based on a change in a dataset.

41. The method of claim 1, wherein each version of a single dataset is associated with a different graph, graph subset, or executable component.

42. The method of claim 1, wherein each version of a single dataset is stored in a different location associated with the data storage system.

43. The method of claim 1, wherein each version of a single dataset is interpreted using a different data storage format.

44. The method of claim 1, wherein each version of a single dataset is accessed using a parameter that varies between executions of the dataflow graph.

20. A system for mapping data stored in a data storage system, the system including

a data storage system storing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;

a mapper that identifies one or more sets of datasets associated with the dataflow graphs, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset, each version of the single dataset representing data received or provided by a different one of the dataflow graphs;

a user interface that receives a mapping between at least two datasets in a given set, and stores the mapping in the data storage system in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.

21. The system of claim 20, wherein the user interface presents the set.

22. The system of claim 20, wherein the user interface presents a list of possible mappings ordered according to a quantification of a match to the one or more criteria.

23. The system of claim 22, wherein the list of possible mappings includes candidates that are more likely to be an instance of a given dataset ordered higher in the list.

24. The system of claim 22, wherein one of the criteria is built into the mapper.

25. The system of claim 22, wherein one of the criteria is received by the user interface.

26. The system of claim 22, wherein at least one of the possible mappings indicates a component of a dataflow graph that represents a dataset, and at least one of the possible mappings indicates a component of a dataflow graph that does not represent a dataset.

27. The system of claim 20, wherein a sub-graph of a dataflow graph including multiple components represents a dataset.

28. The system of claim 27, wherein the sub-graph includes a data component.

29. The system of claim 27, wherein the sub-graph includes an executable component.

30. The system of claim 20, wherein the mapper uses heuristics for determining if a dataset in a given set has one or more characteristics in common with another dataset.

31. The system of claim 30, wherein the characteristics include the quantity of bytes and records in a representation of the dataset.

32. The system of claim 30, wherein the characteristics include the name of a representation of the dataset.

33. The system of claim 30, wherein the characteristics include the date of creation of a representation of the dataset.

34. The system of claim 30, wherein the characteristics include the data format of a representation of the dataset.

35. The system of claim 20, wherein at least one of the datasets of the mapping belongs to a group of datasets known to a data management system.

36. The system of claim 20, wherein the mapper generates a format mapping between datasets in a given set.

37. The system of claim 20, wherein the mapping includes an identifier that points to a record in the data management system that keeps track of the dataset.

38. The system of claim 20, wherein the mapper updates the mapping based on a change in a dataset.

39. A system for mapping data stored in a data storage system, the system including:

means for processing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;

means for identifying one or more sets of datasets, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset each version of the single dataset representing data received or provided by a different one of the dataflow graphs;

means for providing a user interface to receive a mapping between at least two datasets in a given set; and

means for storing the mapping received over the user interface in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.

40. A computer-readable medium storing a computer program for mapping data stored in a data storage system, the computer program including instructions for causing a computer to:

process specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;

identify one or more sets of datasets, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset, each version of the single dataset representing data received or provided by a different one of the dataflow graphs;

provide a user interface to receive a mapping between at least two datasets in a given set; and

store the mapping received over the user interface in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.