External metadata processing

• US 20060015521A1
• Filed: 07/15/2004
• Published: 01/19/2006
• Est. Priority Date: 07/15/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Howey, James K., Bernhardt, Jeffrey R., Lee, Theodore T.
• Granted Patent: US 7,386,566 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

The operation of a data flow is improved by providing for validation of the consistency of the data flow with the external data files it works with, even in the face of changes in those external data files Metadata is stored regarding the external data files. Because external data files may change, this metadata is used when determining whether the external data files are still compatible with the data flow. In such performing such validation, the metadata tracks what changes in the external data files have occurred, and allows increased flexibility in presenting the user with options or in automatically fixing the data flow to correspond with changes in external data files. When the external data files are not available, the stored metadata can be used to validate the data flow to the last information regarding the external data files. The external metadata can be resynchronized to the data flow or to the external data files to provide updated external metadata for later validation.

Claims

1. A method for preparing data flow information regarding a data flow using at least one external data file, each of said external data files comprising columns of information, said method comprising:

storing a data flow model, said data flow model comprising use data describing, for each of said at least one external data file, a use of at least one column from said external data file;

storing metadata comprising data describing at least one column of each of said at least one external data file; and

using said stored metadata to determine whether said data flow model and said external data files are consistent.

2. The method of claim 1, where said data describing at least one column comprises type data relating to said at least one column.

3. The method of claim 1, where using said stored metadata to determine whether said data flow model and said external data files are consistent comprises:

for each external data file, verifying that said stored metadata is consistent with a current state of said external data file.

4. The method of claim 3, where verifying that said stored metadata is consistent with the current state of said external data file comprises:

determining if each column described in said stored metadata is present in said external data file; and

determining if each column in said external data file is described in said stored metadata.

5. The method of claim 4, where said data describing at least one column comprises type data describing a type of data stored in the column, and where verifying that said stored metadata is consistent with the current state of said external data file further comprises:

determining if said type data for each column described in said stored metadata is consistent with the type data for said column in said external data file.

6. The method of claim 3, where using said stored metadata to determine whether said data flow model and said external data files are consistent further comprises:

providing a validation state indicating whether said stored metadata is consistent with said current state of said external data file.

7. The method of claim 3, where using said stored metadata to determine whether said data flow model and said external data files are consistent further comprises:

if said stored metadata is not consistent with said current state of said external data file, resolving said inconsistency.

8. The method of claim 7, where said resolving of said inconsistency comprises:

querying the user to resolve said inconsistency.

9. The method of claim 7, where said resolving of said inconsistency comprises:

using an interface of said data flow to resolve said inconsistency.

10. The method of claim 1, further comprising:

resynchronizing said stored metadata to comprise data reflecting the current data stored in each of said external data file.

11. The method of claim 1, further comprising:

determining whether said data flow model and said external data files are consistent by determining if each said use of at least one column from said external data file described in said use data is possible given a current state for each of said external data file.

12. The method of claim 11, where determining whether said data flow model and said external data files are consistent by determining if each said use of at least one column from said external data file described in said use data is possible given a current state for each of said external data file comprises:

providing a validation state indicating whether said data flow model is consistent with said current state of said external data files.

13. The method of claim 11, where determining whether said data flow model and said external data files are consistent by determining if each said use of at least one column from said external data file described in said use data is possible given a current state for each of said external data file comprises:

if said stored metadata is not consistent with said current state of said external data files, resolving said inconsistency.

14. The method of claim 13, where said resolving of said inconsistency comprises:

querying the user to resolve said inconsistency.

15. The method of claim 13, where said resolving of said inconsistency comprises:

using an interface of said data flow to resolve said inconsistency.

16. The method of claim 11, further comprising:

resynchronizing said data flow to reflect said current state of said external data files.

17. The method of claim 1, where said using said stored metadata to determine whether said data flow model and said external data files are consistent comprises:

verifying that said stored metadata is consistent with said data flow.

18. The method of claim 17, where said verifying that said stored metadata is consistent with said data flow comprises:

determining if each said use of at least one column from said external data file described in said use data is consistent with said stored metadata.

19. The method of claim 18, further comprising:

if each said use of at least one column from said external data file described in said use data is not consistent with said stored metadata, resolving said inconsistency.

20. At least one of an operating system, a computer readable medium having stored thereon a plurality of computer-executable instructions, a co-processing device, a computing device, and a modulated data signal carrying computer executable instructions for performing the method of claim 1.

21. A method for evaluating a data flow model, said data flow model comprising information regarding a data flow, said information comprising use data describing, for each of at least one external data file, a use of at least one column from said external data file, said method comprising:

retrieving metadata comprising data describing at least one column of each of said at least one external data file; and

using said metadata to determine whether said data flow model and said metadata are consistent.

22. The method of claim 21, where using said stored metadata to determine whether said data flow model and said metadata are consistent comprises:

providing a validation state indicating whether said stored metadata is consistent with said metadata.

23. The method of claim 21, where using said metadata to determine whether said data flow model and said metadata are consistent further comprises:

if said stored metadata is not consistent with said current state of said external data file, resolving said inconsistency.

24. The method of claim 23, where said resolving of said inconsistency comprises:

querying the user to resolve said inconsistency.

25. The method of claim 23, where said resolving of said inconsistency comprises:

using an interface of said data flow to resolve said inconsistency.

26. The method of claim 21, further comprising:

resynchronizing said stored metadata to comprise data reflecting the current data stored in each of said external data file.

27. A data flow modeling system for preparing data flow information regarding a data flow using at least one external data file, each of said external data files comprising columns of information, said data flow modeling system comprising:

data flow model storage storing a data flow model, said data flow model comprising use data describing, for each of said at least one external data file, a use of at least one column from said external data file;

metadata storage storing metadata comprising data describing at least one column of each of said at least one external data file; and

consistency checker using said stored metadata to determine whether said data flow model and said external data files are consistent.

28. The data flow modeling system of claim 27, where said data describing at least one column comprises type data relating to said at least one column.

29. The data flow modeling system of claim 27, where said consistency checker, for each external data file, verifies that said stored metadata is consistent with a current state of said external data file.

30. The data flow modeling system of claim 29, where said consistency checker comprises:

validation state determiner for providing a validation state indicating whether said stored metadata is consistent with said current state of said external data file.

31. The data flow modeling system of claim 27, further comprising:

metadata resynchronizer modifying said stored metadata to comprise data reflecting the current data stored in each of said external data file.

32. The data flow modeling system of claim 27, where said consistency checker comprises:

consistency verifier for verifying that said stored metadata is consistent with said data flow.

33. The data flow modeling system of claim 32, where said consistency verifier determines if each said use of at least one column from said external data file described in said use data is consistent with said stored metadata.

34. The data flow modeling system of claim 32, further comprising:

resolver for, if each said use of at least one column from said external data file described in said use data is not consistent with said stored metadata, resolving said inconsistency.

37. The data flow modeling system of claim 34, further comprising:

inconsistency resolver for, if said stored metadata is not consistent with said current state of said external data file, resolving said inconsistency.

35. A data flow modeling system for evaluating a data flow model, said data flow model comprising information regarding a data flow, said information comprising use data describing, for each of at least one external data file, a use of at least one column from said external data file, said data flow modeling system comprising:

metadata storage storing metadata comprising data describing at least one column of each of said at least one external data file; and

consistency checker using said metadata to determine whether said data flow model and said metadata are consistent.

36. The data flow modeling system of claim 35, where said consistency checker comprises:

validation state indicator indicating whether said stored metadata is consistent with said metadata.

38. The data flow modeling system of claim 35, further comprising:

resynchronizer for resynchronizing said stored metadata to comprise data reflecting the current data stored in each of said external data file.