External metadata processing

• US 7,386,566 B2
• Filed: 07/15/2004
• Issued: 06/10/2008
• Est. Priority Date: 07/15/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Theodore T., Howey, James K., Bernhardt, Jeffrey R.
• Primary Examiner(s): Pardo; Thuy N
• Application Number:
  US10/891,609
• Publication Number:
  US 20060015521A1
• Time in Patent Office: 1,426 Days
• Field of Search: 707/200, 707/10, 707/101, 707/102, 707/103.Y, 707/104.1, 715/513, 709/310, 709/230, 709/238
• US Class Current: 707/691
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

The operation of a data flow is improved by providing for validation of the consistency of the data flow with the external data files it works with, even in the face of changes in those external data files Metadata is stored regarding the external data files. Because external data files may change, this metadata is used when determining whether the external data files are still compatible with the data flow. In such performing such validation, the metadata tracks what changes in the external data files have occurred, and allows increased flexibility in presenting the user with options or in automatically fixing the data flow to correspond with changes in external data files. When the external data files are not available, the stored metadata can be used to validate the data flow to the last information regarding the external data files. The external metadata can be resynchronized to the data flow or to the external data files to provide updated external metadata for later validation.

Claims

1. A method for preparing data flow information regarding a data flow using at least one external data file, each of said external data files comprising columns of information, said method comprising:

storing in a first computer readable medium, a data flow model, said data flow model comprising use data describing, for each of said at least one external data file, a use of at least one column from said external data file;

storing in a second computer readable medium, metadata comprising data describing at least one column of each of said at least one external data file;

performing a disconnected validation when the first computer readable medium is inaccessible, the disconnected validation comprising;

identifying a potential problem in said at least one external data file by comparing the stored metadata against said at least one external data file without reference to said data flow model; and

performing a connected validation when both the first computer readable medium and the second computer readable medium are accessible, the connected validation comprising;

using said stored metadata to determine whether said data flow model and said external data files are consistent.

2. The method of claim 1, where said data describing at least one column comprises type data relating to said at least one column.

3. The method of claim 1, where using said stored metadata to determine whether said data flow model and said external data files are consistent comprises:

for each external data file, verifying that said stored metadata is consistent with a current state of said external data file.

4. The method of claim 3, where verifying that said stored metadata is consistent with the current state of said external data file comprises:

determining if each column described in said stored metadata is present in said external data file; and

determining if each column in said external data file is described in said stored metadata.

5. The method of claim 4, where said data describing at least one column comprises type data describing a type of data stored in the column, and where verifying that said stored metadata is consistent with the current state of said external data file further comprises:

determining if said type data for each column described in said stored metadata is consistent with the type data for said column in said external data file.

6. The method of claim 3, where using said stored metadata to determine whether said data flow model and said external data files are consistent further comprises:

providing a validation state indicating whether said stored metadata is consistent with said current state of said external data file.

7. The method of claim 3, where using said stored metadata to determine whether said data flow model and said external data files are consistent further comprises:

if said stored metadata is not consistent with said current state of said external data file, resolving said inconsistency.

8. The method of claim 7, where said resolving of said inconsistency comprises:

querying the user to resolve said inconsistency.

9. The method of claim 7, where said resolving of said inconsistency comprises:

using an interface of said data flow to resolve said inconsistency.

10. The method of claim 1, further comprising:

resynchronizing said stored metadata to comprise data reflecting the current data stored in each of said external data file.

11. The method of claim 1, further comprising:

determining whether said data flow model and said external data files are consistent by determining if each said use of at least one column from said external data file described in said use data is possible given a current state for each of said external data file.

12. The method of claim 11, where determining whether said data flow model and said external data files are consistent by determining if each said use of at least one column from said external data file described in said use data is possible given a current state for each of said external data file comprises:

providing a validation state indicating whether said data flow model is consistent with said current state of said external data files.

13. The method of claim 11, where determining whether said data flow model and said external data files are consistent by determining if each said use of at least one column from said external data file described in said use data is possible given a current state for each of said external data file comprises:

if said stored metadata is not consistent with said current state of said external data files, resolving said inconsistency.

14. The method of claim 13, where said resolving of said inconsistency comprises:

querying the user to resolve said inconsistency.

15. The method of claim 13, where said resolving of said inconsistency comprises:

using an interface of said data flow to resolve said inconsistency.

16. The method of claim 11, further comprising:

resynchronizing said data flow to reflect said current state of said external data files.

17. The method of claim 1, where identifying the potential problem comprises:

verifying that said stored metadata is consistent with said data flow.

18. The method of claim 17, where said verifying that said stored metadata is consistent with said data flow comprises:

determining if each said use of at least one column from said external data file described in said use data is consistent with said stored metadata.

19. The method of claim 18, further comprising:

if each said use of at least one column from said external data file described in said use data is not consistent with said stored metadata, resolving said inconsistency.