Generic Data Staging and Loading Using Enhanced Metadata and Associated Method

• US 20080243884A1
• Filed: 03/27/2008
• Published: 10/02/2008
• Est. Priority Date: 03/29/2007
• Status: Active Grant

• Current Assignee: Blue Yonder Group, Inc. (Panasonic Holdings Corporation)
• Original Assignee: Blue Yonder Group, Inc. (Panasonic Holdings Corporation)
• Inventors: Mehta, Rubesh
• Granted Patent: US 10,387,440 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A system and method is disclosed for providing data staging and loading through a master data management (MDM) system coupled with one or more business entities. The system is operable to invoke batch data loading of data associated with one or more source systems associated with the one or more business entities, into an input staging area. The system is further operable to load the data from the input staging area into a master repository and subsequently load the data from the master repository into an output staging area.

Claims

1. A system for providing data staging and loading through a master data management (MDM) system coupled with one or more business entities, the master data management system operable to:

invoke batch data loading of data associated with one or more source systems into an input staging area;

load the data from the input staging area into a master repository; and

load the data from the master repository into an output staging area.

2. The system of claim 1, wherein the data associated with one or more source systems is fed in bulk mode as a file into the input staging area.

3. The system of claim 1, wherein the input staging area comprises a relational input staging interface.

4. The system of claim 3, wherein the data associated with one or more source systems is loaded into the input staging are through the relational input staging interface.

5. The system of claim 1, wherein the MDM system models the batch data loading as a workflow.

6. The system of claim 1, further operable to:

invoke a data loading workflow for processing the data loaded into the input staging area.

7. The system of claim 6, wherein the data load workflow is operable to execute one or more filter data business rules on the data loaded into the input staging area.

8. The system of claim 7, wherein the one or more filter data business rules is operable to:

capture the data loaded into the input staging area that satisfies the one or more filter data business rules; and

move the data that satisfies the one or more filter data business rules into a temporary data table.

9. The system of claim 6, wherein the data load workflow is operable to execute one or more auto correction rules on the data loaded into the input staging area.

10. The system of claim 6, wherein the data load workflow is operable to enable one or more database constrains operable to:

detect any errors on the data loaded into the input staging area; and

move the detected errors into an error table.

11. The system of claim 6, wherein the data load workflow is operable to execute one or more validation rules on the data loaded into the input staging area.

12. The system of claim 11, wherein the one or more validation rules, is operable to:

evaluate the data loaded into the input staging area; and

move any error records into an error table.

13. The system of claim 6, further operable to:

create a net change table of data associated with the input staging area with reference data; and

spool the net change table of data into a file.

14. A method for data staging and loading through a master data management (MDM) system, comprising:

invoking batch data loading of data associated with one or more source systems into an input staging area;

loading the data from the input staging area into a master repository; and

loading the data from the master repository into an output staging area.

15. The method of claim 14, wherein the data associated with one or more source systems is fed in bulk mode as a file into the input staging area.

16. The method of claim 14, wherein the input staging area comprises a relational input staging interface.

17. The method of claim 16, wherein the data associated with one or more source systems is loaded into the input staging are through the relational input staging interface.

18. The method of claim 14, wherein the MDM system models the batch data loading as a workflow.

19. The method of claim 14, further comprising:

invoking a data loading workflow for processing the data loaded into the input staging area.

20. The method of claim 19, wherein the data load workflow executes one or more filter data business rules on the data loaded into the input staging area.

21. The method of claim 20, wherein the one or more filter data business rules comprise:

capturing the data loaded into the input staging area that satisfies the one or more filter data business rules; and

moving the data that satisfies the one or more filter data business rules into a temporary data table.

22. The method of claim 19, wherein the data load workflow executes one or more auto correction rules on the data loaded into the input staging area.

23. The method of claim 19, wherein the data load workflow further comprises:

enabling one or more database constraints;

detecting any errors on the data loaded into the input staging area; and

moving the detected errors into an error table.

24. The method of claim 19, wherein the data load workflow executes one or more validation rules on the data loaded into the input staging area.

25. The method of claim 24, wherein the one or more validation rules, comprise:

evaluating the data loaded into the input staging area; and

moving any error records into an error table.

26. The method of claim 19, further comprising:

creating a net change table of data associated with the input staging area with reference data; and

spooling the net change table of data into a file.

27. Software for data staging and loading through a master data management (MDM) system, the software embodied in a computer-readable media and when executed operable to:

invoke batch data loading of data associated with one or more source systems into an input staging area;

load the data from the input staging area into a master repository; and

load the data from the master repository into an output staging area.

28. The software of claim 27, wherein the data associated with one or more source systems is fed in bulk mode as a file into the input staging area.

29. The software of claim 27, wherein the input staging area comprises a relational input staging interface.

30. The software of claim 29, wherein the data associated with one or more source systems is loaded into the input staging are through the relational input staging interface.

31. The software of claim 27, wherein the MDM system models the batch data loading as a workflow.

32. The software of claim 27, wherein the software is further operable to:

invoke a data loading workflow for processing the data loaded into the input staging area.

33. The software of claim 32, wherein the data load workflow is operable to execute one or more filter data business rules on the data loaded into the input staging area.

34. The software of claim 33, wherein the one or more filter data business rules is operable to:

capture the data loaded into the input staging area that satisfies the one or more filter data business rules; and

move the data that satisfies the one or more filter data business rules into a temporary data table.

35. The software of claim 32, wherein the data load workflow is operable to execute one or more auto correction rules on the data loaded into the input staging area.

36. The software of claim 32, wherein the data load workflow is operable to enable one or more database constrains operable to:

detect any errors on the data loaded into the input staging area; and

move the detected errors into an error table.

37. The software of claim 32, wherein the data load workflow is operable to execute one or more validation rules on the data loaded into the input staging area.

38. The software of claim 37, wherein the one or more validation rules, is operable to:

evaluate the data loaded into the input staging area; and

move any error records into an error table.

39. The software of claim 32, wherein the software is further operable to:

create a net change table of data associated with the input staging area with reference data; and

spool the net change table of data into a file.