Scheduling for data warehouse ETL processing and data mining execution

• US 7,058,615 B2
• Filed: 04/24/2003
• Issued: 06/06/2006
• Est. Priority Date: 04/24/2003
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: International Business Machines Corporation
• Inventors: Yao, Albert Zhongxing
• Primary Examiner(s): Knight, Anthony
• Assistant Examiner(s): Holmes, Michael B.
• Application Number:
  US10/422,675
• Publication Number:
  US 20040215584A1
• Time in Patent Office: 1,139 Days
• Field of Search: 706/12, 707/100, 707 1- 10, 707/204
• US Class Current: 706/12
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

Providing flexibility in scheduling control for an extract, transfer, and load (“ETL”) scheduler in a data warehouse system, including extracting from a computer memory of the data warehouse system a dependent condition for a scheduled ETL processing step of the ETL scheduler, extracting from computer memory related ETL data for the dependent condition, determining whether the ETL data satisfies the dependent condition, and executing the scheduled step if the ETL data satisfies the dependent condition. A sub-scheduler typically is implemented as an interface between an ETL scheduler and external programs without modification of the ETL scheduler. In typical embodiments, an ETL scheduler includes a multiplicity of scheduled ETL processing steps, and computer memory of a data warehouse system includes dependent conditions for at least one of the scheduled ETL processing steps and ETL data related to at least one dependent condition.

Claims

1. A method for providing flexibility in scheduling control for an extract, transfer, and load (“

ETL”

) scheduler in a data warehouse system, the method comprising;

extracting from a computer memory of the data warehouse system a dependent condition for a scheduled ETL processing step of the ETL scheduler;

extracting from computer memory related ETL data for the dependent condition;

determining whether the ETL data satisfies the dependent condition; and

executing the scheduled step if the ETL data satisfies the dependent condition.

2. The method of claim 1 wherein steps of the method are carried out through a sub-scheduler implemented as an interface between the ETL scheduler and external programs without modification of the ETL scheduler.

3. The method of claim 1 wherein:

the ETL scheduler comprises a multiplicity of scheduled ETL processing steps,

the computer memory of the data warehouse system comprises dependent conditions for at least one of the scheduled ETL processing steps and ETL data related to at least one dependent condition, and

the method further comprises repeating, for each dependent condition for each scheduled ETL processing step for which the operations database has at least one dependent condition, the steps of;

extracting from computer memory a dependent condition for a scheduled ETL processing step of the ETL scheduler;

extracting from computer memory related ETL data for the dependent condition;

determining whether the ETL data satisfies the dependent condition; and

executing the scheduled step if the ETL data satisfies the dependent condition.

4. The method of claim 1 further comprising storing ETL data in computer memory.

5. The method of claim 1 wherein the dependent condition comprises a time dependent condition.

6. The method of claim 1 wherein the dependent condition comprises a system dependent condition.

7. The method of claim 1 wherein the related ETL data comprises extract time data.

8. The method of claim 1 wherein the related ETL data comprises extract system data.

9. A system for providing flexibility in scheduling control for an extract, transfer, and load (“

ETL”

) scheduler in a data warehouse system, the system comprising;

means for extracting from a computer memory of the data warehouse system a dependent condition for a scheduled ETL processing step of the ETL scheduler;

means for extracting from computer memory related ETL data for the dependent condition;

means for determining whether the ETL data satisfies the dependent condition; and

means for executing the scheduled step if the ETL data satisfies the dependent condition.

10. The system of claim 9 further comprising a sub-scheduler implemented as an interface between the ETL scheduler and external programs without modification of the ETL scheduler.

11. The system of claim 9 wherein:

the ETL scheduler comprises a multiplicity of scheduled ETL processing steps,

the computer memory of the data warehouse system comprises dependent conditions for at least one of the scheduled ETL processing steps and ETL data related to at least one dependent-condition, and

the system further comprises means for repeating, for each dependent condition for each scheduled ETL processing step for which the operations database has at least one dependent condition,

means for extracting from computer memory a dependent condition for a scheduled ETL processing step of the ETL scheduler;

means for extracting from computer memory related ETL data for the dependent condition;

means for determining whether the ETL data satisfies the dependent condition; and

means for executing the scheduled step if the ETL data satisfies the dependent condition.

12. The system of claim 9 further comprising means for storing ETL data in computer memory.

13. The system of claim 9 wherein the dependent condition comprises a time dependent condition.

14. The system of claim 9 wherein the dependent condition comprises a system dependent condition.

15. The system of claim 9 wherein the related ETL data comprises extract time data.

16. The system of claim 9 wherein the related ETL data comprises extract system data.

17. A computer program product for providing flexibility in scheduling control for an extract, transfer, and load (“

ETL”

) scheduler in a data warehouse system, the computer program product comprising;

a recording medium;

means, recorded on the recording medium, for extracting from a computer memory of the data warehouse system a dependent condition for a scheduled ETL processing step of the ETL scheduler;

means, recorded on the recording medium, for extracting from computer memory related ETL data for the dependent condition;

means, recorded on the recording medium, for determining whether the ETL data satisfies the dependent condition; and

means, recorded on the recording medium, for executing the scheduled step if the ETL data satisfies the dependent condition.

18. The computer program product of claim 17 further comprising a sub-scheduler implemented as an interface between the ETL scheduler and external programs without modification of the ETL scheduler.

19. The computer program product of claim 17 wherein:

the ETL scheduler comprises a multiplicity of scheduled ETL processing steps,

the computer memory of the data warehouse system comprises dependent conditions for at least one of the scheduled ETL processing steps and ETL data related to at least one dependent condition, and

the computer program product further comprises means, recorded on the recording medium, for repeating, for each dependent condition for each scheduled ETL processing step for which the operations database has at least one dependent condition,

means, recorded on the recording medium, for extracting from computer memory a dependent condition for a scheduled ETL processing step of the ETL scheduler;

means, recorded on the recording medium, for extracting from computer memory related ETL data for the dependent condition;

means, recorded on the recording medium, for determining whether the ETL data satisfies the dependent condition; and

means, recorded on the recording medium, for executing the scheduled step if the ETL data satisfies the dependent condition.

20. The computer program product of claim 17 further comprising means, recorded on the recording medium, for storing ETL data in computer memory.

21. The computer program product of claim 17 wherein the dependent condition comprises a time dependent condition.

22. The computer program product of claim 17 wherein the dependent condition comprises a system dependent condition.

23. The computer program product of claim 17 wherein the related ETL data comprises extract time data.

24. The computer program product of claim 17 wherein the related ETL data comprises extract system data.