Computer-implemented system and method for managing data integrity validation rules

• US 7,318,162 B2
• Filed: 12/18/2002
• Issued: 01/08/2008
• Est. Priority Date: 12/18/2002
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Rineer, Brian C., Jackson, Brent C., Fisher, Anthony F., Gidley, Scott A.
• Primary Examiner(s): Barrón, Jr.; Gilberto
• Assistant Examiner(s): HERRING, VIRGIL A
• Application Number:
  US10/324,134
• Publication Number:
  US 20040123101A1
• Time in Patent Office: 1,847 Days
• Field of Search: 713/193, 707/101, 707/203
• US Class Current: 713/193
• CPC Class Codes: G06F 21/645 using a third party
• CPC Class Codes: H04L 63/123 received data contents, e.g. message integrity
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A computer-implemented system and method for customizing rules used in data integrity validation operations. A data integrity validation application uses the data integrity validation rules to perform the data integrity validation operations. The data integrity validation rules are stored in a knowledge base which is separate from and external to the data integrity validation application. The separately stored and externally located knowledge base allows customization of the data integrity validation rules without requiring recompilation of the data integrity validation application.

Claims

1. A computer-implemented system having a processor, said system for customizing rules used in data integrity validation operations, comprising:

a data integrity validation application configured for operation upon the processor in order to access the data integrity validation rules to perform the data integrity validation operations,

a database containing a plurality of data records for processing by the data integrity validation application;

said processing by the data integrity validation application including receiving the plurality of data records from the database and performing the data integrity validation operations upon the plurality of records that are contained in the database;

wherein the data integrity validation application is stored in a low level compiled language;

a knowledge base containing the data integrity validation rules, wherein the knowledge base has a storage location that is separate from and external to the compiled data integrity validation application,

wherein the separate and external storage location of the knowledge base allows customization of the contained data integrity validation rules without requiring recompilation of the data integrity validation application,

wherein the data integrity validation application has a data access pathway to the separately stored and externally located knowledge base so that the data integrity validation application may perform the data integrity validation operations upon the plurality of data records.

2. The system of claim 1 wherein the compiled data integrity validation application contains an engine that is capable of accessing the data integrity validation rules in the knowledge base and applying the rules to perform the data integrity validation operations.

3. The system of claim 1 wherein the low level compiled language is not in a human-readable format.

4. The system of claim 1 wherein the data integrity validation application comprises a plurality of software modules, wherein at least one of the software modules performs the data integrity validation operations.

5. The system of claim 1 wherein the separate and external storage location of the knowledge base relative to the compiled data integrity validation application allows the data integrity validation rules to be customized without requiring recompilation of the document management application.

6. The system of claim 1 wherein the knowledge base comprises a database to store the data integrity validation rules.

7. The system of claim 1 wherein the knowledge base comprises a set of flat files to store the data integrity validation rules.

8. The system of claim 1 wherein the knowledge base is extensible in that existing data integrity validation rules may be added or updated.

9. The system of claim 1 wherein the data integrity validation rules are directed to cleansing number-based data.

10. The system of claim 1 wherein the data integrity validation rules are directed to cleansing text-based data.

11. The system of claim 1 wherein the data integrity validation rules comprise semantic cleansing rules.

12. The system of claim 1 wherein the data integrity validation rules are directed to analysis-related data cleansing operations.

13. The system of claim 1 wherein the data integrity validation rules are directed to standardization-related data cleansing operations.

14. The system of claim 1 wherein the data integrity validation rules are directed to augmentation-related data cleansing operations.

15. The system of claim 1 wherein the data integrity validation rules are directed to integration-related data cleansing operations.

16. The system of claim 1 wherein the data integrity validation rules in the knowledge base contain analysis reference materials and transformation specifications.

17. The system of claim 1 wherein a first computer server contains the data integrity validation application and a second computer server contains the knowledge base, wherein a network provides data access between the first and second computer servers.

18. The system of claim 17 wherein the network is an Internet network.

19. The system of claim 17 wherein a webbrowser located on a remote client computer accesses the data integrity validation rules stored in the knowledge base in order to customize the data integrity validation rules.

20. The system of claim 1 further comprising:

customization tools that access the data integrity validation rules in the knowledge base and contains at least one user interface to customize the rules.

21. The system of claim 20 wherein the customization tools contains at least one user interface related to testing the data integrity validation rules stored in the knowledge base.

22. The system of claim 20 wherein the customization tools contains at least one user interface related to debugging the data integrity validation rules stored in the knowledge base.

23. The system of claim 22 wherein the user interface is used to view intermediate results of the data integrity validation operations.

24. The system of claim 1 further comprising:

an import and export software facility for transferring data integrity validation rules from the knowledge base to another knowledge base for use in data integrity validation operations.

25. The system of claim 1 wherein the data integrity validation rules comprise data cleansing rules.

26. The system of claim 1 wherein the data integrity validation rules comprise data integration rules.

27. The system of claim 1 wherein the data integrity validation application uses the data integrity validation rules to determine whether duplicate records exist within the plurality of records that are contained in the database.

28. The system of claim 1 wherein another database contains a plurality of data records, wherein the data integrity validation application uses the data integrity validation rules to consolidate the plurality of data records contained within the databases;

wherein the consolidation of the plurality of data records contained within the databases results in a merging of the data records that were contained within the databases and wherein duplicate data records between the databases are purged.

29. A computer-implemented method for managing rules used in data integrity validation operations, comprising the steps of:

retrieving from a knowledge base a data integrity validation rule;

modifying the retrieved data integrity validation rule;

storing the modified data integrity validation rule so that the modified data integrity rule may be used in a data integrity validation operation to be performed by a data integrity validation application, wherein the data integrity validation application is stored in a low level compiled language,

wherein the data integrity validation application retrieves from a database a plurality of data records for processing by the data integrity validation application;

said processing by the data integrity validation application including performing the data integrity validation operation upon the plurality of records that are contained in the database;

wherein the knowledge base has a storage location that is separate from and external to the compiled data integrity validation application,

wherein the separate and external storage location of the knowledge base allows modification of the stored data integrity validation rules without requiring recompilation of the data integrity validation application.

30. A computer-implemented apparatus for managing rules used in data integrity validation operations, comprising:

means for retrieving from a knowledge base a data integrity validation rule;

means for modifying the retrieved data integrity validation rule;

means for storing the modified data integrity validation rule so that the modified data integrity rule may be used in a data integrity validation operation to be performed by a data integrity validation application, wherein the data integrity validation application is stored in a low level compiled language,

wherein the data integrity validation application retrieves from a database a plurality of data records for processing by the data integrity validation application;

said processing by the data integrity validation application including performing the data integrity validation operation upon the plurality of records that are contained in the database;

wherein the knowledge base has a storage location that is separate from and external to the compiled data integrity validation application,

wherein the separate and external storage location of the knowledge base allows modification of the stored data integrity validation rules without requiring recompilation of the data integrity validation application.