Statistical quality monitoring and enhancement

• US 8,140,476 B2
• Filed: 12/16/2009
• Issued: 03/20/2012
• Est. Priority Date: 12/16/2009
• Status: Active Grant

• Current Assignee: ServiceNow Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Aharoni, Ehud, Rosset, Saharon, Neuvirth, Hani
• Primary Examiner(s): Vo, Tim T
• Assistant Examiner(s): SMITH, GARRETT A
• Application Number:
  US12/639,005
• Publication Number:
  US 20110145291A1
• Time in Patent Office: 825 Days
• Field of Search: None
• US Class Current: 707/634
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27

Abstract

A method and computer system for monitoring and controlling the quality of tests performed upon a database. A statistical quality layer is provided, which for each new test to be executed on the database, provides recommendations as to a confidence level to be applied to the test and to a number of additional data records to be added to the database in order to maintain the total expected number of type-I errors of tests performed upon the database. The method and computer system further provide access control of users to the database.

Claims

1. A computerized system having a processor, said computerized system comprising a database management component, the database management component comprising:

a statistical quality layer for controlling quality parameters of a test performed upon a database, the statistical quality layer comprising a compensation determination component for determining a number of data records to be added to the database so as to limit an expected number of errors associated with the database once the test is performed; and

wherein the number of data records is determined such that L(n+δ

) <

aqⁿ−

aq^n+δ

, wherein L is a significance level associated with the test when executed upon n+δ

samples, n is a number of records in the database, δ

is the number of additional data records, q is a number between 0 and 1, and a is the predetermined quantity.

2. The computerized system of claim 1 further comprising a recommendation engine for issuing a recommendation related to the number of data records to be added to the database determined by said compensation determination component.

3. The computerized system of claim 1 further comprising a recommendation engine for issuing a recommendation related to a significance level of the test.

4. The computerized system of claim 1 further comprising a test execution component for executing the test.

5. The computerized system of claim 1 further comprising a report engine for generating a report related to the database or to the test performed upon the database.

6. The computerized system of claim 1 further comprising a manager user interface for receiving a command from a manager of the database.

7. The computerized system of claim 1 further comprising a researcher user interface for receiving a command from a user associated with a test to be performed upon the database.

8. The computerized system of claim 1 further comprising a control component for managing control and information flow within the statistical quality layer.

9. The computerized system of claim 1 further comprising an access control engine for controlling access of a user of the computerized system to the database.

10. The computerized system of claim 1, wherein the errors are type-I errors.

11. The computerized system of claim 1, wherein the recommendation relates to allowing the test, forbidding the test, or stipulating the test upon a condition.

12. method for controlling a database stored on a storage device comprising:

receiving a request for a test to be executed upon the database;

determining a significance level for the test;

determining a number of data records to be added to the database so that the test is in compliance with the significance level, wherein the number of data records is determined such that L(n+δ

)<

aqⁿ−

aq^n+δ

, wherein L is a significance level associated with the test when executed upon n+δ

samples, n is a number of records in the database, δ

is the number of additional data records. q is a number between 0 and 1, and a is the predetermined quantity;

receiving data records as indicated by the number of data records and storing the data records in the database; and

whereby the database is transformed to further include the data records.

13. The method of claim 12 further comprising executing the test upon the database.

14. The method of claim 12, wherein the number of additional data records is determined in accordance with an expected total number of type-I errors related to the database, and with the size of the database.

15. The method of claim 12 further comprising issuing a recommendation related to allowing the test, forbidding the test, or stipulating the test upon a condition.

16. The method of claim 12 further comprising controlling access of a user to the database.

17. The method of claim 12 wherein a manager of the database determines the significance level for the test and the number of additional data records to be added to the database, based upon at least one parameter supplied by a user of the test.

18. The method of claim 17 wherein the at least one parameter is selected from the group consisting of:

power;

effect size;

statistics of the test, and assumptions on data relevant for the test.

19. A computer program product comprising:

a computer readable media comprising program instructions, wherein execution of the program instructions by a processor causes the processor to carry out the steps of;

receiving a request for a test to be executed upon the database;

determining a significance level for the test;

determining a number of data records to be added to the database so that the test is in compliance with the significance level, wherein the number of data records is determined such that L(n+δ

)<

aqⁿ−

aq^n+δ

, wherein L is a significance level associated with the test when executed upon n+δ

samples, n is a number of records in the database, δ

is the number of additional data records, q is a number between 0 and 1, and a is the predetermined quantity.;

receiving data records as indicated by the number of data records and storing the data records in the database; and

whereby the database is transformed to further include the data records.

20. A method for controlling a database stored on a storage device comprising:

receiving a request for a test to be executed upon the database;

determining a significance level for the test;

determining a number of data records to be added to the database so that the test is in compliance with the significance level, wherein the number of additional data records is determined such that L₁(n+δ

)+L₂(n+δ

)+. . .+L_t(n+δ

)<

a, wherein;

L₁, L₂, . . . , L<sub>t−

1</sub> are confidence level functions for previously submitted tests;

L_tis confidence level functions for the test;

n is a number of records in the database;

δ

is the number of additional data records; and

a is the predetermined quantity;

receiving data records as indicated by the number of data records and storing the data records in the database; and

whereby the database is transformed to further include the data records.

21. The method of claim 20 further comprising re-executing tests upon the database, in response to said receiving additional data records.

22. The method of claim 20, wherein the number of additional data records is determined such that current levels of all tests which have been previously executed, sums up to a number smaller than a predetermined quantity.