Specifying a predetermined degree of inconsistency for test data

• US 9,235,607 B1
• Filed: 03/29/2013
• Issued: 01/12/2016
• Est. Priority Date: 03/29/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ross, Max C., Fuller, Alfred R. K.
• Primary Examiner(s): Mofiz, Apu
• Assistant Examiner(s): Bibbee, Jared
• Application Number:
  US13/853,985
• Time in Patent Office: 1,019 Days
• Field of Search: 707/690
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors

Abstract

A method and a test system for specifying a predetermined degree of inconsistency for test data are disclosed. The test system obtains a test policy, which specifies a predetermined degree of inconsistency between write operations and subsequent read operations on a set of data and subsequently receives a request to provide test data to an application. In response to the request to provide test data to the application the test system generates a set of test data including a plurality of entities retrieved from the set of data, based at least in part on the test policy. The test data includes a respective entity that is not consistent with a previous write operation. The test system further provides the set of test data to the application. The application optionally processes the set of test data to produce results, which are used to determine performance of the application.

Claims

1. A method, comprising:

at a respective computer system having one or more processors and memory storing one or more programs that when executed by the one or more processors cause the respective computer system to perform the method, the method comprising;

obtaining a test policy that specifies a predetermined degree of deliberate inconsistency between write operations and subsequent read operations on a set of data;

receiving a request to provide test data to an application; and

in response to the request to provide test data to the application, performing a sequence of operations, including;

accessing a transaction log that includes operations waiting to be performed on a database, wherein the database includes a plurality of entities;

selecting a first subset of operations from the transaction log to perform on the database in accordance with the test policy, wherein a second subset of operations from the transaction log is not included in the first subset of operations;

performing the first subset of operations on the database in accordance with the test policy while excluding the second subset of operations from performance on the database;

retrieving the test data from the database, wherein the test data includes a respective entity of the plurality of entities that is not consistent with a write operation in the second subset of operations; and

providing the test data to the application.

2. The method of claim 1, wherein:

the application processes the test data to produce results; and

the results are used to determine performance of the application when the application receives input data having the predetermined degree of deliberate inconsistency as specified by the test policy.

3. The method of claim 1, wherein the application is a test application and further wherein the test application includes a test version of a production application that is being tested.

4. The method of claim 1, wherein:

the second subset includes a respective operation for modifying the respective entity; and

in accordance with the test policy, the respective operation is deliberately not applied prior to retrieving the test data from the database.

5. The method of claim 1, wherein:

the test policy specifies a predefined percentage of operations that are not to be applied prior to retrieving the test data; and

a number of operations corresponding to the predefined percentage are randomly or pseudo-randomly excluded from the first subset in accordance with the test policy.

6. The method of claim 5, wherein:

the test policy specifies a random seed; and

the operations that are excluded from the first subset of operations are selected pseudo-randomly in accordance with output of a pseudorandom number generator that takes the random seed as an input.

9. The method of claim 1, wherein:

the test policy causes the test data to be generated so as to simulate data retrieval from a high replication datastore that does not guarantee strong consistency of data.

10. The method of claim 1, further comprising:

prior to retrieving the test data, receiving, from a user, a selection of a respective testing environment for testing the application, wherein the respective test environment is one of a plurality of testing environments that are supported by the respective computer system; and

in response to receiving the selection of the respective testing environment, selecting a type of test policy associated with the respective testing environment, wherein the type of test policy is one of a plurality of test policies that are supported by the respective computer system; and

receiving, from the user, a test policy parameter,

wherein obtaining the test policy includes generating the test policy in accordance with the test policy parameter and the type of test policy associated with the respective testing environment.

7. The method of 1, wherein the test policy specifies a probability of selecting a respective operation to be excluded from the first subset of operations based on an age of the respective operation.

8. The method of 1, wherein the test policy specifies a deterministic procedure for identifying operations to be excluded from the first subset of operations.

11. A computer system, comprising:

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

obtaining a test policy that specifies a predetermined degree of deliberate inconsistency between write operations and subsequent read operations on a set of data;

receiving a request to provide test data to an application; and

in response to the request to provide test data to the application, performing sequence of operations, including;

accessing a transaction log that includes operations waiting to be performed on a database, wherein the database includes a plurality of entities;

selecting a first subset of operations from the transaction log to perform on the database in accordance with the test policy, wherein a second subset of operations from the transaction log is not included in the first subset of operations;

performing the first subset of operations on the database while excluding the second subset of operations from performance on the database;

retrieving the test data from the database, wherein the test data includes a respective entity that is not consistent with a write operation in the second subset of operations; and

providing the test data to the application.

12. The system of claim 11, wherein:

the application processes the test data to produce results; and

the results are used to determine performance of the application when the application receives input data having the predetermined degree of deliberate inconsistency as specified by the test policy.

13. The system of claim 11, wherein the test policy is based on a predefined consistency parameter provided by a user testing performance of the application.

14. The system of claim 11, wherein:

the second subset includes a respective operation for modifying the respective entity; and

in accordance with the test policy, the respective operation is deliberately not applied prior to retrieving the test data from the database.

15. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a computer system with one or more processors, cause the computer system to:

obtain a test policy that specifies a predetermined degree of deliberate inconsistency between write operations and subsequent read operations on a set of data;

receive a request to provide test data to an application; and

in response to the request to provide test data to the application, perform a sequence of operations that;

access a transaction log that includes operations waiting to be performed on a database, wherein the database includes a plurality of entities;

select a first subset of operations from the transaction log to perform on the database in accordance with the test policy, wherein a second subset of operations from the transaction log is not included in the first subset of operations;

perform the first subset of operations on the database and exclude the second subset of operations from performance on the database;

retrieve the test data from the database, wherein the test data includes a respective entity that is not consistent with a write operation in the second subset of operations, and;

provide the test data to the application.

16. The non-transitory computer readable storage medium of claim 15, wherein:

the application processes the test data to produce results; and

the results are used to determine performance of the application when the application receives input data having the predetermined degree of deliberate inconsistency as specified by the test policy.

17. The non-transitory computer readable storage medium of claim 15, wherein the test policy is based on a predefined consistency parameter provided by a user testing performance of the application.

18. The non-transitory computer readable storage medium of claim 15, wherein:

the second subset includes a respective operation for modifying the respective entity; and

in accordance with the test policy, the respective operation is deliberately not applied prior to retrieving the test data from the database.