Testing software enhancements in database applications

• US 9,811,444 B2
• Filed: 08/04/2015
• Issued: 11/07/2017
• Est. Priority Date: 08/04/2015
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Freschl, Jeffrey Lallana, Nair, Kiran Hariharan
• Primary Examiner(s): Cao, Diem
• Assistant Examiner(s): KRETZMER, ERIKA A
• Application Number:
  US14/817,550
• Publication Number:
  US 20170039128A1
• Time in Patent Office: 826 Days
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3668 Software testing software testing in telephone exchanges H04M3/242, testing of hardware G06F11/22
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: G06F 2201/80 Database-specific techniques

Abstract

A test system registers a series of tests that reference different software enhancements added to an application within a database system. The test system enables individual software enhancements referenced by the tests and then runs log lines through the application to produce test structured query language (SQL). The database system compares the test SQL with baseline SQL produced by the application without enabling the software enhancements. The database system executes the test SQL and captures performance metrics when the test SQL is different from the baseline SQL. Comparing SQL results avoids processing and capturing performance metrics for log lines not affected by the software enhancements. Incrementally running the log lines with one software enhancement enabled at a time also allows the test system to isolate the performance impact of individual software enhancements on the database system.

Claims

1. A method of testing enhancements for application code operating in a database system, the method comprising the steps of:

receiving, by the database system, from a replay and capture system, a series of tests referencing the enhancements, each of the tests referencing a feature and a distinct value;

receiving, by the database system, two or more log lines from the replay and capture system, each of the log lines describing a user request captured during a prior network session between a user system and the database system;

running, by the database system, a first log line through the application code without the enhancements to produce baseline structured query language (SQL); and

incrementally testing the application code by, for each of the series of tests;

running, by the database system, the first log line with the enhancement enabled in the application code with the feature and value referenced by the test to produce test SQL;

comparing, by the database system, the test SQL with the baseline SQL;

when the comparison indicates a no change between the baseline SQL and the test SQL, not executing the test SQL; and

when the comparison indicates a change between the baseline SQL and the test SQL, executing, by the database system, the test SQL and capturing performance metrics for the executed test SQL; and

running a next one of the series of tests with the enhancement enabled with a corresponding feature and value of the next test; and

repeating the foregoing testing steps until the series of tests is completed.

2. The method of claim 1, wherein the step of comparing the test SQL with the baseline SQL includes:

generating, by the database system, a first hash code from the baseline SQL;

generating, by the database system, second hash code for the test SQL; and

comparing the first hash code with the second hash code to detect a change between the baseline SQL and the test SQL.

3. The method of claim 1, further comprising:

running, by the database system, the same tests with at least one more of the received log lines; and

combining, by the database system, the performance metrics for the same tests for the received log lines that were used in running the same tests.

4. The method of claim 1, further comprising:

creating, by the database system, the enhancements in a SQL code optimizer for the application code; and

running, by the database system, the tests through the SQL code optimizer.

5. The method of claim 1, further comprising:

monitoring, by the database system, baseline times for processing the baseline SQL; and

monitoring, by the database system, test times for processing the test SQL.

6. The method of claim 1, further comprising:

monitoring, by the database system, a number of baseline input/output (I/O) operations performed for the baseline SQL; and

monitoring, by the database system, a number of test I/O operations performed for the test SQL.

7. The method of claim 1, further comprising:

monitoring, by the database system, row counts produced by the baseline SQL; and

monitoring, by the database system, row counts produced by the test SQL.

8. The method of claim 1, wherein the application code is one of a software as a service (SAAS) application, a customer relationship management (CRM) application, or a database application.

9. The method of claim 1, wherein when the comparison indicates that the test SQL is the same as the baseline SQL, additionally sending a same-results message back to the replay and capture system.

10. A computing device for testing code features in an application operated in a database system, comprising:

a processor;

storage memory; and

one or more stored sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of;

storing a series of tests referencing the code features in the application, each of the tests referencing a feature and a distinct value;

storing a set of log lines, each of the log lines describing a user request captured during a prior network session between a user system and the database system;

running, by the database system, the log lines without the features enabled in the application to produce baseline SQL;

submitting the tests and the log lines to the database system for incrementally testing the code features referenced by the tests, by, for each of the series of tests;

running, by the database system, one log line of the log lines with the enhancement enabled in the application with the feature and value referenced by the test to produce test SQL;

comparing, by the database system, the test SQL with the respective baseline SQL generated for the one log line;

when the comparison indicates a no change between the baseline SQL and the test SQL, not executing the test SQL; and

when the comparison indicates a change between the baseline SQL and the test SQL, executing, by the database system, the test SQL and capturing performance metrics for the executed test SQL; and

running a next one of the series of tests, with the enhancement enabled with a corresponding feature and value of the next test; and

repeating the foregoing testing steps until the series of tests is completed;

receiving performance metrics from the database system associated with the different code features; and

displaying the performance metrics for the different code features on a user interface.

11. The computing device of claim 10, wherein the instructions further cause the processor to carry out the steps of:

receiving a first set of the performance metrics from the database system for the log lines run through the application without using the code features;

receiving a second set of the performance metrics from the database system for the log lines run with individual code features enabled in the application; and

comparing the first set of performance metrics with the second set of performance metrics.

12. The computing device of claim 10, wherein the instructions further cause the processor to carry out the steps of:

submitting a first set of the tests to an optimizer in the database system causing the optimizer to generate baseline structured query language (SQL) without using the code features; and

submitting a second set of the tests to the optimizer causing the optimizer to use the code features to generate test SQL.

13. The computing device of claim 12, wherein the database system does not generate the performance metrics for the test SQL that matches the baseline SQL.

14. The computing device of claim 12, wherein the instructions further cause the processor to display the baseline SQL and highlight portions of the test SQL that are different from the baseline SQL.

15. The computing device of claim 10, wherein the instructions further cause the processor to carry out the steps of identifying from the performance metrics a first number of the log lines run through each of the code features with improved database processing times and a second number of the log lines with regressed database processing times.

16. The computing device of claim 10, wherein the instructions further cause the processor to carry out the steps of identifying from the performance metrics a number of database gets for the log lines processed by each of the code features.

17. The computing device of claim 16, wherein the instructions further cause the processor to carry out the steps of comparing the number of database gets for the log lines using the code features with the number of database gets for the log lines not using the code features.

18. The computing device of claim 10, wherein the instructions further cause the processor to carry out the steps of comparing row counts produced by the database system using the code features and row counts produced by the database system without using the code features.

19. A non-transitory computer-readable medium that stores a computer program for a database system, the computer program comprising a set of instructions operable, when executed by a computer processor, to:

receive a series of tests referencing software features added to an application in the database system, each of the tests referencing a feature and a distinct value;

receive a series of log lines, each of the log lines describing a user request captured during a prior network session between a user system and the database system;

run the log lines through the application without enabling the software features to produce baseline structured query language (SQL);

incrementally testing the application by, for each of the series of tests;

run the log lines with individual software features enabled with the feature and value referenced by the tests to produce test SQL;

compare the test SQL with the baseline SQL;

when the comparison indicates a no change between the baseline SQL and the test SQL, not executing the test SQL;

when the comparison indicates a change between the baseline SQL and the test SQL,

execute the test SQL with the database system and capture performance metrics for the executed test SQL;

running a next one of the series of tests, with the enhancement enabled with a corresponding feature and value of the next test; and

repeating the foregoing testing steps until the series of tests is completed.

20. The computer program of claim 19, wherein instructions to compare the test SQL with the baseline SQL includes instructions to:

generate a first hash code from the baseline SQL;

generate second hash codes from the test SQL; and

compare the first hash code with the second hash code to detect a change between the baseline SQL and the test.

21. The computer program of claim 19, including instructions operable to:

monitor differences between the performance metrics for the same log lines run with different software features enabled with the feature and value referenced by different tests;

run different log lines with the same software features enabled; and

monitor differences between the performance metrics for the different log lines run with the same software features enabled.