Identifying bugs in a database system environment

• US 8,583,964 B2
• Filed: 12/23/2010
• Issued: 11/12/2013
• Est. Priority Date: 05/28/2010
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Chen, Zhiqiang, Dubey, Anjesh, Ortiz, Worley, Fischer, Mark A., Kim, Thom, Tsui, Andrew, Hossain, Fiaz, Kothari, Pallav
• Primary Examiner(s): DUNCAN, MARC M
• Application Number:
  US12/977,597
• Publication Number:
  US 20120005537A1
• Time in Patent Office: 1,055 Days
• Field of Search: 714/25, 714/37, 714/38.1, 714/41, 717/124
• US Class Current: 714/38.1
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

A system and method for identifying bugs in a database system. In one embodiment, a method includes running a plurality of tests on a software application, and rerunning one or more tests of the plurality of tests. The method also includes identifying one or more bugs in the one or more tests based on inconsistent test results.

Claims

1. A method for identifying bugs in a database system, the method comprising:

running a plurality of tests on a software application to generate first test results for each of the plurality of tests;

rerunning one or more tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests; and

for any particular ones of the one or more tests that are determined to have subsequent test results that are inconsistent with the first test results, identifying those particular ones of the one or more tests as having bugs that cause those particular ones of the one or more tests to generate inconsistent test results.

2. The method of claim 1, wherein the rerunning comprises:

rerunning only failing tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests.

3. The method of claim 1, wherein the rerunning comprises:

rerunning all of the tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests.

4. The method of claim 1, wherein the rerunning is performed immediately.

5. The method of claim 1, wherein the rerunning is performed at a predefined time period.

6. The method of claim 1, wherein the running of the plurality of tests is performed on each of a plurality of versions of the software application,

wherein the rerunning of the one or more tests is performed for one or more new failed tests that previously passed,

wherein the rerunning of the tests is performed on a first version of the software application where the one or more new failed tests failed, and

wherein the rerunning of the one or more tests is performed on a most recent version of the software application where the one or more new failed tests previously passed.

7. The method of claim 1, wherein the rerunning is performed in a different test environment.

8. A non-transitory computer-readable storage medium carrying one or more sequences of instructions thereon for identifying bugs in a database system, the instructions, when executed by a processor, cause the processor to perform:

running a plurality of tests on a software application to generate first test results for each of the plurality of tests;

rerunning one or more tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests; and

for any particular ones of the one or more tests that are determined to have subsequent test results that are inconsistent with the first test results, identifying those particular ones of the one or more tests as having bugs that cause those particular ones of the one or more tests to generate inconsistent test results.

9. The non-transitory computer-readable storage medium of claim 8, wherein the rerunning comprises:

rerunning only failing tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests.

10. The non-transitory computer-readable storage medium of claim 8, wherein the rerunning comprises:

rerunning all of the tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests.

11. The non-transitory computer-readable storage medium of claim 8, wherein the rerunning is performed immediately.

12. The non-transitory computer-readable storage medium of claim 8, wherein the rerunning is performed at a predefined time period.

13. The non-transitory computer-readable storage medium of claim 8, wherein the running of the plurality of tests is performed on each of a plurality of versions of the software application,

wherein the rerunning of the one or more tests is performed for one or more new failed tests that previously passed,

wherein the rerunning of the tests is performed on a first version of the software application where the one or more new failed tests failed, and

wherein the rerunning of the one or more tests is performed on a most recent version of the software application where the one or more new failed tests previously passed.

14. The non-transitory computer-readable storage medium of claim 8, wherein the rerunning is performed in a different test environment.

15. An apparatus for identifying bugs in a database system, the apparatus comprising:

a processor; and

a storage device storing one or more stored sequences of instructions which when executed by the processor cause the processor to;

run a plurality of tests on a software application to generate first test results for each of the plurality of tests;

rerun one or more tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests; and

for any particular ones of the one or more tests that are determined to have subsequent test results that are inconsistent with the first test results, identify those particular ones of the one or more tests as having bugs that cause those particular ones of the one or more tests to generate inconsistent test results.

16. The apparatus of claim 15, wherein the stored sequences of instructions which when executed by the processor cause the processor to rerun only failing tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests.

17. The apparatus of claim 15, wherein the stored sequences of instructions which when executed by the processor cause the processor to rerun all of the tests of the plurality of tests on the software application to generate subsequent test results for each of the one or more tests.

18. The apparatus of claim 15, wherein the stored sequences of instructions which when executed by the processor cause the processor to rerun one or more tests of the plurality of tests immediately.

19. The apparatus of claim 15, wherein the stored sequences of instructions which when executed by the processor cause the processor to rerun one or more tests of the plurality of tests at a predefined time period.

20. The apparatus of claim 15, wherein the running of the plurality of tests is performed on each of a plurality of versions of the software application,

wherein the rerunning of the one or more tests is performed for one or more new failed tests that previously passed,

wherein the rerunning of the tests is performed on a first version of the software application where the one or more new failed tests failed, and

wherein the rerunning of the one or more tests is performed on a most recent version of the software application where the one or more new failed tests previously passed.

21. A method for identifying bugs in a database system, the method comprising:

running a plurality of tests on a software application to generate first test results for each of the plurality of tests;

rerunning one or more tests of the plurality of tests on the software application without having changed the software application to generate subsequent test results for each of the one or more tests; and

for any particular ones of the one or more tests that are determined to have subsequent test results that are inconsistent with the first test results, identifying those particular ones of the one or more tests as having bugs that cause those particular ones of the one or more tests to generate inconsistent test results.

22. The method of claim 21, further comprising:

comparing the subsequent test results to corresponding ones of the first test results.