Probationary software tests

• US 9,588,875 B2
• Filed: 11/21/2013
• Issued: 03/07/2017
• Est. Priority Date: 03/14/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chan, Philip S. P., Hey, Laurence A., Izard, William J., Ponsford, Matthew
• Primary Examiner(s): Dao, Thuy
• Application Number:
  US14/086,198
• Publication Number:
  US 20140282410A1
• Time in Patent Office: 1,202 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

A method, computer program product, and system is described. A continuous integration environment is identified. A first software test associated with the continuous integration environment is identified. A probationary status for the first software test is determined, the probationary status indicating, at least in part, a potential lack of reliability for the first software test.

Claims

1. A computer-implemented method comprising:

identifying, by one or more computing devices, a continuous integration environment;

identifying, by the one or more computing devices, a first software test associated with the continuous integration environment;

identifying a second software test associated with the continuous integration environment, the second software test having a promoted status;

determining, by the one or more computing devices, a probationary status for the first software test based upon, at least in part, a recency with which the first software test was associated with the continuous integration environment relative to a recency with which another software test was associated with the continuous integration environment, the probationary status indicating, at least in part, a potential lack of reliability for the first software test;

presenting, in a first format, first test result information associated with running the first software test and presenting, in a second format, second test result information associated with the second software test, wherein the first format is distinct from the second format based upon, at least in part, the probationary status of the first software test and the promoted status of the second software test, and wherein one or more of the first test result information and the second test result information include an aggregate pass/fail rate; and

determining a promoted status for the first software test based upon, at least in part, identifying a code consistency over a sequence of a plurality of successful runs for the first software test over a plurality of code builds, the code consistency including a lack of changes to the first software test over the sequence of runs for the first software test, the promoted status indicating a greater degree of reliability for the first software test.

2. The computer-implemented method of claim 1 further comprising:

determining the promoted status for the first software test based upon, at least in part, identifying a sequence of successful runs for the first software test.

3. The computer-implemented method of claim 1 wherein identifying the probationary status is based upon, at least in part:

storing, in a test result database, first test result information associated with running the first software test; and

comparing the first test result information with other test result information in the test result database in order to, at least in part, identify an indicator of the potential lack of reliability for the first software test.

4. The computer-implemented method of claim 1 wherein identifying the probationary status is based upon, at least in part:

identifying the first software test as a previously-promoted software test; and

determining an intermittent reliability of the first software test.

5. The computer-implemented method of claim 1, wherein the recency with which the first software test was associated with the continuous integration environment includes a recency with which the first software test was added to the continuous integration environment.

6. The computer-implemented method of claim 1, wherein the recency with which the first software test was associated with the continuous integration environment includes a recency with which the first software test was updated.

7. The computer-implemented method of claim 1, further comprising:

determining intermittent reliability of the first software test having the promoted status based upon, at least in part, an intermittent pattern of pass/fail results with respect to a particular codebase; and

providing an alert based upon, at least in part, the determined intermittent reliability of the first software test.