FORECASTING A QUALITY OF A SOFTWARE RELEASE USING MACHINE LEARNING

• US 20200133830A1
• Filed: 10/31/2018
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Dell Products BV (Dell Technologies Inc.)
• Original Assignee: Dell Products LP (Dell Technologies Inc.)
• Inventors: Sharma, Rohan, Bikumala, Sathish Kumar, Philip, Sibi, Gadamshetty, Kiran Kumar
• Granted Patent: US 11,347,629 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06Q 10/06312 Adjustment or analysis of established resource schedule, e.g. resource or task levelling, or dynamic rescheduling

Abstract

In some examples, a server may retrieve and parse test results associated with testing a software package. The server may determine a weighted sum of a software feature index associated with a quality of the plurality of features, a defect index associated with the defects identified by the test cases, a test coverage index indicating a pass rate of the plurality of test cases, a release release reliability index associated with results of executing regression test cases included in the test cases, and an operational quality index associated with resources and an environment associated with the software package. The server may use a machine learning algorithm, such as a time series forecasting algorithm, to forecast a release status of the software package. The server may determine, based on the release status, whether the software package is to progress from a current phase to a next phase of a development cycle.

Claims

1. A method comprising:

retrieving, by one or more processors, test results associated with a software package comprising a plurality of features, the test results created by executing a plurality of test cases to test the plurality of features;

parsing, by the one or more processors, the test results to create parsed data;

determining, by the one or more processors and based on the parsed data, a software feature index associated with a quality of the plurality of features at a particular point in time of a development cycle;

determining, by the one or more processors and based on the parsed data, a defect index associated with the defects identified by the test cases;

determining, by the one or more processors and based on the parsed data, a test coverage index indicating a pass rate of the plurality of test cases;

determining, by the one or more processors and based on the parsed data, a release reliability index associated with results of executing regression test cases included in the test cases;

determining, by the one or more processors and based on the parsed data, an operational quality index associated with resources and an environment associated with the software package;

determining, by the one or more processors, a release index based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index;

repeatedly determining a release index at different points in time of a development cycle to create a time series of release indexes, the release index determined based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index;

based at least in part on determining that the time series of the release indexes is stationary, selecting a machine learning algorithm to forecast a release status of the software package; and

forecasting, based on the machine learning algorithm, a release status of the software package.

2. The method of claim 1, further comprising:

determining, in a next phase of the development cycle, a second release index based on a second weighted sum of a second software feature index, a second defect index, a second test coverage index, a second release reliability index, and a second operational quality index;

forecasting a second release status of the software package based on the second release index, the release index, and the at least one previously determined release index;

determining that the second release status fails to satisfy a second predetermined threshold; and

modifying one or more features of the software package.

3. The method of claim 1, wherein determining the release index comprises determining a sum of:

multiplying a first weight with the software feature index,

multiplying a second weight with the defect index,

multiplying a third weight with the test coverage index,

multiplying a fourth weight with the release reliability index, and

multiplying a fifth weight with the operational quality index.

4. The method of claim 1, wherein the machine learning algorithm comprises auto regressive integrated moving average (ARIMA) time series forecasting.

5. The method of claim 1, wherein the development cycle comprises a development independent testing phase, a system integration phase, a user acceptance testing phase, and a production phase.

6. The method of claim 1, wherein the software feature index is determined based at least in part on:

an implementation accuracy associated with the plurality of features; and

a number of unimplemented features in the software package.

7. The method of claim 1, wherein the defect index is determined based at least in part on:

a number of current defects;

a severity of the current defects;

a priority of the current defects; and

a current phase in the development cycle in which the current defects were determined.

8. A computing device comprising:

one or more processors; and

one or more non-transitory computer readable media storing instructions executable by the one or more processors to perform operations comprising;

retrieving test results associated with a software package comprising a plurality of features, the test results created by executing a plurality of test cases to test the plurality of features;

parsing the test results to create parsed data;

determining, based on the parsed data, a software feature index associated with a quality of the plurality of features;

determining, based on the parsed data, a defect index associated with the defects identified by the test cases;

determining, based on the parsed data, a test coverage index indicating a pass rate of the plurality of test cases;

determining, based on the parsed data, a release reliability index associated with results of executing regression test cases included in the test cases;

determining, based on the parsed data, an operational quality index associated with resources and an environment associated with the software package;

determining a release index based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index;

repeatedly determining a release index at different points in time of a development cycle to create a time series of release indexes, the release index determined based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index;

based at least in part on determining that the time series of the release indexes is stationary, selecting a machine learning algorithm to forecast a release status of the software package; and

forecasting, based on the machine learning algorithm, a release status of the software package.

9. The computing device of claim 8, further comprising:

determining, in a next phase of the development cycle, a second release index based on a second weighted sum of a second software feature index, a second defect index, a second test coverage index, a second release reliability index, and a second operational quality index;

forecasting a second release status of the software package based on the second release index, the release index, and the at least one previously determined release index;

determining that the second release status fails to satisfy a second predetermined threshold; and

modifying one or more features of the software package.

10. The computing device of claim 8, wherein determining the release index comprises determining a sum of:

multiplying a first weight with the software feature index,

multiplying a second weight with the defect index,

multiplying a third weight with the test coverage index,

multiplying a fourth weight with the release reliability index, and

multiplying a fifth weight with the operational quality index.

11. The computing device of claim 8, wherein the machine learning algorithm comprises auto regressive integrated moving average (ARIMA) time series forecasting.

12. The computing device of claim 8, wherein the software feature index is determined based at least in part on:

an implementation accuracy associated with the plurality of features; and

a number of unimplemented features in the software package.

13. The computing device of claim 8, wherein the defect index is determined based at least in part on:

a number of current defects;

a severity of the current defects;

a priority of the current defects; and

a current phase in the development cycle in which the current defects were determined.

14. One or more non-transitory computer readable media storing instructions executable by one or more processors to perform operations comprising:

retrieving test results associated with a software package comprising a plurality of features, the test results created by executing a plurality of test cases to test the plurality of features;

parsing the test results to create parsed data;

determining, based on the parsed data, a software feature index associated with a quality of the plurality of features;

determining, based on the parsed data, a defect index associated with the defects identified by the test cases;

determining, based on the parsed data, a test coverage index indicating a pass rate of the plurality of test cases;

determining, based on the parsed data, a release reliability index associated with results of executing regression test cases included in the test cases;

determining, based on the parsed data, an operational quality index associated with resources and an environment associated with the software package;

determining, a release index based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index;

repeatedly determining a release index at different points in time of a development cycle to create a time series of release indexes, the release index determined based at least in part on the software feature index, the defect index, the test coverage index, the release reliability index, and the operational quality index;

based at least in part on determining that the time series of the release indexes is stationary, selecting a machine learning algorithm to forecast a release status of the software package; and

forecasting, based on the machine learning algorithm, a release status of the software package.

15. The one or more non-transitory computer readable media of claim 14, wherein the operations further comprise:

determining, in a next phase of the development cycle, a second release index based on a second weighted sum of a second software feature index, a second defect index, a second test coverage index, a second release reliability index, and a second operational quality index;

forecasting a second release status of the software package based on the second release index, the release index, and the at least one previously determined release index;

determining that the second release status fails to satisfy a second predetermined threshold; and

modifying one or more features of the software package.

16. The one or more non-transitory computer readable media of claim 14, wherein determining the release index comprises determining a sum of:

multiplying a first weight with the software feature index,

multiplying a second weight with the defect index,

multiplying a third weight with the test coverage index,

multiplying a fourth weight with the release reliability index, and

multiplying a fifth weight with the operational quality index.

17. The one or more non-transitory computer readable media of claim 14, wherein the machine learning algorithm comprises auto regressive integrated moving average (ARIMA) time series forecasting.

18. The one or more non-transitory computer readable media of claim 14, wherein the software feature index is determined based at least in part on:

an implementation accuracy associated with the plurality of features; and

a number of unimplemented features in the software package.

19. The one or more non-transitory computer readable media of claim 14, wherein the operations further comprise plotting the time series of the release indexes over time.

20. The one or more non-transitory computer readable media of claim 14, wherein the defect index is determined based at least in part on:

a number of current defects;

a severity of the current defects;

a priority of the current defects; and

a current phase in the development cycle in which the current defects were determined.