Predicting software product quality

• US 9,619,363 B1
• Filed: 09/25/2015
• Issued: 04/11/2017
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: Maplebear Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Chitale, Poonam P., Cox, Catherine M., D'Angelo, Dario, Jiang, Xiyao, Mohammadi-Rashedi, Shahin, Pavela, Thomas J., Rhodes, Jeffrey S., Sadowski, Marian E.
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): Wei, Zheng
• Application Number:
  US14/865,384
• Publication Number:
  US 20170091071A1
• Time in Patent Office: 564 Days
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/3616 using software metrics
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06Q 10/06375 Prediction of business process outcome or impact based on a proposed change

Abstract

Predicting software product quality. Real-time and historic software code metrics, software code defect data for the software product, and test case-related data for the software product are received. A feature predicted fallibility that estimates the number of code defects for a new feature of the software product, a product version projected fallibility that estimates the number of code defects for a new version of a software product, a test case related quality coefficient that estimates the likelihood of discovery of code defects in a new feature, a feature quality and a product quality indexes that are qualitative indications of quality of the new code of a feature and the new product version, are calculated. A report is then output that includes at least the calculated values, whereby developer resources are directed to features of the software product for which the calculated values indicate likelihoods of a high defect densities.

Claims

1. A computer implemented method for predicting software product quality, the method comprising:

receiving, by a computer over a network from an integrated development environment (IDE) server, real-time and historic software code metrics for a software product;

receiving, by the computer over the network from the IDE server, real-time and historic software code defect data for the software product;

receiving, by the computer over a network from the IDE server, real-time and historic test case-related data for the software product;

calculating, by the computer, a feature predicted fallibility that estimates the number of code defects for a new feature of the software product, based on the received real-time and historic software code metrics for a software product, and the received real-time and historic software code defect data for the software product, as determined by calculating the average, over a plurality of product versions for the feature, of the quotient of the new lines of code injected into a product version for the feature divided by number of customer reported problems for the feature requiring code changes, and determining the feature predicted fallibility by calculating the product of the average quotient and the estimated lines of code that will be injected into a new product version for the feature;

calculating, by the computer, a product version projected fallibility that estimates the number of code defects for a new version of a software product, based on an average of all calculated feature predicted fallibilities for all new features of the new version of the software product, as determined by calculating the sum of the feature projected fallibilities for all feature updates in the new product version that will be injecting lines of code;

calculating, by the computer, a test case related quality coefficient that estimates the likelihood of discovery of code defects in a new feature, based on a mathematical correlation between a test case related metric, and the received real-time and historic software code defect data for the software product, as determined by calculating a Pearson Correlation Coefficient, over a plurality of features in a plurality of product versions, between defects uncovered per test case and test cases per thousand lines of code;

calculating, by the computer, a feature quality index that is a qualitative indication of quality of the new code of a feature, based on the calculated feature predicted fallibility and the calculated test case related quality coefficient, as determined by calculating the product of the feature projected fallibility, the feature projected fallibility divided by the number of test cases, and the defects discovered per test case;

calculating, by the computer, a product quality index that is a qualitative indication of quality of the new version of the software product, based on an average of all calculated feature quality indexes for all new features of the new version of the software product, as determined by calculating the average of all the feature quality index values for a product version; and

outputting a report that includes at least the calculated feature predicted fallibility, product version projected fallibility, test case related quality coefficient, feature quality index, and product quality index, whereby developer resources are directed to features of the software product for which the calculated values indicate likelihoods of a high defect densities.

2. The method of claim 1, wherein software code defect data includes developer reported errors.

3. The method of claim 1, wherein test case-related data includes one or more of:

test case code coverage, number of test cases created, test case size, overall duration of the test case;

number of use cases per each test case;

test case size;

number of test case steps executed for a use case;

number of modules exercised by the test case;

number of modules exercised by the test case and that were changed;

central processing unit (CPU) consumption; and

input/output (I/O) operations used by the test case.

4. A computer program product for predicting software product quality, the computer program product comprising:

one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media, the program instructions comprising;

program instructions, executable by a computer, to receive, over a network from an integrated development environment (IDE) server, real-time and historic software code metrics for a software product;

program instructions, executable by the computer, to receive, over the network from the IDE server, real-time and historic software code defect data for the software product;

program instructions, executable by the computer, to receive, over the network from the IDE server, real-time and historic test case-related data for the software product;

program instructions, executable by the computer, to calculate a feature predicted fallibility that estimates the number of code defects for a new feature of the software product, based on the received real-time and historic software code metrics for a software product, and the received real-time and historic software code defect data for the software product, as determined by calculating the average, over a plurality of product versions for the feature, of the quotient of the new lines of code injected into a product version for the feature divided by number of customer reported problems for the feature requiring code changes, and determining the feature predicted fallibility by calculating the product of the average quotient and the estimated lines of code that will be injected into a new product version for the feature;

program instructions, executable by the computer, to calculate a product version projected fallibility that estimates the number of code defects for a new version of a software product, based on an average of all calculated feature predicted fallibilities for all new features of the new version of the software product, as determined by calculating the sum of the feature projected fallibilities for all feature updates in the new product version that will be injecting lines of code;

program instructions, executable by the computer, to calculate a test case related quality coefficient that estimates the likelihood of discovery of code defects in a new feature, based on a mathematical correlation between a test case related metric, and the received real-time and historic software code defect data for the software product, as determined by calculating a Pearson Correlation Coefficient, over a plurality of features in a plurality of product versions, between defects uncovered per test case and test cases per thousand lines of code;

program instructions, executable by the computer, to calculate a feature quality index that is a qualitative indication of quality of the new code of a feature, based on the calculated feature predicted fallibility and the calculated test case related quality coefficient, as determined by calculating the product of the feature projected fallibility, the feature projected fallibility divided by the number of test cases, and the defects discovered per test case;

program instructions, executable by the computer, to calculate a product quality index that is a qualitative indication of quality of the new version of the software product, based on an average of all calculated feature quality indexes for all new features of the new version of the software product, as determined by calculating the average of all the feature quality index values for a product version; and

program instructions, executable by the computer, to output a report that includes at least the calculated feature predicted fallibility, product version projected fallibility, test case related quality coefficient, feature quality index, and product quality index, whereby the calculated values indicate likelihoods of a high defect densities.

5. The computer program product of claim 4, wherein software code defect data includes developer reported errors.

6. The computer program product of claim 4, wherein test case-related data includes one or more of:

test case code coverage, number of test cases created, test case size, overall duration of the test case;

number of use cases per each test case;

test case size;

number of test case steps executed for a use case;

number of modules exercised by the test case;

number of modules exercised by the test case and that were changed;

central processing unit (CPU) consumption; and

input/output (I/O) operations used by the test case.

7. A computer system for predicting software product quality, the computer system comprising:

one or more computer processors, a network adapter connected to a network, one or more computer-readable storage media, and program instructions stored on one or more of the computer-readable storage media for execution by at least one of the one or more processors, the program instructions comprising;

program instructions, executable by a computer, to receive, over a network from an integrated development environment (IDE) server, real-time and historic software code metrics for a software product;

program instructions, executable by the computer, to receive, over the network from the IDE server, real-time and historic software code defect data for the software product;

program instructions, executable by the computer, to receive, over the network from the IDE server, real-time and historic test case-related data for the software product;

program instructions, executable by the computer, to calculate a feature predicted fallibility that estimates the number of code defects for a new feature of the software product, based on the received real-time and historic software code metrics for a software product, and the received real-time and historic software code defect data for the software product, as determined by calculating the average, over a plurality of product versions for the feature, of the quotient of the new lines of code injected into a product version for the feature divided by number of customer reported problems for the feature requiring code changes, and determining the feature predicted fallibility by calculating the product of the average quotient and the estimated lines of code that will be injected into a new product version for the feature;

program instructions, executable by the computer, to calculate a product version projected fallibility that estimates the number of code defects for a new version of a software product, based on an average of all calculated feature predicted fallibilities for all new features of the new version of the software product, as determined by calculating the sum of the feature projected fallibilities for all feature updates in the new product version that will be injecting lines of code;

program instructions, executable by the computer, to calculate a test case related quality coefficient that estimates the likelihood of discovery of code defects in a new feature, based on a mathematical correlation between a test case related metric, and the received real-time and historic software code defect data for the software product, as determined by calculating a Pearson Correlation Coefficient, over a plurality of features in a plurality of product versions, between defects uncovered per test case and test cases per thousand lines of code;

program instructions, executable by the computer, to calculate a feature quality index that is a qualitative indication of quality of the new code of a feature, based on the calculated feature predicted fallibility and the calculated test case related quality coefficient, as determined by calculating the product of the feature projected fallibility, the feature projected fallibility divided by the number of test cases, and the defects discovered per test case;

program instructions, executable by the computer, to calculate a product quality index that is a qualitative indication of quality of the new version of the software product, based on an average of all calculated feature quality indexes for all new features of the new version of the software product, as determined by calculating the average of all the feature quality index values for a product version; and

program instructions, executable by the computer, to output a report that includes at least the calculated feature predicted fallibility, product version projected fallibility, test case related quality coefficient, feature quality index, and product quality index, whereby the calculated values indicate likelihoods of a high defect densities.

8. The computer system of claim 7, wherein software code defect data includes developer reported errors.

9. The computer system of claim 7, wherein test case-related data includes one or more of:

test case code coverage, number of test cases created, test case size, overall duration of the test case;

number of use cases per each test case;

test case size;

number of test case steps executed for a use case;

number of modules exercised by the test case;

number of modules exercised by the test case and that were changed;

central processing unit (CPU) consumption; and

input/output (I/O) operations used by the test case.