Duplicate bug report detection using machine learning algorithms and automated feedback incorporation

• US 10,379,999 B2
• Filed: 01/11/2016
• Issued: 08/13/2019
• Est. Priority Date: 01/11/2016
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Bagal, Prasad V., Joshi, Sameer Arun, Chien, Hanlin Daniel, Diez, Ricardo Rey, Woo, David Cavazos, Su, Emily Ronshien, Chang, Sha
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): Smith, Cheneca
• Application Number:
  US14/992,831
• Publication Number:
  US 20170199803A1
• Time in Patent Office: 1,310 Days
• Field of Search: None
• CPC Class Codes: G06F 11/362 Software debugging
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Duplicate bug report detection using machine learning algorithms and automated feedback incorporation is disclosed. For each set of bug reports, a user-classification of the set of bug reports as including duplicate bug reports or non-duplicate bug reports is identified. Also for each set of bug reports, correlation values corresponding to a respective feature, of a plurality of features, between bug reports in the set of bug reports is identified. Based on the user-classifications and the correlation values, a model is generated to identify any set of bug reports as including duplicate bug reports or non-duplicate bug reports. The model is applied to classify a particular bug report and a candidate bug report as duplicate bug reports or non-duplicate bug reports.

Claims

1. A method comprising:

for each particular set of bug reports, in a first plurality of sets of bug reports, identifying;

(a) a user-classification of the particular set of bug reports as including duplicate bug reports or non-duplicate bug reports;

(b) a first plurality of correlation values, each of which corresponds to a respective feature, of a plurality of features, between bug reports in the particular set of bug reports;

based on (a) and (b), for the first plurality of sets of bug reports, generating a model to identify any set of bug reports as including duplicate bug reports or non-duplicate bug reports;

receiving a request to determine whether a particular bug report is a duplicate of any of a second plurality of bug reports;

identifying a first category associated with the particular bug report;

identifying a first subset of bug reports, of the second plurality of bug reports, associated with the first category;

identifying a second subset of bug reports, of the second plurality of bug reports, that have been previously identified as a duplicate of at least one bug report of the first subset of bug reports;

identifying a set of candidate bug reports that;

(a) includes one or more of the first subset of bug reports;

(b) includes one or more of the second subset of bug reports; and

(c) does not include a third subset of bug reports, of the second plurality of bug reports, that (i) are not associated with the first category and (ii) have not been previously identified as a duplicate of any bug report of the first subset of bug reports;

applying the model to obtain a classification of the particular bug report and a candidate bug report, of the set of candidate bug reports, as duplicate bug reports or non-duplicate bug reports, and refraining from applying the model to classify the particular bug report and any of the third subset of bug reports as duplicate bug reports or non-duplicate bug reports.

2. The method of claim 1, wherein identifying the first plurality of correlation values between bug reports in the particular set of bug reports comprises:

determining a correlation value for a particular feature, in the plurality of features, at least by comparing (a) a first set of one or more values for the particular feature in a first bug report of the particular set of bug reports with (b) a second set of one or more values for the particular feature in a second bug report of the particular set of bug reports.

3. A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, causes performance of operations comprising:

for each particular set of bug reports, in a first plurality of sets of bug reports, identifying;

(a) a user-classification of the particular set of bug reports as including duplicate bug reports or non-duplicate bug reports;

(b) a first plurality of correlation values, each of which corresponds to a respective feature, of a plurality of features, between bug reports in the particular set of bug reports;

based on (a) and (b), for the first plurality of sets of bug reports, generating a model to identify any set of bug reports as including duplicate bug reports or non-duplicate bug reports;

receiving a request to determine whether a particular bug report is a duplicate of any of a second plurality of bug reports;

identifying a first category associated with the particular bug report;

identifying a first subset of bug reports, of the second plurality of bug reports, associated with the first category;

identifying a second subset of bug reports, of the second plurality of bug reports, that have been previously identified as a duplicate of at least one bug report of the first subset of bug reports;

identifying a set of candidate bug reports that;

(a) includes one or more of the first subset of bug reports;

(b) includes one or more of the second subset of bug reports; and

(c) does not include a third subset of bug reports, of the second plurality of bug reports, that (i) are not associated with the first category and (ii) have not been previously identified as a duplicate of any bug report of the first subset of bug reports;

applying the model to obtain a classification of the particular bug report and a candidate bug report, of the set of candidate bug reports, as duplicate bug reports or non-duplicate bug reports, and refraining from applying the model to classify the particular bug report and any of the third subset of bug reports as duplicate bug reports or non-duplicate bug reports.

4. The medium of claim 3, wherein identifying the user-classification of the particular set of bug reports as including duplicate bug reports or including non-duplicate bug reports comprises:

presenting the particular set of bug reports to a user; and

receiving user input indicating whether the particular set of bug reports include duplicate bug reports or non-duplicate bug reports.

5. The medium of claim 3, wherein identifying the first plurality of correlation values between bug reports in the particular set of bug reports comprises:

determining a correlation value for a particular feature, in the plurality of features, at least by comparing (a) a first set of one or more values for the particular feature in a first bug report of the particular set of bug reports with (b) a second set of one or more values for the particular feature in a second bug report of the particular set of bug reports.

6. The medium of claim 3, wherein the operations further comprise:

selecting the candidate bug report, for comparison to the particular bug report, in response to determining that both the candidate bug report and the particular bug report are associated with a same category.

7. The medium of claim 3, wherein the operations further comprise:

selecting the candidate bug report, for comparison to the particular bug report, in response to determining that (a) a previously identified set of duplicate bug reports included a first bug report and a second bug report, (b) the first bug report and the particular bug report are both associated with a same first category, and (c) the second bug report and the candidate bug report are both associated with a same second category.

8. The medium of claim 3, wherein the operations further comprise:

identifying a set of categories associated with each of the second subset of bug reports;

determining a benefit-to-cost ratio for inclusion of bug reports in a particular category, of the set of categories, into the set of candidate bug reports;

including bug reports in the particular category into the set of candidate bug reports based on the benefit-to-cost ratio, wherein the candidate bug report is one of the bug reports in the particular category.

9. The medium of claim 3, wherein the model comprises a plurality of weight assignments, each of which corresponds to a respective feature of the plurality of features.

10. The medium of claim 3, wherein applying the model comprises:

obtaining a second plurality of correlation values corresponding to the plurality of features by;

for each particular feature of the plurality of features, determining a particular correlation value corresponding to correlation between (a) a first set of one or more values for the particular feature in the particular bug report and (b) a second set of values for the particular feature in the candidate bug report;

obtaining a plurality of weighted correlation values from the second plurality of correlation values by;

for each particular correlation value of the second plurality of correlation values;

applying a weight, assigned to a feature corresponding to the particular correlation value, to the particular correlation value.

11. The medium of claim 3, wherein applying the model comprises:

obtaining a second plurality of correlation values corresponding to the plurality of features by;

for each particular feature of the plurality of features, determining a first correlation value corresponding to correlation between (a) a first set of one or more values for the particular feature in the particular bug report and (b) a second set of values for the particular feature in the candidate bug report;

determining a second correlation value corresponding to correlation between (a) the first set of one or more values for the particular feature in the particular bug report and (b) a third set of values for the particular feature in another bug report that has been identified as a duplicate bug report of the candidate bug report;

selecting one of the first correlation value and the second correlation value for the particular feature;

obtaining a plurality of weighted correlation values from the second plurality of correlation values by;

for each particular correlation value of the second plurality of correlation values;

applying a weight, assigned to a feature corresponding to the particular correlation value, to the particular correlation value.

12. The medium of claim 3, wherein the operations further comprise:

receiving user input comprising a correction that indicates that the classification of the particular bug report and the candidate bug report is incorrect;

modifying the model based on the correction to obtain a modified model, comprising;

modifying weights assigned to each feature of the plurality of features such that the particular bug report and the candidate bug report would be classified correctly as duplicate bug reports or non-duplicate bug reports.

13. The medium of claim 3, wherein the operations further comprise:

receiving user input comprising a correction that indicates that the classification of the particular bug report and the candidate bug report is incorrect;

modifying the model based on the correction to obtain a modified model, comprising;

modifying weights assigned to each feature of the plurality of features to generate a potential new model;

determining that the potential new model is an improvement over the model; and

using the potential new model as the modified model.

14. The medium of claim 13, wherein determining that the potential new model is an improvement over the model is based on at least one of:

(a) a true positive rate of system-classifications produced by the potential new model, and (b) a true negative rate of system-classifications produced by the potential new model.

15. The medium of claim 3, wherein the operations further comprise:

receiving user input comprising a particular user-classification of the particular bug report and the candidate bug report as duplicate bug reports or non-duplicate bug reports, wherein the particular user-classification indicates that the classification of the particular bug report and the candidate bug report obtained by the model is incorrect;

generating a new training set of bug reports, the new training set of bug reports comprising;

the particular bug report and the candidate bug report;

the particular user-classification of the particular bug report and the candidate bug report; and

modifying the model based on the new training set of bug reports to obtain a modified model.

16. The medium of claim 3, wherein applying the model comprises classifying the particular bug report and the candidate bug report as non-duplicate bug reports, and wherein the operations further comprise:

receiving user input comprising a correction that indicates that the particular bug report and the candidate bug report are duplicate bug reports; and

modifying the model based on the correction to obtain a modified model.

17. The medium of claim 3, wherein the operations further comprise:

identifying a fourth subset of bug reports, of the second plurality of bug reports, that are associated with a common category as at least one of the second subset of bug reports, wherein at least one of the fourth subset of bug reports is not associated with a common category as the particular bug report and at least one of the fourth subset of bug reports has not been previously identified as a duplicate of at least one of the first subset of bug reports;

including the fourth subset of bug reports into the set of candidate bug reports.

18. The medium of claim 3, wherein the operations further comprise:

identifying a second category, different from the first category, associated with the particular bug report;

identifying a fourth subset of bug reports, of the second plurality of bug reports, associated with the second category;

identifying a fifth subset of bug reports, of the second plurality of bug reports, that have been previously identified as a duplicate of at least one of the fourth subset of bug reports, wherein at least one of the fifth subset of bug reports is not associated with a common category as the particular bug report;

including one or more of the fourth subset of bug reports into the set of candidate bug reports;

including one or more of the fifth subset of bug reports into the set of candidate bug reports.

19. The medium of claim 3, wherein the operations further comprise:

receiving user input comprising a correction that indicates that the classification of the particular bug report and the candidate bug report is incorrect;

modifying the model based on the correction to obtain a modified model, comprising;

adding a new feature to the plurality of features such that applying the modified model, with the new feature, to the particular bug report and the candidate report would result in classifying the particular bug report and the candidate bug report correctly as duplicate bug reports or non-duplicate bug report.

20. A system comprising:

at least one hardware device including a processor; and

the system configured to perform operations comprising;

for each particular set of bug reports, in a first plurality of sets of bug reports, identifying;

(a) a user-classification of the particular set of bug reports as including duplicate bug reports or non-duplicate bug reports;

(b) a first plurality of correlation values, each of which corresponds to a respective feature, of a plurality of features, between bug reports in the particular set of bug reports;

based on (a) and (b), for the first plurality of sets of bug reports, generating a model to identify any set of bug reports as including duplicate bug reports or non-duplicate bug reports;

receiving a request to determine whether a particular bug report is a duplicate of any of a second plurality of bug reports;

identifying a first category associated with the particular bug report;

identifying a first subset of bug reports, of the second plurality of bug reports, associated with the first category;

identifying a second subset of bug reports, of the second plurality of bug reports, that have been previously identified as a duplicate of at least one bug report of the first subset of bug reports;

identifying a set of candidate bug reports that;

(a) includes one or more of the first subset of bug reports;

(b) includes one or more of the second subset of bug reports; and

(c) does not include a third subset of bug reports, of the second plurality of bug reports, that (i) are not associated with the first category and (ii) have not been previously identified as a duplicate any bug report of the first subset of bug reports;

applying the model to obtain a classification of the particular bug report and a candidate bug report, of the set of candidate bug reports, as duplicate bug reports or nonduplicate bug reports, and refraining from applying the model to classify the particular bug report and any of the third subset of bug reports as duplicate bug reports or non-duplicate bug reports.