Revising a test suite using diagnostic efficacy evaluation

• US 20030139906A1
• Filed: 01/18/2002
• Published: 07/24/2003
• Est. Priority Date: 01/18/2002
• Status: Active Grant

• Current Assignee: Agilent Technologies, Inc.
• Original Assignee: Agilent Technologies, Inc.
• Inventors: Barford, Lee A.
• Granted Patent: US 7,139,676 B2
• Time in Patent Office: Days
• US Class Current: 702/183
• CPC Class Codes: G01R 31/318314 Tools, e.g. program interfaces, test suite, test bench, simulation hardware, test compiler, test program languages simulation software G01R31/318357; emulators G06F11/261
• CPC Class Codes: G01R 31/31835 Analysis of test coverage or failure detectability
• CPC Class Codes: G05B 23/0213 Modular or universal configuration of the monitoring system, e.g. monitoring system having modules that may be combined to build monitoring program; monitoring system that can be applied to legacy systems; adaptable monitoring system; using different communication protocols
• CPC Class Codes: G05B 23/0256 injecting test signals and analyzing monitored process response, e.g. injecting the test signal while interrupting the normal operation of the monitored system; superimposing the test signal onto a control signal during normal operation of the monitored system
• CPC Class Codes: G05B 23/0297 Reconfiguration of monitoring system, e.g. use of virtual sensors; change monitoring method as a response to monitoring results

Abstract

A method and a system evaluate an efficacy of a test suite in a model-based diagnostic testing system to determine a revision of the test suite. The evaluation comprises suggesting a test to be added to the test suite based on probabilities of a correct diagnosis and an incorrect diagnosis. The evaluation either alternatively or further comprises identifying a test to be deleted from the test suite based on probabilities of a correct diagnosis for the test suite and for a modified test suite that does not include a test. An efficacy value of each test in the test suite is computed. The system has a computer program comprising instructions that implement an evaluation of diagnostic efficacy of the test suite. The system either is a stand-alone system or is incorporated into the testing system.

Claims

1. A method of determining a revision of a test suite of a model-based diagnostic testing system comprising:

evaluating a diagnostic efficacy of the test suite using a probability of a diagnosis.

2. The method of claim 1, wherein the evaluation comprises:

suggesting a test to add to the test suite to adjust an overall test coverage of the test suite.

3. The method of claim 2, wherein suggesting a test comprises:

creating a simulation database of the test suite;

determining a probability of a correct diagnosis and a probability of an incorrect diagnosis for the test suite using the database; and

creating a list of suggested tests from the determined probabilities.

4. The method of claim 3, wherein each suggested test on the list comprises a test coverage.

5. The method of claim 3, wherein the evaluation further comprises:

identifying a test to delete from the test suite that comprises;

determining a probability of a correct diagnosis for a modified test suite using the database, the modified test suite having a selected test removed from the test suite;

computing an efficacy value associated with the selected test using the determined probabilities of a correct diagnosis for the test suite and the modified test suite; and

generating a list of deletable tests and associated efficacy values.

6. The method of claim 1, wherein the evaluation comprises:

identifying a test to delete from the test suite, the deletable test having a minimal effect on an overall diagnostic efficacy of the test suite.

7. The method of claim 6, wherein identifying a test comprises:

creating a simulation database of the test suite;

determining a probability of a correct diagnosis for the test suite using the database;

determining a probability of a correct diagnosis for a modified test suite using the database, wherein the modified test suite is the test suite having a selected test removed;

computing an efficacy value for the modified test suite using the determined probabilities; and

generating a list of deletable tests using the computed efficacy values.

8. The method of claim 7, wherein determining a probability for a modified test suite is repeated for a plurality of modified test suites, each modified test suite of the plurality being the test suite having a different selected test removed.

9. The method of claim 8, wherein the selected test associated with the modified test suite having a low computed efficacy value relative to other modified test suites is the deletable test.

10. The method of claim 7, wherein evaluating further comprises:

suggesting a test to add to the test suite to adjust an overall test coverage of the test suite that comprises;

determining a probability of an incorrect diagnosis for the test suite using the database; and

creating a list of tests to add from the determined correct and incorrect probabilities for the test suite.

11. A method of evaluating a diagnostic efficacy of a test suite of a model-based diagnostic testing system comprising:

creating a simulation database of the test suite;

determining a probability of one or both of a correct diagnosis and an incorrect diagnosis for the test suite using the database; and

using the determined probability to evaluate the test suite.

12. The method of claim 11, wherein creating a simulation database comprises:

simulating an application of the test suite to a device under test, the device under test comprising one or more components; and

recording a probable result of the application in the simulation database, the simulation database being represented by a table having a plurality of columns and a plurality of rows, the plurality of columns comprising a component pattern, a test result pattern, and a number of occurrences,

wherein the component pattern encodes which component is good or bad, each component of the device under test being represented by a unique position number within the component pattern,

wherein the test result pattern encodes which of the tests of the test suite failed or passed, each test in the test suite being represented by a unique position within the test result pattern,

wherein the number of occurrences represents a number of times that a given combination of the component pattern and the test result pattern occurred during a simulation, the number of occurrences being an integer greater than or equal to zero, and

wherein each row of the plurality of rows corresponds to a different unique pattern of good and bad components.

13. The method of claim 12, wherein determining a probability of one or both of a correct diagnosis and an incorrect diagnosis comprises:

copying to a database copy only those rows of the created simulation database with only one bad component in the component pattern column, all other components in the respective row being good;

sorting the database copy based on the test pattern column, such that the rows with a given test pattern are adjacent to one another, the adjacent rows forming a group of rows;

examining each group of rows to locate a row within each group having a largest number of occurrences relative to other rows within the respective group; and

assigning a diagnosis d to each group, the diagnosis d being the position number of the bad component for the located row.

14. The method of claim 13, wherein determining a probability of one or both of a correct diagnosis and an incorrect diagnosis further comprises creating and initializing a matrix M such that matrix elements M(i,j) of the matrix M are equal to zero for all i and j, where i is an integer that ranges from one to m+1 and where j is an integer that ranges from one to m, where m is the number of tests in the test suite.

15. The method of claim 14, wherein for each group having a test pattern that represents no failed tests, determining a probability further comprises:

adding iteratively for each row r in the group the number of occurrences value of the row r to a current value of the matrix element M(m+1, b) to generate a next value of the matrix element M(m+1, b), where b is a position number of the bad component for the row r.

16. The method of claim 15, wherein for each group having a test pattern that represents at least one failed test, determining a probability further comprises:

adding iteratively for each row r in the group the number of occurrences value of the row r to a current value of the matrix element M(d, b) to generate a next value of the matrix element M(d, b).

17. The method of claim 16, wherein the determined probability of a correct diagnosis P_corr is calculated using

18. The method of claim 17, wherein using the determined probability to evaluate the test suite comprises:

suggesting a test to add to the test suite to improve diagnostic efficacy.

19. The method of claim 18, wherein suggesting a test comprises:

finding a relatively largest value element M(i,j) in the matrix M, where i is not equal to j, the element M(i,j) representing the probability of incorrectly diagnosing component i as the bad component when component j is actually bad; and

suggesting a test t having high coverage for component i and one of either low coverage for component j, if j is not equal to m+1, or coverage being irrelevant, if j is equal to m+1.

20. The method of claim 19, wherein suggesting a test further comprises creating a list of suggested tests, wherein creating a list comprises:

repeating finding and suggesting for each element of a set of largest value elements M(i,j), a test being suggested for each element of the set of elements M(i,j); and

computing a score for each of the suggested tests, the score being computed by dividing the element value M(i,j) by the total accumulated number of occurrences E.

21. The method of claim 20, wherein the list of suggested tests is represented in one or both of human readable form or machine-readable form.

22. The method of claim 17, wherein using the determined probability to evaluate the test suite comprises:

identifying a test t of the test suite that may be deleted from the test suite.

23. The method of claim 22, wherein for identifying a test t to delete from the test suite, the method further comprises:

determining a probability of a correct diagnosis P_corr,t for a modified test suite using the database, the modified test suite having a selected test t removed from the test suite;

computing an efficacy value for the modified test suite using the determined probabilities for the test suite and for the modified test suite; and

generating a list of deletable tests, the deletable tests having a lowest associated efficacy relative to efficacies of other tests in the test suite.

24. The method of claim 23, wherein determining a probability of a correct diagnosis P_corr,t for a modified test suite associated with each of the tests t in the test suite comprises using a modified database created from the database copy, wherein the modified database is created comprising:

copying the database copy into another database copy;

selecting a test t to remove from the test suite;

deleting a position from each test pattern associated with the selected test t from the other database copy; and

copying rows of the other database copy into a modified database, such that any rows that have identical values for the component pattern and the test pattern are combined together in the modified database,

wherein in each row of the modified database that represents a set of combined rows from the other database copy the number of occurrences is a sum of the number of occurrence values for the combined rows.

25. The method of claim 24, wherein the probability of a correct diagnosis P_corr,t for each of the modified test suites is determined in a manner analogous to determining the probability of a correct diagnosis P_corr for the test suite.

26. The method of claim 24, wherein determining a probability of a correct diagnosis P_corr,t for the modified test suite T′

using the modified database comprises;

summing a largest number of occurrences value ν

_max found for each unique test pattern value within the modified database; and

dividing the ν

_max sum by a total number of occurrences E_t, where the total number of occurrences E_t is the sum of all numbers of occurrences in the modified database.

27. The method of claim 23, wherein computing an efficacy value for each of the tests in the test suite comprises computing a difference between the determined probability of a correct diagnosis P_corr,t for the modified test suite corresponding to a selected test t and the determined probability of a correct diagnosis P_corr for the test suite.

28. The method of claim 27, wherein the computed efficacy ε

(t) value further comprises a cost metric c(t) associated with the test t, where ε

(t)=c(t)·

(P_corr,t−

P_corr).

29. The method of claim 27, wherein the generated list of deletable tests comprises an associated efficacy value for each of the deletable tests.

30. The method of claim 23, wherein the generated list is represented in one or both of human readable form and in machine-readable form.

31. The method of claim 11, wherein the created simulation database comprises a Monte Carol simulation of the device under test model, the database having a set of entries, each entry having a field for a number-of-occurrences value, a field for a test result pattern, and a field for a component state pattern.

32. A system that determines efficacy of a test suite of a model-based diagnostic testing system comprising:

a processor;

a memory; and

a computer program stored in the memory and executed by the processor, wherein the computer program comprises instructions that, when executed by the processor, implement evaluating the test suite using a probability of a diagnosis to determine the efficacy.

33. The system of claim 32, wherein the instructions that evaluate the test suite comprise one or both of suggesting a test to add to the test suite, and identifying a test to delete from the test suite.

34. The system of claim 32, wherein the instructions that evaluate the test suite comprise:

creating a simulation database of the test suite;

determining a probability of one or both of a correct diagnosis and an incorrect diagnosis using the database; and

using the determined probability to evaluate the test suite.

35. The system of claim 34, wherein using the determined probability of both a correct diagnosis and an incorrect diagnosis comprises creating a list of suggested tests to add to the test suite, each suggested test having an associated test coverage.

36. The system of claim 34, wherein the instructions that evaluate the test suite further comprise:

determining a probability of a correct diagnosis for a modified test suite using the database, the modified test suite having a selected test removed from the test suite;

and wherein using the determined probability comprises;

computing an efficacy value for the modified test suite using the determined probability of a correct diagnosis for both the test suite and the modified test suite; and

generating a list of tests to delete from the test suite based on the computed efficacy value.

37. The system of claim 36, wherein determining a probability of a correct diagnosis for a modified test suite is repeated for different modified test suites, each different modified test suite having an associated different selected test being removed.

38. The system of claim 33, wherein suggesting a test to add to the test suite and identifying a test to delete from the test suite each comprise a list of respective tests, the lists being represented in one or both of human readable form or machine-readable form.