Automated analysis of a model based diagnostic system

• US 5,922,079 A
• Filed: 10/16/1997
• Issued: 07/13/1999
• Est. Priority Date: 03/08/1996
• Status: Expired due to Fees

• Current Assignee: Agilent Technologies, Inc.
• Original Assignee: Hewlett-Packard Company (HP Inc.)
• Inventors: Manley, Douglas R., Booth, George L., Heumann, John M.
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): Baderman, Scott T.
• Application Number:
  US08/951,287
• Time in Patent Office: 635 Days
• Field of Search: 371/26, 371/27.1, 371/2.1, 395/183.02, 395/183.09, 395/183.08, 395/183.22, 395/185.01, 395/184.01
• US Class Current: 714/26
• CPC Class Codes: G01R 31/31835 Analysis of test coverage or failure detectability
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers

Abstract

An automated analysis system that identifies detectability problems, diagnosability problems, and possible ways to change rank order of diagnoses in a diagnostic system and makes the problems and possible improvements visible to test programmers to aid in test improvement. Components that have no coverage and components that have inadequate coverage (according to a heuristic criteria) are identified as potential detectability problems. Components that are exercised by identical operations in all tests are identified as diagnosability problems. If an incorrect diagnosis is made, the automated analysis system identifies failing tests that have no coverage of any component in the true failure cause. In addition, if an incorrect diagnosis is made, the automated analysis system identifies ways of changing the rank order of diagnoses, including coverages that can be reduced and identification of operation violations that can be eliminated or deliberately added. If no historical data are available, a "diagnosability index" may be computed by randomly sampling from the set of possible failure syndromes and observing the frequencies with which ties occur among the weights of the top-ranked candidate diagnoses. After historical data becomes available, a diagnosability index may be computed from the frequency with which two candidate diagnoses are assigned identical weights by the model-based diagnostic system over a set of representative failures.

Claims

1. A method of identifying improvements to a model-based test system, the model-based test system including a test suite and a model, the method comprising the following steps:

(a) automatically identifying improvements to the test suite;

(b) automatically identifying improvements to the model; and

(c) outputting the improvements identified in steps (a) and (b).

2. The method of claim 1, step (a) further comprising:

identifying components in which failures are not likely detectable.

3. The method of claim 2, step (a) further comprising:

identifying components that are not exercised by any test.

4. The method of claim 2, step (a) further comprising:

identifying components that are inadequately exercised according to a predetermined criteria.

5. The method of claim 1, step (a) further comprising:

identifying pairs of components for which one component cannot be distinguished from the other.

6. The method of claim 5, step (a) further comprising:

identifying sets of components that are exercised by identical operations in all tests.

7. The method of claim 1, step (a) further comprising:

identifying pairs of components for which one component is unlikely to be distinguished from the other.

8. The method of claim 1, wherein the model-based diagnostic system has misidentified a particular set of components as the cause of a set of failing tests instead of identifying the true failure cause components as the cause of the set of failing tests, step (b) further comprising:

identifying probable modeling errors causing the misidentification.

9. The method of claim 8, step (b) further comprising:

identifying components of the true failure cause that are not exercised by the set of failing tests.

10. The method of claim 8, step (b) further comprising:

performing utilization sensitivity analysis on the particular set of components.

11. The method of claim 8, step (c) further comprising:

identifying ways of changing the misidentification by modifying operation violations.

12. A method of identifying improvements to a test suite, the test suite based on a model of a system, the mode not including a description of connection of components in the system, the method comprising the following steps:

(a) automatically identifying components, the failure of which has a probability of detectability by the test suite that is low but greater than zero; and

(b) outputting identities of components identified in step (a).

13. The method of claim 12, further comprising the steps of:

identifying components that are not exercised by any test.

14. The method of claim 12, step (a) further comprising:

identifying components that are inadequately exercised according to a predetermined criteria.

15. A method of identifying improvements to a test suite, the test suite based on a model of a system, the model not including a description of connection of components in the system the method comprising the following steps:

(a) automatically identifying sets of components that are exercised by identical operations in all tests; and

(b) outputting identities of components identified in step (a).

16. A method of identifying improvements to a test suite, the method comprising the following steps:

(a) automatically identifying pairs of components for which one component has a probability that is low but greater than zero from being distinguished from the other; and

(b) outputting identities of the components identified in step (a).

17. A method of identifying improvements to a model-based diagnostic system, wherein the model-based diagnostic system has misidentified a particular set of components as the cause of a set of failing tests instead of identifying the true failure cause components as the cause of the set of failing tests, the method comprising:

(a) automatically identifying components of the true failure cause that are not exercised by the set of failing tests; and

(b) outputting identities of components identified in step (a).

18. A method of identifying improvements to a model-based diagnostic system, wherein the model-based diagnostic system has misidentified a particular set of components as the cause of a set of failing tests instead of identifying the true failure cause components as the cause of the set of failing tests, the method comprising:

(a) automatically identifying a way to increase a diagnostic weight for the true failure cause components; and

(b) outputting the identity of the way of increasing a diagnostic weight identified in step (a).

19. The method of claim 18, step (a) further comprising:

identifying a coverage that may be altered for a set of components within the true failure cause components, resulting in an increased diagnostic weight for the true failure cause components.

20. The method of claim 18, step (a) further comprising:

identifying a particular operation that may be removed from at least one test, the particular operation exercising at least one of the true failure cause components and the particular operation having an operation violation before removal from at least one test and having no operation violation after removal from at least one test.

21. The method of claim 18, step (a) further comprising:

identifying a particular operation that may be added to at least one failing test, the additional operation being one that exercises at least one of the true failure cause components and does not have an operation violation.

22. A method of identifying improvements to a model-based diagnostic system, wherein the model-based diagnostic system has misidentified a particular set of components as the cause of a set of failing tests instead of identifying the true failure cause components as the cause of the set of failing tests, the method comprising:

(a) automatically identifying a way to decrease a diagnostic weight for the particular set of components; and

(b) outputting the identity of the way of decreasing a diagnostic weight identified in step (a).

23. The method of claim 22, step (a) further comprising:

identifying a coverage that may be altered for the particular set of components, resulting in a decreased diagnostic weight for the particular set of components.

24. The method of claim 22, step (a) further comprising:

identifying a particular operation that may be added to at least one failing test, the particular operation being one that exercises at least one of the particular set of components and will have an operation violation when added.