Method and apparatus for identifying the coverage of a test sequence in a rules-based expert system

• US 6,125,359 A
• Filed: 09/23/1997
• Issued: 09/26/2000
• Est. Priority Date: 09/23/1997
• Status: Expired due to Term

• Current Assignee: Unisys Corporation
• Original Assignee: Unisys Corporation
• Inventors: Austin, Thomas K., Lautzenheiser, Ted G., Peters, Thomas R.
• Primary Examiner(s): Hafiz, Tariq R.
• Assistant Examiner(s): PENDER JR, MICHAEL J
• Application Number:
  US08/937,352
• Time in Patent Office: 1,099 Days
• Field of Search: 706/60
• US Class Current: 706/60
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

A method and apparatus for efficiently debugging and/or testing a rules based expert system. To provide guidance when updating a test sequence, the present invention contemplates identifying which rule sets and/or rules were exercised by the test sequence, and which were not exercised. The present invention also contemplates identifying the rule sets and/or rules that were exercised, and the percent of the rule sets/rules that were exercised. This and other information may be useful in identifying appropriate changes for the test sequence so that those rule sets and/or rules that were not exercised during the previous iteration are exercised in a subsequent iteration.

Claims

1. A method for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rules selected from a number of available rules and execute one or more of the rules that are applied, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules in the rules-based expert system, the method comprising the steps of:

a. providing a first indicator for each of the number of available rules of said rules-based expert system having said inference engine;

b. setting the first indicator for a particular rule when that particular rule is applied; and

c. determining the number of rules that were applied during the execution of the test sequence by examining the first indicator for each of the available rules.

2. A method according to claim 1 wherein the first indicator is a count value, and wherein the count value is incremented each time the corresponding rule is applied.

3. A method according to claim 2 wherein said determining step 1(c) determines the number of rules that were applied during the execution of the test sequence by comparing the count value of the first indicator of each rule with a predetermined value.

4. A method according to claim 3 wherein the predetermined value is zero.

5. A method according to claim 1 further comprising the step of determining the percent of the available rules that were applied during the execution of the test sequence.

6. A method according to claim 1 wherein each of the available rules has a second indicator, the second indicator being set for a particular rule when that particular rule is executed.

7. A method according to claim 6 further comprising the step of determining the number of rules that were executed during the execution of the test sequence by examining the second indicator for each of the available rules.

8. A method according to claim 6 wherein the second indicator is a count value, and wherein the count value is incremented each time the corresponding rule is executed.

9. A method according to claim 8 further comprising the step of determining the number of rules that were executed during the execution of the test sequence by comparing the count value of the second indicator of each rule with a predetermined value.

10. A method according to claim 9 wherein the predetermined value is zero.

11. A method according to claim 6 further comprising the step of determining the percent of the available rules that were actually executed during the execution of the test sequence.

12. A method for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rule sets selected from a number of available rule sets, each of the available rule sets having a listing of rules selected from a number of available rules, the execution of one of the number of rule sets causing each of the rules in the corresponding listing of rules to be applied, and selected ones of the number of rules that are applied being actually executed by the rules-based expert system, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rule sets in the rules-based expert system, the method comprising the steps of:

a. providing a rule set indicator for each of the number of available rule sets of said rules-based expert system having said inference engine;

b. setting the rule set indicator for a particular rule set when that particular rule set is executed; and

c. determining the number of rules sets that were executed during the execution of the test sequence by examining the rule set indicator for each of the available rule sets.

13. A method according to claim 12 wherein the rule set indicator is a count value, and wherein the count value is incremented each time the corresponding rule set is executed.

14. A method according to claim 13 wherein said determining step 12(c) determines the number of rule sets that were executed during the execution of the test sequence by comparing the count value of the rule set indicator of each rule set with a predetermined value.

15. A method according to claim 14 wherein the predetermined value is zero.

16. A method according to claim 12 further comprising the step of identifying which of the rule sets were executed during the execution of the test sequence.

17. A method according to claim 16 wherein the rule sets are identified by appending a rule set name to a listing of executed rule set names when the corresponding rule set is executed.

18. A method according to claim 17 further comprising the step of determine the percentage of available rule sets that were executed during the execution of the test sequence.

19. A method according to claim 16 further comprising the step of identifying which of the available rules were applied during the execution of the test sequence.

20. A method according to claim 19 wherein the available rules that are applied are identified by examining the listing of rules that are associated with each of the rules sets that are identified as having been executed during the execution of the test sequence.

21. A method according to claim 20 wherein the available rules that are not applied during the execution of the test sequence are identified by comparing the rules that were identified as having been applied during the execution of the test sequence with all available rules.

22. A method according to claim 20 wherein the available rules that are actually executed are identified by maintaining a test indicator for each rule, wherein the test indicator is set for a particular rule when that particular rule is executed.

23. A method according to claim 22 further comprising the step of determining the number of rules that were executed during the execution of the test sequence by examining the test indicator for each of the available rules.

24. A method according to claim 23 further comprising the step of determining the percent of the available rules that were actually executed during the execution of the test sequence.

25. A method according to claim 16 wherein the available rule sets that are not executed during the execution of the test sequence are identified by comparing the rule sets that were identified as having been executed during the execution of the test sequence with all available rule sets.

26. A method for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rules selected from a number of available rules and execute one or more of the rules that are applied, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules in the rules-based expert system, the method comprising the steps of:

a. providing a first indicator for each of the number of available rules of said rules-based expert system having said inference engine;

b. setting the first indicator for a particular rule when that particular rule is applied; and

c. determining which rules were applied during the execution of the test sequence by examining the first indicator for each of the available rules.

27. A method according to claim 26 further comprising the steps of:

d. providing a second indicator for each of the number of available rules;

e. setting the second indicator for a particular rule when that particular rule is actually executed; and

f. determining which rules were actually executed during the execution of the test sequence by examining the second indicator for each of the available rules.

28. A method for identifying the test coverage of a test sequence that is executed in a rules-based expert system having said inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rules selected from a number of available rules and execute one or more of the rules that are applied, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules in the rules-based expert system, the method comprising the steps of:

a. providing a first indicator for each of the number of available rules of said rules-based expert system having said inference engine;

b. setting the first indicator for a particular rule when that particular rule is executed; and

c. determining which of the rules were executed during the execution of the test sequence by examining the first indicator for each of the available rules.

29. A method for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rules selected from a number of available rules and execute one or more of the rules that are applied, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules in the rules-based expert system, the method comprising the steps of:

a. providing a first indicator for each of the number of available rules of said rules-based expert system having said inference engine;

b. setting the first indicator for a particular rule when that particular rule is executed; and

c. determining the number of rules that were executed during the execution of the test sequence by examining the first indicator for each of the available rules.

30. A method for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference system, the execution of the test sequence causing the rules-based expert system to apply a number of rule sets selected from a number of available rule sets, each of the available rule sets having a listing of rules selected from a number of available rules, the execution of one of the number of rule sets causing each of the rules in the corresponding listing of rules to be applied, and selected ones of the number of rules that are applied being actually executed by the rules-based expert system, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules in the rules-based expert system, the method comprising the steps of:

a. providing a rule set indicator for each of the number of available rule sets of said rules-based expert system having said inference engine;

b. setting the rule set indicator for a particular rule set when that particular rule set is executed; and

c. determining which rule sets were executed during the execution of the test sequence by examining the rule set indicator for each of the available rule sets.

31. A method according to claim 30 further comprising the steps of:

d. identifying which rules where applied during the execution of the test sequence by examining the corresponding listing of rules for each of the rule sets that are determined to have been executed during the execution of the test sequence.

32. A method according to claim 31 further comprising the steps of:

e. providing a test indicator for each of the number of available rules;

f. setting the test indicator for a particular rule when that particular rule is actually executed; and

g. determining which rules were actually executed during the execution of the test sequence by examining the test indicator for each of the available rules.

33. An apparatus for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rules selected from a number of available rules and execute one or more of the rules that are applied, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules in the rules-based expert system, comprising:

a. storing module for storing a first indicator for each of the number of available rules of said rules-based expert system having said inference engine;

b. setting module coupled to said storing module for setting the first indicator for a particular rule when that particular rule is applied; and

c. determining module coupled to said storing module for determining the number of rules that are applied during the execution of the test sequence by examining the first indicator for each of the available rules.

34. An apparatus according to claim 33 wherein said storing module comprises a number of counter means, one for each of the available rules, each of said counter means being incremented each time the corresponding rule is applied.

35. An apparatus according to claim 34 wherein said determining module determines the number of rules that were applied during the execution of the test sequence by determining which rules have a count value in the corresponding counter means that is greater than a predetermined value.

36. An apparatus according to claim 33 wherein each of the available rules has a second indicator, the second indicator being set for a particular rule when that particular rule is executed.

37. An apparatus according to claim 36 wherein said determining module further determines the number of rules that were executed during the execution of the test sequence by examining the second indicator for each of the available rules.

38. An apparatus for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rule sets selected from a number of available rule sets, each of the available rule sets having a listing of rules selected from a number of available rules, the execution of one of the number of rule sets causing each of the rules in the corresponding listing of rules to be applied, and selected ones of the number of rules that are applied being actually executed by the rules-based expert system, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules sets in the rules-based expert system, comprising:

a. storing means for storing a rule set indicator for each of the number of available rule sets of said rules-based expert system having said inference engine;

b. setting means coupled to said storing means for setting the rule set indicator for a particular rule set when that particular rule set is executed; and

c. determining means coupled to said storing means for determining the number of rules sets that were applied during the execution of the test sequence by examining the rule set indicator for each of the available rule sets.

39. An apparatus according to claim 38 wherein said storing means comprises a number of counter means, one for each of the available rules sets, each of said counter means being incremented each time the corresponding rule set is applied.

40. An apparatus according to claim 39 wherein said determining means determines the number of rule sets that were executed during the execution of the test sequence by determining which rule sets have a count value in the corresponding counter means that is greater than a predetermined value.

41. An apparatus according to claim 38 wherein said determining means further determines the percentage of available rule sets that were executed during the execution of the test sequence.

42. An apparatus according to claim 38 wherein said determining means further determines which of the rule sets were executed during the execution of the test sequence.

43. An apparatus according to claim 42 wherein said determining means deter mines which of the rule sets that were executed during the execution of the test sequence by appending a rule set name to a listing of executed rule set names when the corresponding rule set is executed.

44. An apparatus according to claim 43 wherein said determining means further determines which of the available rules were applied during the execution of the test sequence.

45. An apparatus according to claim 44 wherein said determining means determines which of the available rules were applied by examining the listing of rules that are associated with each of the rules sets that are identified as having been executed during the execution of the test sequence.

46. An apparatus according to claim 45 wherein said determining means determines which of the available rule sets are not executed during the execution of the test sequence by comparing the rule sets that were identified as having been executed during the execution of the test sequence with all available rule sets.

47. An apparatus according to claim 45 wherein said determining means determines which of the available rules are not applied during the execution of the test sequence by comparing the rules that were identified as having been applied during the execution of the test sequence with all available rules.

48. An apparatus according to claim 47 wherein said determining means determines the available rules that are actually executed by maintaining a test indicator for each rule, wherein the test indicator is set for a particular rule when that particular rule is executed.

49. An apparatus according to claim 48 wherein said determining means determines the number of rules that were executed during the execution of the test sequence by examining the test indicator for each of the available rules.

50. An apparatus according to claim 49 wherein said determining means further determines the percent of the available rules that were actually executed during the execution of the test sequence.

51. An apparatus for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rules selected from a number of available rules and execute one or more of the rules that are applied, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rules in the rules-based expert system, comprising:

a. storing module for storing a first indicator for each of the number of available rules of said rules-based expert system having said inference engine;

b. setting module coupled to said storing module for setting the first indicator for a particular rule when that particular rule is applied; and

c. determining module coupled to said storing module for determining which rules were applied during the execution of the test sequence by examining the first indicator for each of the available rules.

52. An apparatus according to claim 51 wherein said storing module further stores a second indicator for each of the number of available rules, and said setting module sets the second indicator for a particular rule when that particular rule is actually executed, and wherein said determining module determines which rules were actually executed during the execution of the test sequence by examining the second indicator for each of the available rules.

53. An apparatus for identifying the test coverage of a test sequence that is executed in a rules-based expert system having an inference engine, the execution of the test sequence causing the rules-based expert system to apply a number of rule sets selected from a number of available rule sets, each of the available rule sets having a listing of rules selected from a number of available rules, the execution of one of the number of rule sets causing each of the rules in the corresponding listing of rules to be applied, and selected ones of the number of rules that are applied being actually executed by the rules-based expert system, the test coverage indicating the degree that the test sequence was successful in exercising each of the number of available rule sets in the rules-based expert system, comprising the steps of:

a. storing module for storing a rule set indicator for each of the number of available rule sets of said rules-based expert system having said inference engine;

b. setting module coupled to said storing module for setting the rule set indicator for a particular rule set when that particular rule set is executed; and

c. determining module coupled to said storing module for determining which rules sets were executed during the execution of the test sequence by examining the rule set indicator for each of the available rule sets.

54. An apparatus according to claim 53 wherein said determining module further determines which rules where applied during the execution of the test sequence by examining the corresponding listing of rules for each of the rule sets that are determined to have been executed during the execution of the test sequence.

55. An apparatus according to claim 54 wherein said storing module stores a test indicator for each of the number of available rules, said setting means sets the test indicator for a particular rule when that particular rule is actually executed, and said determining means determines which rules were actually executed during the execution of the test sequence by examining the test indicator for each of the available rules.