System, method and computer product for incremental improvement of algorithm performance during algorithm development

• US 20030093250A1
• Filed: 11/08/2001
• Published: 05/15/2003
• Est. Priority Date: 11/08/2001
• Status: Active Grant

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Goebel, Kai Frank
• Granted Patent: US 7,050,950 B2
• Time in Patent Office: Days
• US Class Current: 703/2
• CPC Class Codes: G06F 17/10 Complex mathematical operations function generation by table look-up G06F1/03; evaluation of elementary functions by calculation G06F7/544

Abstract

System, method and computer product for incremental improvement of algorithm performance during algorithm development. In one aspect of this disclosure, there is a system, method and computer readable medium that stores instructions for instructing a computer system, to assess the performance of an algorithm during development. In this embodiment, a design of experiments component establishes an acceptable number of experiments for analyzing incremental improvements of the algorithm. An experiment performance component runs the established number of experiments for the algorithm. A simulation component simulates the behavior of the algorithm using results from the experiment performance component.

Claims

1. A system for assessing the performance of an algorithm during development, comprising:

a design of experiments component that establishes an acceptable number of experiments for analyzing the algorithm;

an experiment performance component that runs the established number of experiments for the algorithm; and

a simulation component that simulates the behavior of the algorithm using results from the experiment performance component.

2. The system according to claim 1, wherein the design of experiments component selects an appropriate level of experiments for analyzing the algorithm.

3. The system according to claim 2, wherein the design of experiments component selects the appropriate level of experiments from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

4. The system according to claim 1, wherein the experiment performance component comprises a performance metric component that evaluates the results of the experiments run for the algorithm.

5. The system according to claim 4, wherein the experiment performance component comprises an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

6. The system according to claim 5, wherein the experiment performance component runs the established number of experiments for the algorithm after adjustment of logic or parameters.

7. The system according to claim 1, further comprising a module insertion component that inserts a module into the algorithm.

8. The system according to claim 7, wherein the experiment performance component runs the established number of experiments for the algorithm including the inserted module.

9. The system according to claim 1, wherein the simulation component performs a Monte Carlo simulation.

10. The system according to claim 1, wherein the simulation component uses at least one confusion matrix to simulate the behavior of the algorithm.

11. The system according to claim 1, further comprising a simulation performance component that evaluates the performance of the simulation of the algorithm.

12. The system according to claim 11, wherein the simulation performance component comprises an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

13. The system according to claim 12, wherein the simulation component runs the simulation for the algorithm after adjustment of logic or parameters.

113. The computer-readable medium according to claim 12, further comprising instructions for running the Monte Carlo simulation for the algorithm after adjustment of logic or parameters.

14. A system for assessing the performance of an algorithm during development, comprising:

a design of experiments component that establishes an acceptable number of experiments for analyzing the algorithm;

an experiment performance component that runs the established number of experiments for algorithm;

a simulation component that simulates the behavior of the algorithm using results from the experiment performance component; and

a simulation performance component that evaluates the performance of the simulation for the algorithm.

15. The system according to claim 14, wherein the design of experiments component selects an appropriate level of experiments for analyzing the algorithm.

16. The system according to claim 15, wherein the design of experiments component selects the appropriate level of experiments from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

17. The system according to claim 14, wherein the experiment performance component comprises a performance metric component that evaluates the results of the experiments run for the algorithm.

18. The system according to claim 17, wherein the experiment performance component comprises an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

19. The system according to claim 18, wherein the experiment performance component runs the established number of experiments for the portion of the algorithm after adjustment of parameters.

20. The system according to claim 14, further comprising a module insertion component that inserts a module into the algorithm.

21. The system according to claim 20, wherein the experiment performance component runs the established number of experiments for the algorithm including the inserted module.

22. The system according to claim 14, wherein the simulation component performs a Monte Carlo simulation.

23. The system according to claim 14, wherein the simulation component uses at least one confusion matrix to simulate the behavior of the algorithm.

24. The system according to claim 14, wherein the simulation performance component comprises an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

25. The system according to claim 24, wherein the simulation component runs the simulation for the algorithm after adjustment of logic or parameters.

26. A system for assessing the performance of an algorithm during development, comprising:

a design of experiments component that establishes an acceptable number of experiments for analyzing the algorithm;

an experiment performance component that runs the established number of experiments for the algorithm and uses a performance metric to evaluate the results of the experiments;

a Monte Carlo simulation component that simulates the behavior of the algorithm using results from the experiment performance component with a Monte Carlo simulation; and

a simulation performance component that evaluates the performance of the Monte Carlo simulation for the algorithm.

27. The system according to claim 26, wherein the design of experiments component selects an appropriate level of experiments from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

28. The system according to claim 26, wherein the experiment performance component comprises an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

29. The system according to claim 28, wherein the experiment performance component runs the established number of experiments for the algorithm after adjustment of logic or parameters.

30. The system according to claim 26, wherein the Monte Carlo simulation component uses at least one confusion matrix to simulate the behavior of the algorithm.

31. The system according to claim 26, wherein the simulation performance component comprises an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

32. The system according to claim 31, wherein the simulation component runs the simulation for the algorithm after adjustment of parameters.

33. A system for assessing the performance of an algorithm during development, comprising:

a design of experiments component that establishes an acceptable number of experiments for analyzing the algorithm;

an experiment performance component that runs the established number of experiments for the algorithm;

a performance metric component that evaluates the results of the experiments run for the algorithm; and

an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

34. The system according to claim 33, wherein the design of experiments component selects an appropriate level of experiments for analyzing the algorithm from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

35. The system according to claim 33, wherein the experiment performance component runs the established number of experiments for the algorithm after adjustment of logic or parameters.

36. The system according to claim 33, further comprising a module insertion component that inserts a module into the algorithm.

37. The system according to claim 36, wherein the experiment performance component runs the established number of experiments for the algorithm including the inserted module.

38. A system for assessing the performance of an algorithm during development, comprising:

a Monte Carlo simulation component that simulates the behavior of the algorithm with a Monte Carlo simulation, wherein the Monte Carlo simulation component uses at least one confusion matrix to simulate the behavior of the algorithm;

a simulation performance component that evaluates the performance of the Monte Carlo simulation for the algorithm; and

an algorithm adjustment component that adjusts logic or parameters of the algorithm for unacceptable results.

39. The system according to claim 38, wherein the Monte Carlo simulation component runs the simulation for the algorithm after adjustment of logic or parameters.

40. A method for assessing the performance of an algorithm during development, comprising:

using design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm; and

simulating the behavior of the algorithm using results from the experiments.

41. The method according to claim 40, wherein the using of design of experiments comprises selecting an appropriate level of experiments for analyzing the algorithm.

42. The method according to claim 41, wherein the appropriate level of experiments comprise at least one of full factorial experiments, fractional factorial experiments or screening experiments.

43. The method according to claim 40, further comprising using a performance metric to evaluate the results of the experiments run for the algorithm.

44. The method according to claim 43, further comprising adjusting logic or parameters of the algorithm for unacceptable results.

45. The method according to claim 44, further comprising running the established number of experiments for the algorithm after adjustment of logic or parameters.

46. The method according to claim 40, further comprising inserting a module into the algorithm.

47. The method according to claim 46, further comprising running the established number of experiments for the algorithm including the inserted module.

48. The method according to claim 40, wherein the simulating comprises using at least one confusion matrix to simulate the behavior of the algorithm.

49. The method according to claim 40, further comprising evaluating the performance of the simulation for the algorithm.

50. The method according to claim 49, further comprising adjusting logic or parameters of the algorithm for unacceptable results.

51. The method according to claim 50, further comprising running the simulation for the algorithm after adjustment of logic or parameters.

52. A method for assessing the performance of an algorithm during development, comprising:

using design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm;

simulating the behavior of the algorithm using results from the experiments; and

evaluating the performance of the simulation for the algorithm.

53. The method according to claim 52, wherein the using of design of experiments comprises selecting an appropriate level of experiments for analyzing the algorithm from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

54. The method according to claim 52, further comprising using a performance metric to evaluate the results of the experiments run for the algorithm.

55. The method according to claim 54, further comprising adjusting logic or parameters of the algorithm for unacceptable results.

56. The method according to claim 55, further comprising running the established number of experiments for the algorithm after adjustment of logic or parameters.

57. The method according to claim 52, further comprising inserting a module into the algorithm.

58. The method according to claim 57, further comprising running the established number of experiments for the algorithm including the inserted module.

59. The method according to claim 52, wherein the simulation is a Monte Carlo simulation.

60. The method according to claim 52, wherein the simulating uses at least one confusion matrix to simulate the behavior of the algorithm.

61. The method according to claim 52, further comprising adjusting logic or parameters of the algorithm for unacceptable results.

62. The method according to claim 61, further comprising running the simulation for the algorithm after adjustment of logic or parameters.

63. A method for assessing the performance of an algorithm during development, comprising:

using a design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm;

using a performance metric to evaluate the results of the experiments;

using a Monte Carlo simulation to simulate the behavior of the algorithm using results from the performance metric; and

evaluating the performance of the Monte Carlo simulation for the algorithm.

64. The method according to claim 63, wherein using the design of experiments comprises selecting an appropriate level of experiments from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

65. The method according to claim 63, further comprising adjusting logic or parameters of the algorithm for unacceptable results.

66. The method according to claim 65, further comprising running the established number of experiments for the algorithm after adjustment of logic or parameters.

67. The method according to claim 63, wherein the simulating comprises using at least one confusion matrix to simulate the behavior of the algorithm.

68. The method according to claim 63, further comprising adjusting logic or parameters of the algorithm for unacceptable results.

69. The method according to claim 68, further comprising running the simulation for the algorithm after adjustment of logic or parameters.

70. A method for assessing the performance of an algorithm during development, comprising:

using a design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm;

evaluating the results of the experiments run for the algorithm with a performance metric; and

adjusting logic or parameters of the algorithm for unacceptable results.

71. The method according to claim 70, wherein the using of design of experiments comprises selecting an appropriate level of experiments for analyzing the algorithm from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

72. The method according to claim 70, further comprising running the established number of experiments for the algorithm after adjustment of parameters.

73. The method according to claim 70, further comprising inserting a module into the algorithm.

74. The method according to claim 73, further comprising running the established number of experiments for the algorithm including the inserted module.

75. A method for assessing the performance of an algorithm during development, comprising:

simulating the behavior of the algorithm with a Monte Carlo simulation, wherein the Monte Carlo simulation uses at least one confusion matrix to simulate the behavior of the algorithm;

evaluating the performance of the Monte Carlo simulation for the algorithm; and

adjusting logic or parameters of the algorithm for unacceptable results.

76. The method according to claim 75, further comprising running the Monte Carlo simulation for the algorithm after adjustment of logic or parameters.

77. A computer-readable medium storing computer instructions for instructing a computer system to assess the performance of an algorithm during development, the computer instructions comprising:

using design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm; and

simulating the behavior of the algorithm using results from the experiments.

78. The computer-readable medium according to claim 77, wherein the using of design of experiments comprises instructions for selecting an appropriate level of experiments for analyzing the algorithm.

79. The computer-readable medium according to claim 78, wherein the appropriate level of experiments comprise at least one of full factorial experiments, fractional factorial experiments or screening experiments.

80. The computer-readable medium according to claim 77, further comprising instructions for using a performance metric to evaluate the results of the experiments run for the algorithm.

81. The computer-readable medium according to claim 80, further comprising instructions for adjusting logic or parameters of the algorithm for unacceptable results.

82. The computer-readable medium according to claim 81, further comprising instructions for running the established number of experiments for the algorithm after adjustment of logic or parameters.

83. The computer-readable medium according to claim 77, further comprising instructions for inserting a module into the algorithm.

84. The computer-readable medium according to claim 83, further comprising instructions for running the established number of experiments for the algorithm including the inserted module.

85. The computer-readable medium according to claim 77, wherein the simulating comprises instructions for using at least one confusion matrix to simulate the behavior of the algorithm.

86. The computer-readable medium according to claim 77, further comprising instructions for evaluating the performance of the simulation for the algorithm.

87. The computer-readable medium according to claim 86, further comprising instructions for adjusting logic or parameters of the algorithm for unacceptable results.

88. The computer-readable medium according to claim 87, further comprising instructions for running the simulation for the algorithm after adjustment of logic or parameters.

89. A computer-readable medium storing computer instructions for instructing a computer system to assess the performance of an algorithm during development, the computer instructions comprising:

using design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm;

simulating the behavior of the algorithm using results from the experiments; and

evaluating the performance of the simulation for the algorithm.

90. The computer-readable medium according to claim 89, wherein the using of design of experiments comprises instructions for selecting an appropriate level of experiments for analyzing the algorithm from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

91. The computer-readable medium according to claim 89, further comprising instructions for using a performance metric to evaluate the results of the experiments run for the algorithm.

92. The computer-readable medium according to claim 91, further comprising instructions for adjusting logic or parameters of the algorithm for unacceptable results.

93. The computer-readable medium according to claim 92, further comprising instructions for running the established number of experiments for the algorithm after adjustment of logic or parameters.

94. The computer-readable medium according to claim 89, further comprising instructions for inserting a module into the algorithm.

95. The computer-readable medium according to claim 94, further comprising instructions for running the established number of experiments for the algorithm including the inserted module.

96. The computer-readable medium according to claim 89, wherein the simulation is a Monte Carlo simulation.

97. The computer-readable medium according to claim 89, wherein the simulating uses at least one confusion matrix to simulate the behavior of the algorithm.

98. The computer-readable medium according to claim 89, further comprising instructions for adjusting logic or parameters of the algorithm for unacceptable results.

99. The computer-readable medium according to claim 98, further comprising instructions for running the simulation for the algorithm after adjustment of logic or parameters.

100. A computer-readable medium storing computer instructions for instructing a computer system to assess the performance of an algorithm during development, the computer instructions comprising:

using a design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm;

using a performance metric to evaluate the results of the experiments;

using a Monte Carlo simulation to simulate the behavior of the algorithm using results from the performance metric; and

evaluating the performance of the Monte Carlo simulation for the he algorithm.

101. The computer-readable medium according to claim 100, wherein using the design of experiments comprises instructions for selecting an appropriate level of experiments from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

102. The computer-readable medium according to claim 100, further comprising instructions for adjusting logic or parameters of the algorithm for unacceptable results.

103. The computer-readable medium according to claim 102, further comprising instructions for running the established number of experiments for the algorithm after adjustment of logic or parameters.

104. The computer-readable medium according to claim 100, wherein the simulating comprises instructions for using at least one confusion matrix to simulate the behavior of the algorithm.

105. The computer-readable medium according to claim 100, further comprising instructions for adjusting logic or parameters of the algorithm for unacceptable results.

106. The computer-readable medium according to claim 105, further comprising instructions for running the simulation for the algorithm after adjustment of logic or parameters.

107. A computer-readable medium storing computer instructions for instructing a computer system to assess the performance of an algorithm during development, the computer instructions comprising:

using a design of experiments to establish an acceptable number of experiments for analyzing the algorithm;

running the established number of experiments for the algorithm;

evaluating the results of the experiments run for the algorithm with a performance metric; and

adjusting logic or parameters of the algorithm for unacceptable results.

108. The computer-readable medium according to claim 107, wherein the using of design of experiments comprises instructions for selecting an appropriate level of experiments for analyzing the algorithm from at least one of full factorial experiments, fractional factorial experiments or screening experiments.

109. The computer-readable medium according to claim 107, further comprising instructions for running the established number of experiments for the algorithm after adjustment of logic or parameters.

110. The computer-readable medium according to claim 107, further comprising instructions for inserting a module into the algorithm.

111. The computer-readable medium according to claim 110, further comprising instructions for running the established number of experiments for the algorithm including the inserted module.

112. A computer-readable medium storing computer instructions for instructing a computer system to assess the performance of an algorithm during development, the computer instructions comprising:

simulating the behavior of the algorithm with a Monte Carlo simulation, wherein the Monte Carlo simulation uses at least one confusion matrix to simulate the behavior of the algorithm;

evaluating the performance of the Monte Carlo simulation for the algorithm; and

adjusting logic or parameters of the algorithm for unacceptable results.