System and methods for fault path testing through automated error injection

• US 20030172321A1
• Filed: 03/11/2002
• Published: 09/11/2003
• Est. Priority Date: 03/11/2002
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Oldfield, Barry J., Rust, Robert A., Wolin, Dale Haddon
• Granted Patent: US 7,020,803 B2
• Time in Patent Office: Days
• US Class Current: 714/41
• CPC Class Codes: G06F 11/2215 to test error correction or detection circuits

Abstract

The system and methods described herein relate to testing and verifying the fault tolerance in fault tolerant systems. Fault logic integrated into a fault tolerant system permits automated testing of fault paths in system firmware and hardware dedicated to handling fault scenarios. Advantages of the disclosed system and methods include the ability to inject errors without the need to modify system firmware or hardware. Errors can be injected in a controlled manner and asynchronously to normal system firmware execution which permits improved coverage of firmware error paths. The automated error injection capability disclosed is applicable in both the development and production of fault tolerant systems.

Claims

1. A fault tolerant device comprising:

a controller;

fault logic integrated into the controller; and

a fault injection module configured to inject fault commands into the fault logic.

2. A fault tolerant device as recited in claim 1, further comprising:

an independent error injection port through which the fault logic receives fault commands.

3. A fault tolerant device as recited in claim 1, further comprising:

one or more registers integrated into the fault logic to receive one or more error injection lock-out keys.

4. A fault tolerant system comprising:

a fault tolerant device having integrated fault logic, the fault logic configured to simulate faults in the fault tolerant device based on fault commands; and

an independent error injection port on the fault tolerant device through which the fault logic receives external fault commands.

5. A fault tolerant system as recited in claim 4, further comprising:

a test device connectable to the fault tolerant device; and

an external fault injection module executable on the test device to generate the external fault commands.

6. A fault tolerant system as recited in claim 4, further comprising:

an internal fault injection module executable on the fault tolerant device to generate internal fault commands and inject the internal fault commands into the fault logic.

7. A fault tolerant disk array comprising:

a controller;

a fault injection module configured to generate fault commands; and

fault logic integrated into the controller and configured to simulate faults in the fault tolerant disk array based on fault commands.

8. A fault tolerant disk array as recited in claim 7, further comprising:

an independent error injection port through which the fault logic receives fault commands.

9. In a fault tolerant system, a controller comprising:

a memory control device to manage data storage and retrieval; and

fault logic integrated into the memory control device to interpret fault instructions and simulate faults on the controller.

10. A controller as recited in claim 9, further comprising:

a processor; and

a fault module executable on the processor to generate fault instructions.

11. A controller as recited in claim 9, further comprising:

an independent error injection port to receive fault instructions from an external test device.

12. A method of validating fault tolerance within a fault tolerant device comprising:

receiving a fault instruction through an independent error injection port on the fault tolerant device; and

simulating a fault with fault logic integrated into the fault tolerant device based on the fault instruction.

13. A method as recited in claim 12, wherein the receiving further comprises:

generating the fault instruction in a test device that is external to the fault tolerant device; and

sending the fault instruction from the test device to the fault tolerant device through the independent error injection port.

14. A method as recited in claim 13, wherein the generating further comprises executing a fault module on the test device.

15. A method as recited in claim 12, further comprising:

detecting the fault; and

responding to the fault in a tolerant manner.

16. A method as recited in claim 12, further comprising:

receiving an error injection lock-out key; and

enabling the fault logic based on the error injection lock-out key.

17. A method as recited in claim 12, further comprising implementing drive-fight avoidance to prevent damage to electronics in the fault tolerant device.

18. A method as recited in claim 12, further comprising:

detecting a bad controller in the fault tolerant device; and

injecting faults into the fault tolerant device to disable interfaces to the bad controller and fence off the bad controller from the fault tolerant device.

19. A processor-readable medium comprising processor-executable instructions configured for:

receiving a fault instruction through an independent error injection port on a fault tolerant device; and

simulating a fault with fault logic integrated into the fault tolerant device based on the fault instruction.

20. A method of validating fault tolerance in a fault tolerant system comprising:

during a power-up process;

generating an internal fault instruction in a fault tolerant device;

injecting the internal fault instruction into fault logic integrated into the fault tolerant device; and

based on the internal fault instruction, simulating a fault in the fault tolerant device with the fault logic.

21. A method as recited in claim 20, wherein the generating further comprises executing a fault module on a system processor.

22. A method as recited in claim 20, further comprising:

detecting the fault; and

responding to the fault in a tolerant manner.

23. A method as recited in claim 20, further comprising:

receiving an error injection lock-out key; and

enabling the fault logic based on a value of the error injection lock-out key.

24. A method as recited in claim 20, further comprising implementing drive-fight avoidance to prevent damage to electronics in the fault tolerant system.

25. A method as recited in claim 20, further comprising:

detecting a bad controller in the fault tolerant system; and

injecting faults into the fault tolerant system to disable interfaces to the bad controller and fence off the bad controller from the fault tolerant system.

26. A method as recited in claim 20, further comprising:

after the power-up process;

generating an external fault instruction in a test device coupled to the fault tolerant device;

injecting the external fault instruction into the fault logic through an independent error injection port on the fault tolerant device; and

simulating a fault with the fault logic based on the external fault instruction.

27. A processor-readable medium comprising processor-executable instructions configured for:

during a power-up process;

generating an internal fault instruction in a fault tolerant device;

injecting the internal fault instruction into fault logic integrated into the fault tolerant device; and

based on the internal fault instruction, simulating a fault in the fault tolerant device with the fault logic.

28. A method of testing a fault tolerant system comprising:

during a system power-up process,

generating an internal fault instruction within the system; and

implementing the internal fault instruction with fault logic integrated into the system; and

after the system power-up process,

receiving an external fault instruction from outside the system; and

implementing the external fault instruction with the fault logic.

29. A method as recited in claim 28, further comprising:

checking a value of an error injection lock-out key; and

enabling the fault logic based on the value of the error injection lock-out key.

30. A method as recited in claim 28, wherein the generating further comprises:

executing a fault module on a system processor, the fault module configured to generate fault instructions; and

injecting the internal fault instruction into the fault logic from the system processor.

31. A method as recited in claim 28, wherein the receiving further comprises:

generating the external fault instruction in an external test device; and

injecting the external fault instruction into the fault logic through an independent error injection port on the system.

32. A processor-readable medium comprising processor-executable instructions configured for:

during a system power-up process,

generating an internal fault instruction within a fault tolerant system; and

implementing the internal fault instruction with fault logic integrated into the fault tolerant system; and

after the system power-up process,

receiving an external fault instruction from outside the fault tolerant system; and

implementing the external fault instruction with the fault logic.

33. A method of validating fault tolerance in a fault tolerant system comprising:

generating a fault instruction in a test device;

injecting the fault instruction into a fault tolerant device through an independent error injection port of the fault tolerant device; and

based on the fault instruction, simulating a fault in the fault tolerant device through fault logic integrated into the fault tolerant device.

34. A method as recited in claim 33, wherein the generating further comprises executing a fault injection module on the test device.

35. A method as recited in claim 33, further comprising:

detecting the fault; and

responding to the fault in a tolerant manner.

36. A method as recited in claim 33, further comprising:

loading an error injection lock-out key into the fault tolerant device; and

based on the error injection lock-out key, enabling the simulating.