System and method for interposition-based selective simulation of faults for access requests to a data storage system

• US 20070050686A1
• Filed: 09/01/2005
• Published: 03/01/2007
• Est. Priority Date: 09/01/2005
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Keeton, Kimberly, Merchant, Arif, Wilkes, John
• Granted Patent: US 7,467,333 B2
• Time in Patent Office: Days
• US Class Current: 714/48
• CPC Class Codes: G06F 11/261 by simulating additional hardware, e.g. fault simulation

Abstract

According to one embodiment, a method comprises intercepting, at an interposition agent, requests for accessing a data storage device. The method further comprises determining, by the interposition agent, at least one of the requests to impact, and selectively simulating, by the interposition agent, a fault for the selected at least one of the requests. According to another embodiment, a system comprises at least one data storage device, and at least one requester operable to request access to the at least one data storage device. The system further comprises at least one interposition agent communicatively interposed between the requestor(s) and the data storage device(s) to intercept requests for access from the requestor(s) to the data storage device(s), wherein the interposition agent is operable to selectively determine ones of the intercepted requests to impact and selectively simulate a fault for the selected at least one of the requests.

Claims

1. A method comprising:

intercepting, at an interposition agent, requests for accessing a data storage device;

determining, by said interposition agent, at least one of said requests to impact; and

selectively simulating, by said interposition agent, a fault for the selected at least one of said requests.

2. The method of claim 1 wherein the fault comprises at least one of the following:

SCSI timeout, power failure, firmware bug, intermittent offline period, cache failure, cache contention, disk array mirrored controller failure, malicious data corruption, malicious data deletion, accidental data corruption, accidental data deletion, snapshot system out of space, snapshot retention period expiration, failure of backup creation process, malicious backup deletion, accidental backup deletion, backup takes longer than allotted window of time, failure of tape media, failure of drive, backup process misconfiguration, failure of network link, out of space, logical volume misconfiguration, file system misconfiguration, file system aging, malicious file system data corruption, malicious file system data deletion, accidental file system data corruption, accidental file system data deletion, malicious file system metadata corruption, malicious file system metadata deletion, accidental file system metadata corruption, accidental file system metadata deletion, database management system server overload, storage system misconfiguration, malicious database management system data corruption, malicious database management system data deletion, accidental database management system data corruption, accidental database management system data deletion, malicious database management system log corruption, malicious database management system log deletion, accidental database management system log corruption, accidental database management system log deletion, malicious database management system catalog corruption, malicious database management system catalog deletion, accidental database management system catalog corruption, and accidental database management system catalog deletion.

3. The method of claim 2 wherein said selectively simulating comprises simulating at least one of the following manifestations of said fault:

request failure, request slowdown, read slowdown, write slowdown, read returns incorrect results, snapshot creation failure, snapshot restore failure, recovery request failure, backup creation request failure, write failure, write records incorrect data, write records correct data to incorrect location, logical volume creation failure, logical volume modification failure, and logical volume deletion failure.

4. The method of claim 1 wherein said selectively simulating comprises simulating a manifestation of said fault.

5. The method of claim 4 further comprising:

determining from a model said manifestation to simulate for said fault.

6. The method of claim 4 wherein said simulating said manifestation comprises simulating at least one of the following:

delayed response, no response, return corrupt data, loss of at least a portion of requested data, and returned error.

7. The method of claim 1 further comprising:

implementing the interposition agent communicatively interposed between an operating system layer and storage device layer.

8. The method of claim 1 wherein the data storage device comprises at least one selected from the group consisting of:

disk drive, disk array device, tape device, remote storage device, backup storage system, file system, database, and persistent non-volatile memory.

9. The method of claim 1 further comprising:

receiving campaign information by said interposition agent.

10. The method of claim 9 where said campaign information specifies at least one of the following:

type of said fault desired to be simulated, identification of the selected at least one of said requests for which said fault is to be simulated, frequency of said fault to be injected, duration of said fault, and scope of said fault to be simulated.

11. The method of claim 1 further comprising:

determining, by said interposition agent, an action to take for simulating said fault for the selected at least one of said requests.

12. The method of claim 11 wherein said determining comprises:

using, by said interposition agent, a model to determine said action to take for simulating said fault, where said model maps a plurality of different faults to respective manifestations.

13. The method of claim 1 further comprising:

selectively passing to said data storage device, by said interposition agent, ones of said requests that are not determined as requests to impact.

14. The method of claim 1 further comprising:

selectively not impacting certain ones of said requests by said interposition agent.

15. The method of claim 1 further comprising:

implementing said interposition agent communicatively interposed between at least one requestor making said requests and said data storage device.

16. The method of claim 15 wherein said implementing comprises:

implementing said interposition agent transparent to said at least one requestor.

17. The method of claim 16 further comprising:

implementing said interposition agent transparent to said data storage device.

18. The method of claim 1 further comprising:

evaluating behavior of at least one requestor making said selected at least one of said requests for which said fault is simulated.

19. The method of claim 1 further comprising:

said interposition agent maintaining an unmodified copy of requested data and a modified copy of said requested data.

20. The method of claim 19 wherein said selectively simulating said fault further comprises:

returning, by said interposition agent, said modified copy of said requested data for the selected at least one of said requests.

21. The method of claim 20 wherein said selectively simulating said fault further comprises:

returning, by said interposition agent, said unmodified copy of said requested data for requests that are not to be impacted.

22. The method of claim 1 further comprising using said selectively simulated fault for at least one of the following:

a) understanding response of a requestor to said fault, b) testing functionality of said requestor, c) tuning behavior of said requestor, d) understanding worst-case behavior of said requester, e) continual, proactive testing of said requester, f) tuning storage system design, and g) using measurements of recovery behavior to calibrate analytical models used for automated storage system design.

23. A method comprising:

determining by a controller, from a model, an error manifestation for a root cause fault desired to be simulated for a data storage device;

intercepting, by an interposition agent, requests for accessing said data storage device;

determining, by said interposition agent, at least one of the intercepted requests to impact with said error manifestation; and

selectively simulating, by said interposition agent, said root cause fault for the determined ones of the intercepted requests to impact by imposing said determined error manifestation on the determined ones of the intercepted requests to impact.

24. The method of claim 23 further comprising:

receiving information at said controller specifying said root cause fault desired.

25. The method of claim 23 further comprising:

receiving information at said interposition agent specifying said requests to impact.

26. The method of claim 23 further comprising:

receiving campaign information at said controller.

27. The method of claim 26 wherein said campaign information specifies at least one of the following:

type of said root cause fault desired to be simulated, identification of the requests to impact, frequency of said root cause fault to be injected, duration of said root cause fault, and scope of said root cause fault to be simulated.

28. A system comprising:

at least one data storage device;

at least one requestor operable to request access to said at least one data storage device; and

at least one interposition agent communicatively interposed between said at least one requestor and said at least one data storage device to intercept requests for access from said at least one requestor to said at least one data storage device, wherein said interposition agent is operable to selectively determine ones of said intercepted requests to impact and selectively simulate a fault for the selected at least one of said requests.

29. The system of claim 28 wherein said at least one interposition agent is transparent to said at least one requestor.

30. The system of claim 28 wherein said at least one interposition agent is transparent to said at least one data storage device.

31. The system of claim 28 wherein said at least one interposition agent is communicatively interposed between an operating system and said at least one data storage device.

32. The system of claim 28 wherein said at least one interposition agent is operable to selectively simulate said fault by simulating at least one manifestation of said fault.

33. The system of claim 28 further comprising:

at least one controller operable to determine an error manifestation for said fault.

34. The system of claim 33 wherein said at least one controller is integrated within said at least one interposition agent.

35. An interposition agent communicatively interposed between at least one requestor and a data storage device to intercept requests for access from said at least one requestor to said data storage device, said interposition agent comprising:

means for determining ones of the intercepted requests to impact, wherein not all of said intercepted requests are to be impacted; and

means for selectively simulating a fault for the determined ones of the intercepted requests to impact.

36. The interposition agent of claim 35 further comprising:

means for determining, from a failure model, an error manifestation for said fault, wherein said means for selectively simulating imposes the determined error manifestation on the determined ones of the intercepted requests to impact.

37. The interposition agent of claim 35 further comprising:

means for receiving an indication of a fault desired to be simulated.

38. The interposition agent of claim 35 further comprising:

means for receiving an indication of ones of intercepted requests desired to be impacted.

39. An interposition agent communicatively interposed between at least one requestor and at least one data storage device to intercept requests for access from said at least one requestor to said at least one data storage device, said interposition agent comprising:

controller operable to use a model to determine an error manifestation for a specified root cause fault for said at least one data storage device; and

simulator operable to simulate said determined error manifestation for said at least one data storage device for requests.

40. The interposition agent of claim 39 wherein said interposition agent is transparent to said at least one requester.

41. The interposition agent of claim 40 wherein said interposition agent is transparent to said at least one data storage device.

42. The interposition agent of claim 39 wherein said interposition agent is communicatively interposed between an operating system and said at least one data storage device.

43. The interposition agent of claim 39 wherein said model maps a plurality of different root cause faults to respective manifestations.

44. Computer-executable software code stored to a computer-readable media, said computer-executable software code comprising:

code for determining from a model an error manifestation for a root cause fault desired for a data storage device;

code for receiving intercepted requests for accessing a data storage device;

code for determining at least one of said intercepted requests to impact with said error manifestation; and

code for selectively simulating said root cause fault for the determined at least one of said intercepted requests by imposing said determined error manifestation on the determined at least one of said intercepted requests.

45. The computer-executable software code of claim 44 wherein said model maps a plurality of different root cause faults to respective manifestations.

46. The computer-executable software code of claim 44 further comprising:

code for receiving information specifying said root cause fault desired.

47. The computer-executable software code of claim 44 further comprising:

code for receiving information specifying said requests to impact.