Selective automated power cycling of faulty disk in intelligent disk array enclosure for error recovery

• US 6,959,399 B2
• Filed: 09/24/2001
• Issued: 10/25/2005
• Est. Priority Date: 09/24/2001
• Status: Active Grant

• Current Assignee: Xyratex Technology Limited (Seagate Technology Holdings Plc)
• Original Assignee: International Business Machines Corporation
• Inventors: Chu, Davis Qi-Yu, King, Allen
• Primary Examiner(s): BADERMAN, SCOTT T
• Application Number:
  US09/961,950
• Publication Number:
  US 20030093721A1
• Time in Patent Office: 1,492 Days
• Field of Search: 714/6, 714/7, 714/23, 714/42, 714/36, 711/114, 713/324, 713/330
• US Class Current: 714/6.12
• CPC Class Codes: G11B 25/043 using rotating discs

Abstract

A disk array storage system and error recovery method wherein recovery from disk errors is achieved using automated selective power cycling. Initially, identification is made of a faulty disk drive in the array that exhibits an error condition in which the drive fails to perform a requested operation. The faulty disk drive is selectively power cycled while power to other disk drives in the array is maintained. Following the power cycling sequence, the requested operation is retried.

Claims

1. An error recovery method for recovering from disk errors in a disk array storage system having an array of disk drives adapted to store data, the method comprising the steps of:

identifying a faulty disk drive in said array that exhibits an error condition in which said drive fails to perform a requested operation;

determining whether automated selective disk drive power cycling is supported;

upon said determining producing a negative result, marking said faulty disk drive offline;

upon said determining producing a positive result, selectively power cycling said disk drive while maintaining power to other disk drives in said array; and

retrying said requested operation following said power cycling sequence.

2. A method in accordance with claim 1 wherein said power cycling sequence includes removing and then restoring power at a power input to said disk drive.

3. A method in accordance with claim 1 wherein said power cycling sequence includes removing and then restoring power to one of at least two power inputs to said disk drive.

4. A method in accordance with claim 1 wherein said power cycling sequence is implemented in part by a disk drive host adapter.

5. A method in accordance with claim 1 wherein said power cycling sequence is implemented in part by a processor that controls power supplied to said disk drive.

6. A method in accordance with claim 1 wherein said power cycling sequence is implemented by a disk drive host adapter and a processor that controls power supplied to said disk drive.

7. A method in accordance with claim 6 wherein said determining includes said host adapter querying said processor to determine whether said processor supports said power cycling sequence.

8. A method in accordance with claim 6 wherein said power cycling sequence includes said host adapter querying said processor to determine the status of said disk drive prior to issuing a power cycling command to said processor.

9. A method in accordance with claim 6 wherein said power cycling sequence includes said host adapter issuing a disk drive power off command to said processor followed by a disk drive power on command.

10. A method in accordance with claim 6 wherein said determining and said power cycling sequence collectively include:

a feature support inquiry sent by said host adapter to said processor to determine whether said processor supports said power cycling sequence;

a first disk drive status inquiry sent by said host adapter to said processor to determine the presence of said disk drive;

a power off command sent by said host adapter to said processor to power off said disk drive;

a second disk drive status inquiry sent by said host adapter to said processor to determine whether said disk drive has been powered off;

a power on command sent by said host adapter to said processor to power on said disk drive; and

a third disk drive status inquiry sent by said host adapter to said processor to determine whether said disk drive has been powered on.

11. A method in accordance with claim 6 wherein said host adapter is a RAID controller and said processor is a SEP device.

12. A method in accordance with claim 1 further including said disk drive being identified by virtue of requesting said disk drive to perform said requested operation multiple times without success.

13. A method in accordance with claim 1 further including determining whether said disk drive is a candidate for said power cycling sequence by considering whether said error condition is due to circumstances that cannot be corrected by power cycling.

14. A method in accordance with claim 1 wherein said determining includes determining whether a processor controlling power supplied to said disk drive supports said power cycling sequence.

15. A method in accordance with claim 1 further including determining whether a cumulative power cycle threshold for said disk drive has been reached representing said disk drive having undergone repeated power cycling over plural power cycling sessions.

16. A method in accordance with claim 1 further including determining whether a power cycle retry threshold for said disk drive has been reached representing said disk drive having undergone repeated power cycling in a current power cycling session.

17. A method in accordance with claim 16 further including resetting a power cycle retry count if said retry of said requested operation is successful.

18. A method in accordance with claim 16 further including retrying said power cycling sequence as necessary until said power retry threshold has been reached.

19. A method in accordance with claim 1 wherein said method is performed prior to said disk drive being designated offline.

20. A method in accordance with claim 1 wherein said method is performed using a SAF-TE INQUIRY command, a SAF-TE READ BUFFER command configured to implement a SEP Read Device Slot Status command, and SAF-TE WRITE BUFFER command configured to implement a modified SEP Perform Slot Operation command containing a disk drive power on Flag, a disk drive power off Flag, and an Identify Flag for providing an indication that said disk drive is in a power off state.

38. A storage system in accordance with claim 16 further including means for retrying said power cycling sequence as necessary until said power retry threshold has been reached.

21. A disk array storage system having an array of disk drives adapted to store data, said system comprising:

means for identifying a faulty disk drive in said array that exhibits an error condition in which said drive fails to perform a requested operation;

means for determining whether automated selective disk drive power cycling is supported;

means responsive to said determining producing a negative result for marking said faulty disk drive offline;

means responsive to said determining producing a positive result for selectively power cycling said disk drive while maintaining power to other disk drives in said array; and

means for retrying said requested operation following said power cycling sequence.

22. A storage system in accordance with claim 21 wherein said power cycling means includes means for removing and then restoring power at a power input to said disk drive.

23. A storage system in accordance with claim 21 wherein said power cycling means includes means for removing and then restoring power to one of at least two power inputs to said disk drive.

24. A storage system in accordance with claim 21 wherein said power cycling means is implemented in part by a disk drive host adapter.

25. A storage system in accordance with claim 21 wherein said power cycling means is implemented in part by a processor that controls power supplied to said disk drive.

26. A storage system in accordance with claim 21 wherein said power cycling means is implemented by a disk drive host adapter and a processor that controls power supplied to said disk drive.

27. A storage system in accordance with claim 26 wherein said determining means includes means in said host adapter for querying said processor to determine whether said processor supports said power cycling sequence.

28. A storage system in accordance with claim 26 wherein said power cycling means includes means in said host adapter for querying said processor to determine the presence of said disk drive prior to issuing a power cycling command to said processor.

29. A storage system in accordance with claim 26 wherein said power cycling means includes means in said host adapter for issuing a disk drive power off command to said processor followed by a disk drive power on command.

30. A storage system in accordance with claim 26 wherein said determining means and said power cycling means collectively include:

means for sending a feature support inquiry from said host adapter to said processor to determine whether said processor supports said power cycling sequence;

means for sending a first disk drive status inquiry from said host adapter to said processor to determine the status of said disk drive;

means for sending a power off command sent from said host adapter to said processor to power off said disk drive;

means for sending a second disk drive status inquiry from said host adapter to said processor to determine whether said disk drive has been powered off;

means for sending a power on command from said host adapter to said processor to power on said disk drive; and

means for sending a third disk drive status inquiry from said host adapter to said processor to determine whether said disk drive has been powered on.

31. A storage system in accordance with claim 26 wherein said host adapter is a RAID controller and said processor is a SEP device.

32. A storage system in accordance with claim 21 wherein said identifying means identifies said disk drive by virtue of requesting said disk drive to perform said requested operation multiple times without success.

33. A storage system in accordance with claim 21 further including means for determining whether said disk drive is a candidate for said power cycling sequence by considering whether said error condition is due to circumstances that cannot be corrected by power cycling.

34. A storage system in accordance with claim 21 wherein said determining means includes means for determining whether a processor controlling power supplied to said disk drive supports said power cycling sequence.

35. A storage system in accordance with claim 21 further including means for determining whether a cumulative power cycle threshold for said disk drive has been reached representing said disk drive having undergone repeated power cycling over plural power cycling session.

36. A storage system in accordance with claim 21 further including means for determining whether a power cycle retry threshold for said disk drive has been reached representing said disk drive having undergone repeated power cycling in a current power cycling session.

37. A storage system in accordance with claim 36 further including means for resetting a power cycle retry count if said retry of said requested operation is successful.

39. A storage system in accordance with claim 21 wherein said power cycling of said disk drive is performed prior to said disk drive being designated offline.

40. A storage system in accordance with claim 21 wherein said power cycling means utilizes a SAF-TE INQUIRY command, a SAF-TE READ BUFFER command configured to implement a SEP Read Device Slot Status command, and SAF-TE WRITE BUFFER command implement a modified SEP Perform Slot Operation command containing a disk drive power on Flag, a disk drive power off Flag, and an Identify Flag for providing an indication that said disk drive is in a power off state.

41. A computer program product for performing error recovery in a disk array storage system having an array of disk drives adapted to store data, comprising:

one or more data storage media for holding program instructions;

program instruction means stored on said media for identifying a faulty disk drive in said array that exhibits an error condition in which said drive fails to perform a requested operation;

program instruction means recorded on said media for determining whether automated selective disk drive power cycling is supported;

program instruction means recorded on said media responsive to said determining producing a negative result for marking said faulty disk drive offline;

program instruction means stored on said media responsive to said determining producing a positive result for selectively power cycling said disk drive while maintaining power to other disk drives in said array; and

program instruction means stored on said media for retrying said requested operation following said power cycling sequence.

42. A host adapter adapted for communication with a host computer and managing at least one intelligent enclosure containing an enclosure processor and an array of disk drives, the processor being configured to independently control power supplied to said disk drives, said host adapter being configured to perform an error recovery procedure comprising the steps of:

attempting one or more times without success to have a disk drive in the intelligent enclosure perform a requested operation;

determining whether said disk drive is a candidate for automated selective disk drive power cycling;

querying said enclosure processor to determine whether said enclosure processor supports automated selective disk drive power cycling;

upon said determining producing a negative result, marking said faulty disk drive offline;

upon said determining producing a positive result, determining whether a cumulative power cycling threshold for said disk drive has been reached;

determining whether a power cycling threshold for said disk drive has been reached;

issuing a first read command to said enclosure processor to verify that said disk drive is in a power on state;

issuing a first write command to said enclosure processor to cycle said disk drive to a power off state while maintaining other disk drives in said enclosure in a power on state;

issuing a second read command to said enclosure processor to verify that said disk drive is in a power off state;

issuing a second write command to said enclosure processor to cycle said disk drive to a power on state;

issuing a third read command to said enclosure processor to verify that said disk drive is in a power on state; and

re-attempting to have said disk drive perform said requested operation.

43. An intelligent enclosure adapted for communication with a host adapter and containing an enclosure processor controlling power to an array of disk drives, said intelligent enclosure being configured to perform an error recovery procedure comprising the steps of:

attempting one or more times without success to have a disk drive in the intelligent enclosure perform a requested operation;

responding to a query from said host adapter seeking to determine whether said enclosure processor supports automated selective disk drive power cycling;

upon said response being negative, advising said host adaptor of said status so that said disk drive can be marked offline by said host adaptor;

upon said response being positive, responding to a first read command from said host adapter seeking to verify that said disk drive is in a power on state;

responding to a first write command from said host adapter seeking to cycle said disk drive to a power off state while maintaining other disk drives in said enclosure in a power on state;

responding to a second read command from said host adapter seeking to verify that said disk drive is in a power off state;

responding to a second write command from said adapter seeking to cycle said disk drive to a power on state; and

responding to a third read command from said adapter seeking to verify that said disk drive is in a power on state.