Testing components of a computerized storage network system having a storage unit with multiple controllers

• US 6,754,853 B1
• Filed: 12/07/2000
• Issued: 06/22/2004
• Est. Priority Date: 12/07/2000
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: LSI Logic Corporation (Broadcom, Inc.)
• Inventors: DeKoning, Rodney A., Binford, Charles D.
• Primary Examiner(s): Iqbal, Nadeem
• Assistant Examiner(s): BONURA, TIMOTHY M
• Application Number:
  US09/731,991
• Time in Patent Office: 1,293 Days
• Field of Search: 714/42, 714/43, 714/5, 714/55
• US Class Current: 714/42
• CPC Class Codes: G06F 11/2221 to test input/output devices or peripheral units

Abstract

An array controller of a data storage system initiates a test of another array controller of the data storage system to determine the operational condition of the controller under test (CUT) as well as an array of storage devices to which the CUT is connected and a network fabric over which the CUT receives commands from host devices of the data storage system. If the CUT or devices connected thereto are not functioning properly, the controller initiating the test can diagnose the problem. The controller initiating the test instructs the CUT to perform certain normal operating functions, e.g. data read and write functions, and checks whether the functions are completed correctly. Additionally, a loopback test checks the operation of the network fabric, and the read and write functions also check the operation of the storage devices.

Claims

1. A method for testing an operational condition of a first controller in a computerized data storage system having the first controller, a second controller and one or more storage devices, the storage devices having data stored thereon, and the first and second controllers controlling the storage devices to store and retrieve the data onto and from the storage devices, comprising the steps of:

sending a test command from the second controller to the first controller to cause the first controller to execute predetermined operating functions wherein the predetermined operating functions include functions to test operations involving the first controller accessing the storage devices and wherein the predetermined operating functions include functions to test operations involving the first controller accessing a SAN fabric coupling the first controller to a host system;

responding to the test command by the first controller attempting to perform the predetermined operating functions;

analyzing an outcome of the first controller attempting to perform the predetermined operating fictions;

determining from the analyzed outcome whether the first controller was successful in performing the predetermined operating functions; and

determining the operational condition of the first controller depending on the determination of whether the first controller was successful in performing the predetermined operating functions.

2. A method as defined in claim 1 comprising the further steps of:

attempting to access the storage devices by the first controller as part of the predetermined operating functions; and

determining whether the first controller was successful in attempting to access the storage devices.

3. A method as defined in claim 2 comprising the further steps of:

generating test data by the second controller;

writing the test data from the second controller to the storage devices;

attempting to read the test data by the first controller from the storage devices; and

determining whether the first controller was successful in attempting to read the test data from the storage devices.

4. A method as defined in claim 3 comprising the further steps of:

stopping accesses to the storage devices by the first controller when the first controller was not successful in reading the test data from the storage devices.

5. A method as defined in claim 3 comprising the further steps of:

generating test information by the second controller;

generating the test data from the test information;

passing the test information from the second controller to the first controller; and

comparing the test data with the test information to determine whether the first controller was successful in reading the test data from the storage devices.

6. A method as defined in claim 3 comprising the further steps of:

generating a time stamp by the second controller;

generating the test data from the time stamp;

passing the time stamp from the second controller to the first controller; and

comparing the test data with the time stamp to determine whether the first controller was successful in reading the test data from the storage devices.

7. A method as defined in claim 3 comprising the further step of:

issuing a read command from the first controller to itself prior to reading the test data.

8. A method as defined in claim 7 comprising the further steps of:

issuing the read command externally to the first controller;

returning the read command to the first controller; and

responding to the read command by the first controller reading the test data from the storage devices.

9. A method as defined in claim 8, wherein the computerized data storage system is part of a networked storage system which includes a network device connected to the computerized data storage system through the first controller, comprising the further step of:

attempting to send the read command from the first controller to the network device with instructions for the network device to send the read command back to the first controller.

10. A method as defined in claim 9 comprising the further step of:

determining whether the network device is operating depending on whether the network device sent the read command back to the first controller.

11. A method as defined in claim 7 comprising the further steps of:

issuing the read command internally to the first controller; and

responding to the read command by the first controller reading the test data from the storage devices.

12. A method as defined in claim 2 comprising the further steps of:

generating test data by the first controller;

attempting to write the test data by the first controller to the storage devices;

reading the test data by the second controller from the storage devices; and

determining by the second controller whether the first controller was successful in attempting to write the test data to the storage devices.

13. A method as defined in claim 12 comprising the further step of:

stopping accesses to the storage devices by the first controller when the first controller was not successful in attempting to write the test data to the storage devices.

14. A method as defined in claim 12 comprising the further steps of:

generating test information by the second controller;

passing the test information from the second controller to the first controller;

generating the test data from the test information;

comparing the test data read by the second controller to the test information generated by the second controller; and

determining by the second controller whether the first controller was successful in attempting to write the test data to the storage devices based upon the comparing of the test data to the test information.

15. A method as defined in claim 12 comprising the further steps of:

generating a time stamp by the second controller;

passing the time stamp from the second controller to the first controller;

generating the test data from the time stamp;

comparing the test data read by the second controller to the time stamp generated by the second controller; and

determining by the second controller whether the first controller was successful in attempting to write the test data to the storage devices based upon the comparing of the test data to the time stamp.

16. A method as defined in claim 12 comprising the further step of:

issuing a write command from the first controller to itself using the SAN fabric prior to writing the test data.

17. A method as defined in claim 16 comprising the further steps of:

issuing the write command externally to the first controller;

returning the write command to the first controller; and

responding to the write command by the first controller writing the test data to the storage devices.

18. A method as defined in claim 17, wherein the computerized data storage system is part of a networked storage system which includes a network device connected to the computerized data storage system through the first controller, comprising the further step of:

attempting to send the write command from the first controller to the network device with instructions for the network device to return the write command back to the first controller.

19. A method as defined in claim 18 comprising the further step of:

determining by the second controller whether the network device is operating depending on whether the network device return the write command to the first controller.

20. A method as defined in claim 16 comprising the further steps of:

issuing the write command internally to the first controller; and

responding to the write command by the first controller writing the test data to the storage devices.

21. A method as defined in claim 2 comprising the further step of:

determining by the second controller whether the storage devices are operating depending on whether the first controller was successful in attempting to access the storage devices.

22. A method as defined in claim 1 comprising the further steps of:

attempting to detect by the first controller any first errors occurring under the test;

sending a test response, including information regarding any first errors, from the first controller to the second controller;

attempting to detect by the second controller any second errors occurring under the test; and

determining by the second controller the operational condition of the first controller depending on any first and second errors that were detected.

23. A method as defined in claim 1 comprising the further steps of:

resetting the first controller upon determining that the first controller was not successful in performing the predetermined operating functions; and

repeating the previous steps at least once.

24. A storage array for servicing data access requests received from host devices through a SAN network, comprising:

an array of storage devices containing data on behalf of the host devices;

a first array controller connected to the network and to the array of storage devices to receive first data access requests from the host devices and to access the data contained on the array of storage devices in accordance with the first data access requests; and

a second array controller connected to the network, the first array controller and the array of storage devices to receive second data access requests from the host devices, to access the data contained on the array of storage devices in accordance with the second data access requests and to exchange a test command with the first array controller; and

a memory connected to the first and second array controllers and containing firmware instructions for causing the first and second array controllers to perform the following steps;

initiating by the second array controller a test of the first array controller to determine an operational condition of the first array controller;

sending the test command from the second array controller to the first array controller to cause the first array controller to perform predetermined operating functions wherein the predetermined operating functions include functions to test operations involving the first controller accessing the storage devices and wherein the predetermined operating functions include functions to test operations involving the first controller accessing a SAN fabric coupling the first controller to the host devices;

attempting to perform by the first array controller the predetermined operating functions;

determining by the second array controller whether the first array controller was successful in attempting to perform the predetermined operating functions; and

determining by the second array controller the operational condition of the first array controller depending on the determination of whether the first array controller was successful in attempting to perform the predetermined operating functions.

25. A storage array as defined in claim 24 wherein:

the firmware instructions further cause the first and second array controller to perform a test of a read data function of the first array controller in which the second array controller generates test data and writes the test data to the array of storage devices and the first array controller reads the test data from the array of storage devices and detects whether the test data is correct.

26. A storage array as defined in claim 25 wherein:

the firmware instructions further cause the second array controller to generate test information, generate the test data from the test information and pass the test information to the first array controller and further cause the first array controller to detect whether the test data read by the first array controller is correct based on the test information.

27. A storage array as defined in claim 26 wherein:

the firmware instructions further cause the first array controller to send a test response to the second array controller indicating whether the test data is detected as being correct and further cause the second array controller to determine the operational condition of the first array controller and the array of storage devices depending on the indication in the test response of whether the test data is detected as being correct by the first array controller.

28. A storage array as defined in claim 24 wherein:

the firmware instructions further cause the first and second array controller to perform a test of a write data function of the first array controller in which the first array controller generates test data and writes the test data to the array of storage devices and the second array controller reads the test data from the array of storage devices and detects whether the test data is correct.

29. A storage array as defined in claim 28 wherein:

the firmware instructions further cause the second array controller to generate test information, to pass the test information to the first array controller and to detect whether the test data written by the first array controller is correct based on the test information and further cause the first array controller to generate the test data from the test information.

30. A storage array as defined in claim 29 wherein:

the firmware instructions further cause the second array controller to determine the operational condition of the first array controller and the array of storage devices depending on the detection of whether the test data written by the first array controller is correct.

31. A storage array as defined in claim 28 wherein:

the firmware instructions further cause the first array controller to issue a write command to itself to perform the write data function under the test.

32. A storage array as defined in claim 31 wherein:

the firmware instructions further cause the first array controller to send the write command to the network with instructions for the network to return the write command to the first array controller and further cause the first array controller to respond when it receives the write command by performing the write data function.