Techniques for maintaining operation of data storage system during a failure

• US 7,293,198 B2
• Filed: 03/25/2004
• Issued: 11/06/2007
• Est. Priority Date: 03/25/2004
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Strickland, Stephen, Dorr, Timothy, Burroughs, John V.
• Primary Examiner(s): Baderman; Scott
• Assistant Examiner(s): BONURA, TIMOTHY M
• Application Number:
  US10/808,839
• Publication Number:
  US 20050223284A1
• Time in Patent Office: 1,321 Days
• Field of Search: 714/4, 714/5, 714/6, 714/15, 714/22, 714/51
• US Class Current: 714/15
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality

Abstract

A data storage system has a first storage processor, a second storage processor, and a communications subsystem. The communications subsystem has (i) an interfacing portion interconnected between the first storage processor and the second storage processor, (ii) a clock circuit coupled to the interfacing portion, and (iii) a controller coupled to the interfacing portion and the clock circuit. The controller is configured to enable operation of the interfacing portion to provide communications between the first and second storage processors, sense a failure within the clock circuit, and reset the interfacing portion in response to the sensed failure to enable one of the first and second storage processors to continue operation. Such resetting of the interfacing portion prevents the remaining storage processor from locking up, thus freeing that storage processor so that it is capable of continuing to operate even after the failure.

Claims

1. A data storage system, comprising:

a first storage processor;

a second storage processor; and

a communications subsystem having (i) an interfacing portion interconnected between the first storage processor and the second storage processor, (ii) a clock circuit coupled to the interfacing portion, and (iii) a controller coupled to the interfacing portion and the clock circuit, the controller being configured to;

enable operation of the interfacing portion to provide communications between the first and second storage processors;

sense a failure within the clock circuit; and

reset the interfacing portion in response to the sensed failure to enable one of the first and second storage processors to continue operation;

wherein the controller of the communications subsystem includes;

a watchdog stage which is configured to generate an error signal in response to loss of a clock signal from the clock circuit within a predetermined timeout period;

wherein the interfacing portion of the communications subsystem includes a first interface device coupled to the first storage processor, a second interface device coupled to the second storage processor, and a communications bus connecting the first and second interface devices together;

wherein the controller of the communications subsystem further includes;

an output stage coupled to the watchdog stage, the output stage being configured to provide a reset signal to the first interface device in response to the error signal, the reset signal enabling the second storage processor to continue operation;

wherein (i) the first interface device is disposed at one end of the communications bus and (ii) the second interface device is disposed at another end of the communications bus to form a communications pathway between the first and second storage processors; and

wherein the controller, when enabling operation of the interfacing portion, is configured to;

direct the first interface device coupled to the first storage processor and the second interface device coupled to the second storage processor to concurrently operate as communications end points of the communications pathway formed between the first and second storage processors to exchange cached data between the first and second storage processors through the first interface device, the second interface device and the communications bus.

2. The data storage system of claim 1 wherein the interfacing portion of the communications subsystem includes a Cache Mirroring Interface (CMI) bus.

3. The data storage system of claim 2 wherein the interfacing portion of the communications subsystem further includes:

a first interface device having a first PCI interface coupled to the first storage processor and a first CMI interface coupled to the CMI bus, and

a second interface device having a second PCI interface coupled to the second storage processor and a second CMI interface coupled to the CMI bus.

4. The data storage system of claim 1 wherein the interfacing portion of the communications subsystem includes:

a first interface coupled to the first storage processor;

a second interface coupled to the second storage processor; and

a switch coupled to the controller of the communications subsystem, the switch being disposed between the first and second interface.

5. The data storage system of claim 4 wherein the first storage processor receives power from a first power supply, wherein the second storage processor receives power from a second power supply, and wherein the controller of the communications subsystem is further configured to:

open the switch in response to loss of a power supply signal from one of the first and second power supplies.

6. A communications subsystem for a data storage system having a first storage processor and a second storage processor, the communications subsystem comprising:

an interfacing portion configured to interconnect the first storage processor with the second storage processor;

a clock circuit coupled to the interfacing portion; and

a controller coupled to the interfacing portion and the clock circuit, the controller being configured to;

enable operation of the interfacing portion to provide communications between the first and second storage processors;

sense a failure within the clock circuit; and

reset the interfacing portion in response to the sensed failure to enable one of the first and second storage processors to continue operation;

wherein the controller includes;

a watchdog stage which is configured to generate an error signal in response to loss of a clock signal from the clock circuit within a predetermined timeout period;

wherein the interfacing portion includes a first interface device configured to couple to the first storage processor, a second interface device configured to couple to the second storage processor, and a communications bus connecting the first and second interface devices together;

wherein the controller further includes;

an output stage coupled to the watchdog stage, the output stage being configured to provide a reset signal to the first interface device in response to the error signal, the reset signal enabling the second storage processor to continue operation;

wherein (i) the first interface device is disposed at one end of the communications bus and (ii) the second interface device is disposed at another end of the communications bus to form a communications pathway between the first and second storage processors; and

wherein the controller, when enabling operation of the interfacing portion, is configured to;

direct the first interface device coupled to the first storage processor and the second interface device coupled to the second storage processor to concurrently operate as communications end points of the communications pathway formed between the first and second storage processors to exchange cached data between the first and second storage processors through the first interface device, the second interface device and the communications bus.

7. The communications subsystem of claim 6 wherein the interfacing portion includes a Cache Mirroring Interface (CMI) bus.

8. The communications subsystem of claim 7 wherein the interfacing portion further includes:

a first interface device having a first PCI interface configured to couple to the first storage processor and a first CMI interface configured to couple to the CMI bus, and

a second interface device having a second PCI interface configured to couple to the second storage processor and a second CMI interface configured to couple to the CMI bus.

9. The communications subsystem of claim 6 wherein the interfacing portion of the communications subsystem includes:

a first interface coupled to the first storage processor;

a second interface coupled to the second storage processor; and

a switch coupled to the controller, the switch being disposed between the first and second interface.

10. The communications subsystem of claim 9 wherein the first storage processor receives power from a first power supply, wherein the second storage processor receives power from a second power supply, and wherein the controller is further configured to:

open the switch in response to loss of a power supply signal from one of the first and second power supplies.

11. In a data storage system having (i) a first storage processor, (ii) a second storage processor and (iii) a communications subsystem coupled to the first and second storage processors, a method for operating the data storage system during a failure within the communications subsystem, the method comprising:

while the first and second storage processors perform data storage operations, enabling operation of the communications subsystem to provide communications between the first and second storage processors;

sensing a failure within a critical portion of the communications subsystem; and

resetting an interfacing portion of the communications subsystem in response to the sensed failure to enable one of the first and second storage processors to continue operation;

wherein the communications subsystem is configured to exchange cached data for cache coherency between the first and second storage processors; and

wherein sensing the failure within the critical portion of the communications subsystem includes detecting a malfunction within the communication subsystem which prevents the communications subsystem from exchanging cached data for cache coherency between the first and second storage processors.

12. In a data storage system having (i) a first storage processor, (ii) a second storage processor and (iii) a communications subsystem coupled to the first and second storage processors, a method for operating the data storage system during a failure within the communications subsystem, the method comprising:

while the first and second storage processors perform data storage operations, enabling operation of the communications subsystem to provide communications between the first and second storage processors;

sensing a failure within a critical portion of the communications subsystem; and

resetting an interfacing portion of the communications subsystem in response to the sensed failure to enable one of the first and second storage processors to continue operation;

wherein the critical portion of the communications subsystem includes clock circuitry;

wherein sensing the failure includes;

generating an error signal in response to loss of a clock signal from the clock circuitry within a predetermined timeout period;

wherein the communications subsystem includes a first interface device coupled to the first storage processor, and a second interface device coupled to the second storage processor, the first and second interface devices being connected together through a communications bus;

wherein resetting the interfacing portion includes;

outputting a reset signal to the first interface device to enable the second storage processor to continue operation; and

wherein (i) the first interface device is disposed at one end of the communications bus and (ii) the second interface device is disposed at another end of the communications bus to form a communications pathway between the first and second storage processors; and

wherein enabling operation of the communications subsystem to provide the communications between the first and second storage processors includes;

directing the first interface device coupled to the first storage processor and the second interface device coupled to the second storage processor to concurrently operate as communications end points of the communications pathway formed between the first and second storage processors to exchange cached data between the first and second storage processors through the first interface device, the second interface device and the communications bus.