Accessing multiple non-volatile semiconductor memory modules in an uneven manner

• US 7,865,761 B1
• Filed: 06/28/2007
• Issued: 01/04/2011
• Est. Priority Date: 06/28/2007
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Chilton, Kendell
• Primary Examiner(s): KIM, HONG CHONG
• Application Number:
  US11/823,499
• Time in Patent Office: 1,286 Days
• Field of Search: 711/103, 711/118, 711/114, 711/143, 711/5, 714/2, 714/6, 714/7
• US Class Current: 714/2
• CPC Class Codes: G06F 12/023 Free address space management
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 13/1668 Details of memory controller
• CPC Class Codes: G06F 2212/7208 Multiple device management, e.g. distributing data over multiple flash devices
• CPC Class Codes: G06F 2212/7211 Wear leveling
• CPC Class Codes: G11C 16/3495 Circuits or methods to detect or delay wearout of nonvolatile EPROM or EEPROM memory devices, e.g. by counting numbers of erase or reprogram cycles, by using multiple memory areas serially or cyclically

Abstract

A data storage apparatus (e.g., a flash memory appliance) includes a set of memory modules, an interface and a main controller coupled to the each memory module and to the interface. Each memory module has non-volatile semiconductor memory (e.g., flash memory). The interface is arranged to communicate with a set of external devices. The main controller is arranged to (i) store data within and (ii) retrieve data from the non-volatile semiconductor memory of the set of memory modules in an uneven manner on behalf of the set of external devices to unevenly wear out the memory modules over time. Due to the ability of the data storage apparatus to utilize each memory module through its maximum life and to stagger the failures of the modules, such a data storage apparatus is well-suited as a high availability storage device, e.g., a semiconductor cache for a fault tolerant data storage system.

Claims

1. A data storage apparatus, comprising:

a set of memory modules, each memory module of the set of memory modules having non-volatile semiconductor memory;

an interface arranged to communicate with a set of external devices; and

a main controller coupled to each memory module and to the interface, the main controller being arranged to (i) store data within and (ii) retrieve data from the non-volatile semiconductor memory of the set of memory modules in an uneven manner on behalf of the set of external devices to unevenly wear out the memory modules over time;

wherein the each memory module has, as the non-volatile semiconductor memory, flash memory;

wherein the main controller, when storing data within and retrieving data from the non-volatile semiconductor memory, is arranged to carry out data storage operations on a memory space formed by the flash memory; and

wherein the main controller is arranged to;

implement a data protection scheme which utilizes striping and parity across multiple memory modules of the set of memory modules, the multiple memory modules including a particular memory module, and

replace the particular memory module with a previously unused hot standby memory module when the current storage capacity value of the flash memory of the particular memory module transitions from being greater than a predefined capacity threshold to no longer being greater than the predefined capacity threshold.

2. A data storage apparatus as in claim 1 wherein the set of memory modules includes (i) a most-written memory module as the particular memory module, and (ii) remaining memory modules which have been written with data less often than the most-written memory module;

and

wherein, prior to replacement of the particular memory module, the main controller is arranged to write new data from the set of external devices to the most-written memory module more often than to the remaining memory modules.

3. A data storage apparatus as in claim 2 wherein each memory module further has a control circuit coupled to the flash memory of that memory module, the control circuit being arranged to update a current storage capacity value of the flash memory of that memory module over time.

4. A data storage apparatus as in claim 3 wherein the main controller is further arranged to:

query the current storage capacity value of the flash memory of the particular memory module; and

treat the particular memory module as (i) an operable memory module when the current storage capacity value of the flash memory of the particular memory module exceeds the predefined capacity threshold, and (ii) a non-operable memory module when the current storage capacity value of the flash memory of the particular memory module does not exceed the predefined capacity threshold.

5. A data storage apparatus as in claim 4 wherein the main controller is further arranged to:

maintain a mapping of a logical memory space to the memory space formed by the flash memory; and

re-map the logical memory space in response to replacement of the particular memory module with the previously unused hot standby memory module.

6. A data storage apparatus as in claim 4 wherein the main controller is further arranged to:

monitor power consumption and temperature of the set of memory modules; and

provide a warning signal when at least one of (i) the power consumption of the set of memory modules falls outside a predetermined power range and (ii) the temperature of the set of memory modules falls outside a predetermined temperature range.

7. A data storage apparatus as in claim 4 wherein the main controller is further arranged to:

manage at least a portion of the memory space formed by the flash memory as a cache to a set of magnetic disk drives in accordance with a write-back caching scheme.

8. A data storage apparatus as in claim 2 wherein replacement of the particular memory module with the previously unused hot standby memory module includes decoupling the particular memory module from the main controller while maintaining coupling between the main controller and the remaining memory modules.

9. A data storage apparatus as in claim 1, further comprising:

another main controller coupled to each memory module, the other main controller being arranged to (i) store data within and (ii) retrieve data from the flash memory of the set of memory modules in an uneven manner on behalf of the set of external devices to unevenly wear out the memory modules over time.

10. A data storage apparatus as in claim 9, further comprising:

a rack-mount enclosure defining a front opening and a rear opening, each memory module being arranged to install within the rack-mount enclosure through the front opening, each main controller being arranged to install within the rack-mount enclosure through the rear opening; and

wherein operation of at least one main controller in combination with at least one memory module within the rack-mount enclosure forms a flash storage appliance.

11. A data storage apparatus as in claim 10 wherein each main controller and each memory module is arranged as a hot-pluggable field replaceable unit to enable the flash storage appliance to remain operational during replacement of any module.

12. A data storage apparatus as in claim 1 wherein the main controller is arranged to:

monitor a wear characteristic of each memory module of the set of memory modules, the wear characteristic identifying an amount of wear for that memory module.

13. A data storage apparatus as in claim 12 wherein the set of memory modules includes other memory modules, the particular memory module currently being the most-written memory module of the set of memory modules;

and wherein the main controller is further arranged to;

receive new data to be stored in the set of memory modules from the set of external devices; and

store the new data more often on the particular memory module of the set of memory modules in response to the particular memory module having a wear characteristic indicating that the particular memory module currently is the most-written memory module of the set of memory modules.

14. A data storage apparatus as in claim 13 wherein the main controller is further arranged to:

monitor replacement of the particular memory module with the previously unused hot standby memory module, the previously unused hot standby memory module having a wear characteristic indicating that the previously unused hot standby memory module is not the most-written memory module of the set of memory modules;

receive additional new data to be stored in the set of memory modules from the set of external devices; and

store the additional new data more often on another memory module of the set of memory modules in response to the other memory module having a wear characteristic indicating that the other memory module is now the most-written memory module of the set of memory modules following replacement of the particular memory module.

15. A method of operating a data storage apparatus having a set of memory modules, each memory module having non-volatile semiconductor memory, the method comprising:

establishing communications with a set of external devices;

receiving data access requests from the set of external devices; and

in response to the data access requests, storing data within and retrieving data from the non-volatile semiconductor memory of the set of memory modules in an uneven manner on behalf of the set of external devices to unevenly wear out the memory modules over time;

wherein the each memory module has, as the non-volatile semiconductor memory, flash memory;

wherein storing the data within and retrieving the data from the non-volatile semiconductor memory, includes carrying out data storage operations on a memory space formed by the flash memory; and

wherein the method further comprises;

implementing a data protection scheme which utilizes striping and parity across multiple memory modules of the set of memory modules, the multiple memory modules including a particular memory module, and

replacing the particular memory module with a previously unused hot standby memory module when the current storage capacity value of the flash memory of the particular memory module transitions from being greater than a predefined capacity threshold to no longer being greater than the predefined capacity threshold.

16. A method as in claim 15 wherein the set of memory modules includes other memory modules, the particular memory module currently being the most-written memory module of the set of memory modules;

and wherein the method further comprises;

writing new data from the set of external devices to the particular memory module more often than to the other memory modules in response to the particular memory module currently being the most-written memory module of the set of memory modules.

17. A method as in claim 16, further comprising:

monitoring replacement of the particular memory module with the previously unused hot standby memory module, the previously unused hot standby memory module currently not being the most-written memory module of the set of memory modules; and

writing additional new data from the set of external devices more often to one of the other memory modules in response to the one of the other memory modules currently being the most-written memory module of the set of memory modules following replacement of the particular memory module.

18. A method as in claim 16 wherein replacing the particular memory module with the previously unused hot standby memory module includes:

decoupling the particular memory module from a main controller while maintaining coupling between the main controller and the other memory modules.

19. A method as in claim 15, further comprising:

querying a current storage capacity value of the flash memory of the particular memory module; and

treating the particular memory module as (i) an operable memory module when the current storage capacity value of the flash memory of the particular memory module exceeds the predefined capacity threshold, and (ii) a non-operable memory module when the current storage capacity value of the flash memory of the particular memory module does not exceed the predefined capacity threshold.

20. A method as in claim 15, further comprising:

managing at least a portion of the memory space formed by the flash memory as a cache to a set of magnetic disk drives in accordance with a write-back caching scheme.