RESILIENCY TO MEMORY FAILURES IN COMPUTER SYSTEMS

• US 20130311823A1
• Filed: 05/21/2012
• Published: 11/21/2013
• Est. Priority Date: 05/21/2012
• Status: Active Grant

• Current Assignee: CRAY Incorporated (Hewlett-Packard Enterprise Company)
• Original Assignee: Preston P. Briggs, III
• Inventors: Kaplan, Laurence S., Briggs, Preston P. III, Ohlrich, Miles A., Leslie, Willard H.
• Granted Patent: US 9,535,804 B2
• Time in Patent Office: Days
• US Class Current: 714/15
• CPC Class Codes: G06F 11/08 Error detection or correction by redundancy in data representation, e.g. by using checking codes
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/1008 in individual solid state devices G06F11/1004 takes precedence
• CPC Class Codes: G06F 11/1016 Error in accessing a memory location, i.e. addressing error
• CPC Class Codes: G06F 11/1068 in sector programmable memories, e.g. flash disk G06F11/1072 takes precedence
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1405 at machine instruction level
• CPC Class Codes: G06F 11/141 for bus or memory accesses
• CPC Class Codes: G06F 11/1479 Generic software techniques for error detection or fault masking
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 : Management of blocks
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0673 : Single storage device

Abstract

A resiliency system detects and corrects memory errors reported by a memory system of a computing system using previously stored error correction information. When a program stores data into a memory location, the resiliency system executing on the computing system generates and stores error correction information. When the program then executes a load instruction to retrieve the data from the memory location, the load instruction completes normally if there is no memory error. If, however, there is a memory error, the computing system passes control to the resiliency system (e.g., via a trap) to handle the memory error. The resiliency system retrieves the error correction information for the memory location and re-creates the data of the memory location. The resiliency system stores the data as if the load instruction had completed normally and passes control to the next instruction of the program.

Claims

1. A method in a computing system for correcting memory errors reported by a memory system, the method comprising:

executing a load instruction of a processor of the computing system to load a data word from an address of the memory system;

when a memory error does not occur during executing of the load instruction, providing the data word loaded from the memory as the result of the load instruction;

when a memory error does occur during execution of the load instruction,

retrieving error correction information associated with the address;

re-creating from the error correction information for the data word for the address of the memory; and

providing the re-created data word as a result of the load instruction

wherein the executing of the load instruction, when a memory error does not occur, needs no additional overhead.

2. The method of claim 1 wherein the memory error is reported as uncorrectable by the memory system.

3. The method of claim 1 including when storing a data word at the address of the memory,

retrieving error correction information associated with the address,

regenerating the error correction information based on the data word to be stored;

storing the regenerated error correction information; and

storing the data word at the address of the memory.

4. The method of claim 1 wherein the memory is a distributed memory that is stored at multiple nodes of the computing system and the error correction information for a data word stored in a first memory of a first node includes other data words stored in the memory of other nodes.

5. The method of claim 4 wherein when the first node fails, re-creating at a second node the data word of the first node from the error correction information for that data word stored at the other nodes.

6. The method of claim 5 including notifying the other nodes that the data word that was previously stored at the first node is now stored at the second node.

7. The method of claim 1 wherein when the memory system loads a cache line that includes the data word, but does not identify the specific data word of the cache line that caused the memory error, ensuring that each of the data words of the cache line has error correction information that is not dependent on the other data words of the cache line.

8. The method of claim 1 wherein the error correction information is based on an erasure code that is a systematic code.

9. The method of claim 1 wherein the error correction information is based on an erasure code that is a maximal distance separable code.

10. The method of claim 1 wherein when a memory error occurs during a load instruction of a second data word while re-creating the data word, recursively re-creating the second data word.

11. A computing system that provides resilient access to memory, the computing system comprising:

a memory system having a memory storing data words at addresses and providing load and store access to the memory, wherein the memory system signals a memory error when a load access fails;

a processor with an instruction set that includes a load instruction; and

a storage medium that includes instructions of;

a re-create data word component that re-creates the data word of the memory using error correction information that includes at least one other data word of the memory and at least one correction word;

such that the processor executes a load instruction of an application program for loading a data word by

issuing to the memory system a load request specifying the address of the data word to load;

when a memory error is not signaled, providing the data word provided by the memory system as the loaded data word; and

when a memory error is signaled, executing the re-create data word component and providing the re-created data word as the loaded data word.

12. The computing system of claim 11 wherein the re-creating of the data word is transparent to the application program.

13. The computing system of claim 11 wherein the storage medium further includes instructions of a store component for storing a data word in the memory, and wherein the store component generates error correction information for the data word, stores the error correction information, and executes a store instruction to store the data word in memory.

14. The computing system of claim 13 wherein the application program invokes the store component to store a data word in the memory.

15. The computing system of claim 13 wherein the re-create data word component invokes the store component to store the re-created data word in the memory.

16. The computing system of claim 11 wherein the error correction information is based on a two-dimensional check word code.

17. The computing system of claim 11 wherein the providing of resilient access to the memory incurs no additional overhead when loading from the memory without a memory error in comparison to not providing such resilient access.

18. The computing system of claim 11 wherein the error correction information is stored in the memory.

19. A computer-readable storage medium containing computer-executable instructions of an application program interface for providing resiliency to memory accesses of an application program, the instructions comprising:

a segment register component that registers a segment of memory that is to be resilient for the application program, the registered segment having a segment descriptor indicating number of data words, number of check words, size of a check group, location of the data words, and location of the check words;

a segment reference component that maps the registered segment into the address space of the application program and registers a re-create data word component to process memory errors that occur when the application program accesses the registered segment; and

a segment write component that stores a data word in the registered segment by generating a check word for the data word, storing the generated check word, and storing the data word in the registered segment.

20. The computer-readable storage medium of claim 19 wherein the instructions further include a segment load component that generates a segment descriptor for a process of the application program other than the process that registered the registered segment.

21. The computer-readable storage medium of claim 19 wherein the instructions further include a memory copy component that stores multiple data words in the registered segment by generating check words for the multiple data words, storing the check words, and storing the multiple data words in the registered segment.

22. The computer-readable storage medium of claim 19 wherein the instructions further include a user error correction function component that registers a user error correction function for re-creating a data word when a memory error occurs when accessing the registered segment and for storing a data word in the registered segment wherein the re-create data word component and the segment write component use the registered user error correction function to provide resiliency.

23. A method in a replacement node for reconstructing a resilient segment of a first node of a network of nodes with a distributed memory, the first node hosting a portion of the resilient segment, the method comprising:

receiving by the replacement node an indication that the first node is to be replaced;

for each data word of the resilient segment,

collecting by the replacement node from nodes other than the first node error correction information sufficient to re-create that data word, the error correction information for a data word of the first node being stored at nodes other than the first node;

re-creating that data word based on the collected error correction information; and

storing the re-created data word in a replacement resilient segment of the replacement node;

for each check word stored in memory of the first node, re-creating and storing by the replacement node that check word; and

notifying the nodes of the network that the replacement node has replaced the first node.

24. The method of claim 23 wherein the resilient segment is to be replaced when the first node fails.

25. The method of claim 24 wherein when another node receives a memory error when attempting to access a data word of the resilient segment after the first node, the other node independently re-creates the data word from error correction information stored at nodes other than the first node.

26. A method in a computing system for storing data words and check words for providing resiliency in a memory, the method comprising:

storing data words in the memory, each data word being stored at an address within the memory; and

for each check unit of data words,

generating a check word for that check unit;

generating a check word address for storing the generated check word for that check unit such that the same check word address can be regenerated from each of the addresses of the data words of that check unit and cannot be regenerated from any other addresses of data words; and

storing the generated check word at the generated check word address within the memory.

27. The method of claim 26 wherein the memory provides a cache line of data words per access and a non-specific memory error signal covering multiple data words of the cache line and wherein each of the multiple data words covered by the same non-specific memory error signal is in a different check group.

28. The method of claim 26 wherein the check units form a two-dimensional check group.

29. The method of claim 26 wherein the memory is distributed across multiple nodes and wherein at least two of the data words of a check unit are stored at different nodes.

30. The method of claim 26 wherein the memory is distributed across multiple nodes and wherein each of the data words of a check unit is stored at a different node.

31. A method in a computing system for storing data words and check words to support resiliency of a memory, the memory providing a cache line of data words per access and a non-specific memory error signal covering multiple data words of the cache line, the method comprising:

storing the data words in the memory; and

for each check unit of error correction information,

generating a check word for that check unit such that each of the multiple data words covered by the same non-specific memory error signal is in a different check unit; and

storing the generated check word.

32. The method of claim 31 wherein the non-specific memory signal covers the entire cache line.

33. The method of claim 31 wherein the non-specific memory signal covers half of the cache line.

34. A method in a computing system for storing data words and check words to support resiliency of a memory, the memory being distributed across multiple nodes, the method comprising:

storing data words in the memory; and

for each check unit of error correction information,

generating a check word for that check unit such that at least two of the data words of that check unit are stored at different nodes; and

storing the generated check word.

35. The method of claim 34 wherein each of the data words of a check group is stored at a different node.

36. The method of claim 34 wherein the check units are organized into check groups and a resilient segment is associated with multiple check groups.

37. A method in a computing system for adapting an application program to access a resilient segment of the application program, the method comprising:

modifying the application program code to invoke a register component of a resiliency system to register a data structure of the application program as a resilient segment; and

modifying the application program to invoke a store data word component of the resiliency system to store data words in the resilient segment, rather than directly by the application program executing a store instruction,

38. The method of claim 37 wherein the modifying of the application program is performed by a compiler.

39. The method of claim 38 wherein a compiler directive specifies the data structure that is to be a resilient segment.