Restoring distributed shared memory data consistency within a recovery process from a cluster node failure

• US 9,063,887 B2
• Filed: 03/13/2013
• Issued: 06/23/2015
• Est. Priority Date: 11/19/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Aronovich, Lior, Levy, Asaf, Loya, Liran
• Primary Examiner(s): Chau, Dung K
• Application Number:
  US13/801,797
• Publication Number:
  US 20130232379A1
• Time in Patent Office: 832 Days
• Field of Search: 707/769, 707/770, 707/999.01, 707/999.102, 709/213, 709/226, 709/205
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2043 where the redundant components share a common memory address space
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064
• CPC Class Codes: G06F 2201/825 the problem or solution involving locking
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2209/523 Mode
• CPC Class Codes: G06F 3/121 Facilitating exception or error detection and recovery, e.g. fault, media or consumables depleted
• CPC Class Codes: G06F 9/3863 using multiple copies of the architectural state, e.g. shadow registers
• CPC Class Codes: G06F 9/526 Mutual exclusion algorithms
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A set of data structures are stored in a distributed shared memory (DSM) component and in persistent storage. The DSM component is organized as a matrix of page. The data structure of the set of data structures occupies a column in the matrix of pages. A recovery file is maintained in the persistent storage. The recovery file consists of entries and each one of the entries corresponds to a column in the matrix of pages by a location of each one of the entries.

Claims

1. A method for restoring distributed shared memory (DSM) data consistency within a recovery process from a failure of a node in a cluster of nodes by a processor device, comprising:

storing a set of data structures in a DSM component and in persistent storage;

organizing the DSM component as a matrix of pages, wherein a data structure of the set of data structures occupies a column in the matrix of pages;

maintaining a recovery file in the persistent storage, wherein the recovery file consists of a plurality of entries and each one of the plurality of entries corresponds to a column in the matrix of pages by a location of each one of the plurality of entries;

incorporating into each one of the plurality of entries in the recovery file an indication that an associated column in the matrix of pages is assigned with the data structure of the set of data structures;

incorporating into each one of the plurality of entries in the recovery file identifying key properties of the data structure of the set of data structures and a specification of the location of the data structure in the persistent storage when the associated column in the matrix of pages is assigned; and

defining each column in the matrix of pages to consist of a plurality of pages, wherein at least one page of the plurality of pages includes the identifying key properties of at least one of a plurality of data structures and additional pages of the plurality of pages include updatable properties of the at least one of the plurality of data structures.

2. The method of claim 1, further including performing one of:

updating the recovery file during an addition of a new data structure to the set of data structures by performing one of;

updating one of the plurality of entries in the recovery file, which corresponds by the location to the column in the matrix of pages of the recovery file where the new data structure has been inserted, with the identifying key properties of the new data structure and with the location of the new data structure in the persistent storage, and

modifying those of the plurality of entries in the recovery file, which correspond by the location of the plurality of entries in the recovery file to the columns in the matrix of pages, wherein the assignments of the columns in the matrix of pages with existing data structures have been modified during the addition of the new data structure, with updated identifying key properties and storage locations reflecting the new assignments.

3. The method of claim 2, further including modifying the recovery file such that the updating is performed in memory and then written to the recovery file in the persistent storage.

4. The method of claim 1, further including using the recovery file during the recovery process by performing one of:

denoting pages of the plurality of pages in the matrix of pages, whose contents is invalid, as invalid DSM pages,

for an invalid DSM page of the plurality of pages containing identifying key properties,

reassigning one of the plurality of pages if one of;

one of the entries in the plurality of entries in the recovery file corresponding to the column in the matrix of pages containing one of the plurality of invalid DSM pages indicates that there is no assignment of the column to the data structure in the persistent storage, and

the one of the entries in the plurality of entries indicates that there is the assignment but the data structure does not exist in the specified location in the persistent storage, then that DSM page is reset to be unassigned

loading contents of the data structure from the persistent storage into the one of the plurality of pages and setting the one of the plurality of pages as assigned to the data structure if the one of the entries in the plurality of entries in the recovery file corresponding to the column in the matrix of pages indicates that there is the assignment of the column to the data structure in the persistent storage and there is an actual data structure in the specified location in the persistent storage, and

setting the one of the plurality of pages as valid.

5. The method of claim 4, further including, for an additional invalid DSM page containing updateable properties, performing one of:

resetting the additional invalid DSM page to be unassigned if a DSM page containing the identifying key properties, which is located within a same column in the matrix of pages, is not assigned with the data structure,

loading contents of the data structure from the persistent storage into the additional invalid DSM page, and setting the additional invalid DSM page as assigned to the data structure, if the DSM page containing the identifying key properties, which is located within the same column in the matrix of pages, is assigned with the data structure, and

setting the additional invalid DSM page as valid.

6. The method of claim 4, further including determining a DSM page of the plurality of pages as invalid if the DSM page is invalid in a local DSM of a node performing a recovery operation and if the DSM page is determined to be invalid in a plurality of DSMs of all other operational cluster nodes in reply to a query sent by the node to each of the operational cluster nodes.

7. The method of claim 6, further including blocking access, except for the recovery operation, by a DSM mechanism to the plurality of pages determined to be invalid for a duration of the recovery operation.

8. A system for restoring distributed shared memory (DSM) data consistency within a recovery process from a failure of a node in a cluster of nodes in a computing environment, comprising:

the DSM;

a DSM component in the DSM having set of data structures;

a plurality of nodes in the cluster of nodes in the DSM;

a storage system associating with the cluster of nodes, wherein the storage system includes at least persistent storage; and

at least one processor device, operable in the DSM, in communication with the plurality of nodes and the storage system, wherein the at least one processor device;

stores a set of the data structures in the DSM component and in the persistent storage;

organizes the DSM component as a matrix of pages, wherein a data structure of the set of data structures occupies a column in the matrix of pages;

maintains a recovery file in the persistent storage, wherein the recovery file consists of a plurality of entries and each one of the plurality of entries corresponds to a column in the matrix of pages by a location of each one of the plurality of entries;

incorporates into each one of the plurality of entries in the recovery file an indication that an associated column in the matrix of pages is assigned with the data structure of the set of data structures;

incorporates into each one of the plurality of entries in the recovery file identifying key properties of the data structure of the set of data structures and a specification of the location of the data structure in the persistent storage when the associated column in the matrix of pages is assigned; and

defines each column in the matrix of pages to consist of a plurality of pages, wherein at least one page of the plurality of pages includes the identifying key properties of at least one of a plurality of data structures and additional pages of the plurality of pages include updatable properties of the at least one of the plurality of data structures.

9. The system of claim 8, wherein the at least one processor device performs one of:

updating the recovery file during an addition of a new data structure to the set of data structures by performing one of;

updating one of the plurality of entries in the recovery file, which corresponds by the location to the column in the matrix of pages of the recovery file where the new data structure has been inserted, with the identifying key properties of the new data structure and with the location of the new data structure in the persistent storage, and

modifying those of the plurality of entries in the recovery file, which correspond by the location of the plurality of entries in the recovery file to the columns in the matrix of pages, wherein the assignments of the columns in the matrix of pages with existing data structures have been modified during the addition of the new data structure, with updated identifying key properties and storage locations reflecting the new assignments.

10. The system of claim 9, wherein the at least one processor device modifies the recovery file such that the updating is performed in memory and then written to the recovery file in the persistent storage.

11. The system of claim 8, wherein the at least one processor device uses the recovery file during the recovery process by performing one of:

denoting pages of the plurality of pages in the matrix of pages, whose contents is invalid, as invalid DSM pages,

for an invalid DSM page of the plurality of pages containing identifying key properties,

reassigning one of the plurality of pages if one of;

one of the entries in the plurality of entries in the recovery file corresponding to the column in the matrix of pages containing one of the plurality of invalid DSM pages indicates that there is no assignment of the column to the data structure in the persistent storage, and

the one of the entries in the plurality of entries indicates that there is the assignment but the data structure does not exist in the specified location in the persistent storage, then that DSM page is reset to be unassigned,

loading contents of the data structure from the persistent storage into the one of the plurality of pages and setting the one of the plurality of pages as assigned to the data structure if the one of the entries in the plurality of entries in the recovery file corresponding to the column in the matrix of pages indicates that there is the assignment of the column to the data structure in the persistent storage and there is an actual data structure in the specified location in the persistent storage, and

setting the one of the plurality of pages as valid.

12. The system of claim 11, wherein the at least one processor device, for an additional invalid DSM page containing updateable properties, performs one of:

resetting the additional invalid DSM page to be unassigned if a DSM page containing the identifying key properties, which is located within a same column in the matrix of pages, is not assigned with the data structure,

loading contents of the data structure from the persistent storage into the additional invalid DSM page, and setting the additional invalid DSM page as assigned to the data structure, if the DSM page containing the identifying key properties, which is located within the same column in the matrix of pages, is assigned with the data structure, and

setting the additional invalid DSM page as valid.

13. The system of claim 11, wherein the at least one processor device determines a DSM page of the plurality of pages as invalid if the DSM page is invalid in a local DSM of a node performing a recovery operation and if the DSM page is determined to be invalid in a plurality of DSMs of all other operational cluster nodes in reply to a query sent by the node to each of the operational cluster nodes.

14. The system of claim 13, further including a DSM mechanism controlled by the at least one processor device, wherein the DSM mechanism blocks access, except for the recovery operation, by the DSM component to the plurality of pages determined to be invalid for a duration of the recovery operation.

15. A computer program product for restoring distributed shared memory (DSM) data consistency within a recovery process from a failure of a node in a cluster of nodes by a processor device, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion that stores a set of data structures in a DSM component and in persistent storage;

a second executable portion that organizes the DSM component as a matrix of pages, wherein a data structure of the set of data structures occupies a column in the matrix of pages;

a third executable portion that maintains a recovery file in the persistent storage, wherein the recovery file consists of a plurality of entries and each one of the plurality of entries corresponds to a column in the matrix of pages by a location of each one of the plurality of entries;

a fourth executable portion that performs each of;

incorporating into each entry in the recovery file an indication that an associated column in the matrix of pages is assigned with the data structure of the set of data structures;

incorporating into each entry in the recovery file identifying key properties of the data structure of the set of data structures and a specification of the location of the data structure in the persistent storage when the associated column in the matrix of pages is assigned; and

a fifth executable portion that defines each column in the matrix of pages to consist of a plurality of pages, wherein at least one page of the plurality of pages includes the identifying key properties of at least one of a plurality of data structures and additional pages of the plurality of pages include updatable properties of the at least one of the plurality of data structures.

16. The computer program product of claim 15, further including a fifth executable portion that performs one of:

updating the recovery file during an addition of a new data structure to the set of data structures by performing one of;

updating the entry in the recovery file, which corresponds by the location to the column in the matrix of pages of the recovery file where the new data structure has been inserted, with the identifying key properties of the new data structure and with the location of the new data structure in the persistent storage, and

modifying those of the plurality of entries in the recovery file, which correspond by the location of the plurality of entries in the recovery file to the columns in the matrix of pages, wherein the assignments of the columns in the matrix of pages with existing data structures have been modified during the addition of the new data structure, with updated identifying key properties and storage locations reflecting the new assignments.

17. The computer program product of claim 16, further including a sixth executable portion that modifies the recovery file such that the updating is performed in memory and then written to the recovery file in the persistent storage.

18. The computer program product of claim 15, further including a sixth executable portion that uses the recovery file during the recovery process by performing one of:

denoting pages of the plurality of pages with invalid contents that are located in the matrix of pages as invalid DSM pages,

for an invalid DSM page of the plurality of pages containing identifying key properties,

reassigning one of the plurality of pages if one of;

one of the entries in the plurality of entries in the recovery file corresponding to the column in the matrix of pages indicating that there is no assignment of the column to the data structure in the persistent storage, and

the one of the entries in the plurality of entries indicates that there is the assignment but there the data structure does not exist in the specified location in the persistent storage, then that DSM page is reset to be unassigned,

loading contents of the data structure from the persistent storage into the one of the plurality of pages and setting the one of the plurality of pages as assigned to the data structure if the one of the entries in the plurality of entries in the recovery file corresponding to the column in the matrix of pages indicates that there is the assignment of the column to the data structure in the persistent storage and there is an actual data structure in the specified location in the persistent storage, and

setting the one of the plurality of pages as valid.

19. The computer program product of claim 18, further including a seventh executable portion that, for an additional invalid DSM page containing updateable properties, performs one of:

resetting the additional invalid DSM page to be unassigned if a DSM page containing the identifying key properties, which is located within a same column in the matrix of pages, is not assigned with the data structure,

loading contents of the data structure from the persistent storage into the additional invalid DSM page, and setting the additional invalid DSM page as assigned to the data structure, if the DSM page containing the identifying key properties, which is located within the same column in the matrix of pages, is assigned with the data structure, and

setting the additional invalid DSM page as valid.

20. The computer program product of claim 18, further including a seventh executable portion that determines a DSM page of the plurality of pages as invalid if the DSM page is invalid in a local DSM of a node performing a recovery operation and if the DSM page is determined to be invalid in a plurality of DSMs of all other operational cluster nodes in reply to a query sent by the node to each of the operational cluster nodes.

21. The computer program product of claim 20, further including an eighth executable portion that blocks access by a DSM mechanism, except for the recovery operation, by the DSM component to the plurality of pages determined to be invalid for a duration of the recovery operation.