Providing local cache coherency in a shared storage environment

• US 9,501,410 B2
• Filed: 03/15/2013
• Issued: 11/22/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Kalekar, Santosh, Pendharkar, Niranjan, Marathe, Shailesh
• Primary Examiner(s): Sadler, Nathan
• Application Number:
  US13/838,125
• Publication Number:
  US 20140281273A1
• Time in Patent Office: 1,348 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0817 using directory methods
• CPC Class Codes: G06F 12/0833 in combination with broadcast means (e.g. for invalidation or updating)
• CPC Class Codes: G06F 2212/1008 Correctness of operation, e.g. memory ordering
• CPC Class Codes: G06F 2212/1024 Latency reduction
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 2212/311 In host system
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Multiple nodes of a cluster have associated non-shared, local caches, used to cache shared storage content. Each local cache is accessible only to the node with which it is associated, whereas the cluster-level shared storage is accessible by any of the nodes. Attempts to access the shared storage by the nodes of the cluster are monitored. Information is tracked concerning the current statuses of the local caches of the nodes of the cluster. Current tracked local cache status information is maintained, and stored such that it is accessible by the multiple nodes of the cluster. The current tracked local cache status information is used in conjunction with the caching functionality to determine whether specific nodes of the cluster are to access their local caches or the shared storage to obtain data corresponding to specific regions of the shared storage.

Claims

1. A computer implemented method in a shared storage server for providing cache coherency within a cluster of multiple nodes, the method comprising:

communicating with a cache coherency instance at each of the multiple nodes of the cluster, wherein

the each of the multiple nodes comprises a local cache that is configured to cache shared storage content of the shared storage server,

each local cache is only accessible by the each node with which the each local cache is associated, and

the shared storage content is accessible by the multiple nodes of the cluster;

receiving one or more updates sent from the cache coherency instance executing on the each of the multiple nodes, wherein

the one or more updates represent one or more operations that are configured to access the shared storage content, when performed by the each node, and

each of the one or more updates is sent from a corresponding cache coherency instance after a predetermined number of operations are performed by the each node;

monitoring attempts to access the shared storage content by the each of the multiple nodes of the cluster based on the updates;

tracking statuses of the local caches of the multiple nodes of the cluster by determining whether an access attempt at a node affects the status of the local cache associated with the node and each of the other local caches of the other nodes in the cluster,

wherein

the access attempt comprises one or more accesses that modify the shared storage content and one or more accesses that do not modify the shared storage content, and

each status stored in the shared storage server indicates whether the each local cache has a current version of each portion of shared storage content; and

upon a subsequent access attempt by one node of the multiple nodes to a portion of the shared storage content in the local cache associated with the one node;

accessing a specific portion of shared storage content from the local cache associated with the one node, if a determination is made from the statuses stored in the shared storage server that the current version of the specific portion of shared storage content is stored in the local cache associated with the one node, and

accessing the specific portion of shared storage content from the shared storage server, if a determination is made from the statuses that the current version of the specific portion of shared storage content is not stored on the local cache associated with the one node.

2. The method of claim 1, wherein the monitoring the attempts to access the shared storage content by the each of the nodes of the cluster further comprises:

monitoring at least some operations being performed by a specific node of the cluster that result in an update being made to the local cache associated with the specific node.

3. The method of claim 1, wherein the monitoring the attempts to access the shared storage content by the nodes of the cluster further comprises:

monitoring any operations that move content between nodes of the cluster and the shared storage server.

4. The method of claim 1, wherein the tracking the statuses of the local caches of the multiple nodes of the cluster further comprises:

tracking any nodes of the cluster that contain current shared storage content in their local caches, for any regions of the shared storage content.

5. The method of claim 1, wherein the tracking the statuses of the local caches of the multiple nodes of the cluster further comprises:

tracking any nodes of the cluster that contain non-current shared storage content in their associated local caches, for any regions of the shared storage content.

6. The method of claim 1, further comprising:

maintaining current tracked local cache status information; and

responsive to at least one operation targeting at least one region of the shared storage content, updating the current tracked local cache status information to indicate local cache status changes for multiple local caches, for the at least one region of the shared storage content.

7. The method of claim 1, further comprising:

maintaining current tracked local cache status information; and

responsive to a read operation by a specific node of the cluster targeting at least one region of the shared storage content, updating the current tracked local cache status information to indicate a local cache status change for the local cache associated with the specific node, for the at least one region of the shared storage content.

8. The method of claim 1, further comprising:

maintaining current tracked local cache status information; and

responsive to a first write operation by a specific node of the cluster targeting at least one region of the shared storage content, updating the current tracked local cache status information to indicate a local cache status change for the local cache associated with the specific node, and to indicate local cache status changes for additional local caches associated with additional nodes of the cluster, for the at least one region of the shared storage content.

9. The method of claim 1, further comprising:

maintaining current tracked local cache status information; and

storing the current tracked local cache status information in a data structure, wherein

at least a portion of the data structure is stored on the shared storage server such that it is accessible by the multiple nodes of the cluster.

10. The method of claim 9, wherein

the data structure further comprises a bitmap residing on the shared storage server,

the bitmap comprises, for each specific node of the cluster, a separate bit set representing a current tracked local cache status of the local cache associated with the specific node, and

each bit set comprises one bit representing each tracked region of the shared storage content.

11. The method of claim 1, further comprising:

maintaining current tracked local cache status information; and

storing the current tracked local cache status information in a data structure, wherein

a portion of the data structure is stored on the shared storage server such that it is accessible by the multiple nodes of the cluster, and, for each specific node of the cluster, a portion of the data structure is stored locally by and is accessible only to the specific node.

12. The method of claim 11, wherein

the data structure further comprises:

for each specific node of the cluster, a set of data variables representing a current tracked local cache status of the local cache associated with the specific node, and

a single, shared set of data variables indicating a current content status of the shared storage content, and

each specific data variable of each data variable set represents a specific region of the shared storage content.

13. The method of claim 1, further comprising:

maintaining current tracked local cache status information; and

representing specific data volumes of the shared storage content in the current tracked local cache status information.

14. The method of claim 1, further comprising:

initializing all of the tracked local cache status information to indicate that, for each specific node of the cluster, the local cache associated with the specific node is non-current for every region of the shared storage content.

15. At least one non-transitory computer readable-storage medium in a shared storage server for providing cache coherency within a cluster of multiple nodes, the at least one non-transitory computer readable-storage medium storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to:

communicate with a cache coherency instance at each of the multiple nodes of the cluster, wherein

the each of the multiple nodes comprises a local cache that is configured to cache shared storage content of the shared storage server,

each local cache is only accessible by the each node with which the each local cache is associated, and

the shared storage content is accessible by the multiple nodes of the cluster;

receive one or more updates sent from the cache coherency instance executing on the each of the multiple nodes, wherein

the one or more updates represent one or more operations that are configured to access the shared storage content, when performed by the each node, and

each of the one or more updates is sent from a corresponding cache coherency instance after a predetermined number of operations are performed by the each node;

monitor attempts to access the shared storage content by the each of the multiple nodes of the cluster based on the updates;

track statuses of the local caches of the multiple nodes of the cluster by determining whether an access attempt at a node affects the status of the local cache associated with the node and each of the other local caches of the other nodes in the cluster,

wherein

the access attempt comprises one or more accesses that modify the shared storage content and one or more accesses that do not modify the shared storage content, and

each status stored in the shared storage server indicates whether the each local cache has a current version of each portion of shared storage content; and

upon a subsequent access attempt by one node of the multiple nodes to a portion of the shared storage content in a local cache associated with the one node;

access a specific portion of shared storage content from the local cache associated with the one node, if a determination is made from the statuses stored in the shared storage server that the current version of the specific portion of shared storage content is stored in the local cache associated with the one node, and

access the specific portion of shared storage content from the shared storage server, if a determination is made from the statuses that the current version of the specific portion of shared storage content is not stored on the local cache associated with the one node.

16. The at least one non-transitory computer readable-storage medium of claim 15 further storing computer executable instructions to:

use information gleaned from monitoring node-level attempts to access the shared storage content to determine the statuses of the local caches.

17. The at least one non-transitory computer readable-storage medium of claim 15 further storing computer executable instructions to:

track any nodes of the cluster that contain current shared storage content in their local caches, for any regions of the shared storage content; and

track any nodes of the cluster that contain non-current shared storage content in their associated local caches, for any regions of the shared storage content.

18. The at least one non-transitory computer readable-storage medium of claim 15 further storing computer executable instructions to:

responsive to a read operation by a specific node of the cluster targeting at least one region of the shared storage content, updating current tracked local cache status information to indicate a local cache status change for the local cache associated with the specific node, for the at least one region of the shared storage content; and

responsive to a first write operation by the specific node of the cluster targeting the at least one region of the shared storage content, updating the current tracked local cache status information to indicate the local cache status change for the local cache associated with the specific node, and to indicate local cache status changes for additional local caches associated with additional nodes of the cluster, for the at least one region of the shared storage content.

19. A computer system for providing cache coherency within a cluster of multiple nodes, the computer system comprising:

a processor; and

system memory, storing;

a first module to communicate with a cache coherency instance at each of the multiple nodes of the cluster, wherein

the each of the multiple nodes comprises a local cache that is configured to cache shared storage content of the shared storage server,

each local cache is only accessible by the each node with which the each local cache is associated, and

the shared storage content is accessible by the multiple nodes of the cluster;

a second module to receive one or more updates sent from the cache coherency instance executing on the each of the multiple nodes, wherein

the one or more updates represent one or more operations that are configured to access the shared storage content, when performed by the each node, and

each of the one or more updates is sent from a corresponding cache coherency instance after a predetermined number of operations are performed by the each node;

a third module to monitor attempts to access the shared storage content by the each of the multiple nodes of the cluster based on the updates;

a fourth module to track statuses of the local caches of the multiple nodes of the cluster by determining whether an access attempt at a node affects the status of the local cache associated with the node and each of the other local caches of the other nodes in the cluster, wherein

the access attempt comprises one or more accesses that modify the shared storage content and one or more accesses that do not modify the shared storage content, and

each status stored in the shared storage server indicates whether each local cache has a current version of each portion of shared storage content; and

a fifth module to, upon a subsequent access attempt by one of the nodes to a portion of shared storage content in a local cache associated with the one node,

access a specific portion of shared storage content from the local cache associated with the one node, if a determination is made from the statuses stored in the shared storage server that the current version of the specific portion of shared storage content is stored in the local cache associated with the one node, and

access the specific portion of shared storage content from the shared storage server, if a determination is made from the statuses that the current version of the specific portion of shared storage content is not stored on the local cache associated with the one node.