Maintaining global namespace consistency for a distributed filesystem
First Claim
1. A computer-implemented method for maintaining global namespace consistency for a distributed filesystem, the method comprising:
- collectively managing a set of data stored in the distributed filesystem using two or more cloud controllers, wherein collectively managing the set of data comprises storing the set of data for the distributed filesystem in one or more cloud storage systems and maintaining metadata for a hierarchical namespace for the distributed filesystem to manage and access the files and directories in which the set of data is stored into a global namespace, wherein the cloud controllers cache and ensure data consistency for the set of data stored in the cloud storage systems;
receiving at a first cloud controller a request from a client, wherein the request specifies a namespace operation for a namespace target in the global namespace of the distributed filesystem, wherein the requested namespace operation changes at least one of the presence or the location of the namespace target in the hierarchical namespace but does not modify the contents of a filesystem object that is associated with the namespace target, wherein the namespace operation is a rename operation and the filesystem object is a directory; and
contacting a second, distinct cloud controller that manages a portion of the global namespace for the distributed filesystem that includes the namespace target, wherein the second cloud controller detects and blocks any additional namespace operations that are initiated for the namespace target by any other cloud controllers in the distributed filesystem during a time interval in which the namespace operation is being executed to preserve global namespace consistency for the distributed filesystem,wherein, upon determining that the rename operation is valid and allowed, the second cloud controller commits the namespace operation by broadcasting notification of the rename operation to the other cloud controllers for the distributed filesystem and sending an incremental metadata snapshot to the other cloud controllers for the distributed filesystem to notify the other cloud controllers of the renamed directory;
wherein the second cloud controller receives multiple rename requests for the directory in a short time interval;
wherein the second cloud controller conveys the order of the resulting multiple rename operations for the directory in the incremental metadata snapshot to preserve the sequential nature of the rename operations; and
wherein a third cloud controller receiving the incremental metadata snapshot replays the rename operations in the order conveyed in the incremental metadata snapshot to match the global namespace of the second cloud controller consistently.
7 Assignments
0 Petitions
Accused Products
Abstract
The disclosed embodiments disclose techniques for maintaining global name consistency for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a client request to perform a namespace operation upon a filesystem object. The cloud controller contacts the cloud controller that manages (“owns”) the portion of the global namespace for the distributed filesystem that includes the filesystem object. This second cloud controller ensures the consistency of the filesystem object across the distributed filesystem during the namespace operation.
36 Citations
15 Claims
-
1. A computer-implemented method for maintaining global namespace consistency for a distributed filesystem, the method comprising:
-
collectively managing a set of data stored in the distributed filesystem using two or more cloud controllers, wherein collectively managing the set of data comprises storing the set of data for the distributed filesystem in one or more cloud storage systems and maintaining metadata for a hierarchical namespace for the distributed filesystem to manage and access the files and directories in which the set of data is stored into a global namespace, wherein the cloud controllers cache and ensure data consistency for the set of data stored in the cloud storage systems; receiving at a first cloud controller a request from a client, wherein the request specifies a namespace operation for a namespace target in the global namespace of the distributed filesystem, wherein the requested namespace operation changes at least one of the presence or the location of the namespace target in the hierarchical namespace but does not modify the contents of a filesystem object that is associated with the namespace target, wherein the namespace operation is a rename operation and the filesystem object is a directory; and contacting a second, distinct cloud controller that manages a portion of the global namespace for the distributed filesystem that includes the namespace target, wherein the second cloud controller detects and blocks any additional namespace operations that are initiated for the namespace target by any other cloud controllers in the distributed filesystem during a time interval in which the namespace operation is being executed to preserve global namespace consistency for the distributed filesystem, wherein, upon determining that the rename operation is valid and allowed, the second cloud controller commits the namespace operation by broadcasting notification of the rename operation to the other cloud controllers for the distributed filesystem and sending an incremental metadata snapshot to the other cloud controllers for the distributed filesystem to notify the other cloud controllers of the renamed directory; wherein the second cloud controller receives multiple rename requests for the directory in a short time interval; wherein the second cloud controller conveys the order of the resulting multiple rename operations for the directory in the incremental metadata snapshot to preserve the sequential nature of the rename operations; and wherein a third cloud controller receiving the incremental metadata snapshot replays the rename operations in the order conveyed in the incremental metadata snapshot to match the global namespace of the second cloud controller consistently. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
-
14. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for maintaining global namespace consistency for a distributed filesystem, the method comprising:
-
collectively managing a set of data stored in the distributed filesystem using two or more cloud controllers, wherein collectively managing the set of data comprises storing the set of data for the distributed filesystem in one or more cloud storage systems and maintaining metadata for a hierarchical namespace for the distributed filesystem to manage and access the files and directories in which the set of data is stored into a global namespace, wherein the cloud controllers cache and ensure data consistency for the set of data stored in the cloud storage systems; receiving at a first cloud controller a request from a client, wherein the request specifies a namespace operation for a namespace target in the global namespace of the distributed filesystem, wherein the requested namespace operation changes at least one of the presence or the location of the namespace target in the hierarchical namespace but does not modify the contents of a filesystem object that is associated with the namespace target, wherein the namespace operation is a rename operation and the filesystem object is a directory; and contacting a second, distinct cloud controller that manages a portion of the global namespace for the distributed filesystem that includes the namespace target, wherein the second cloud controller detects and blocks any additional namespace operations that are initiated for the namespace target by any other cloud controllers in the distributed filesystem during a time interval in which the namespace operation is being executed to preserve global namespace consistency for the distributed filesystem wherein upon determining that the rename operation is valid and allowed, the second cloud controller commits the namespace operation by broadcasting notification of the rename operation to the other cloud controllers for the distributed filesystem and sending an incremental metadata snapshot to the other cloud controllers for the distributed filesystem to notify the other cloud controllers of the renamed directory; wherein the second cloud controller receives multiple rename requests for the directory in a short time interval; wherein the second cloud controller conveys the order of the resulting multiple rename operations for the directory in the incremental metadata snapshot to preserve the sequential nature of the rename operations, and wherein a third cloud controller receiving the incremental metadata snapshot replays the rename operations in the order conveyed in the incremental metadata snapshot to match the global namespace of the second cloud controller consistently.
-
-
15. A cloud controller that maintains global namespace consistency for a distributed filesystem, comprising:
-
a processor; a storage mechanism that stores metadata for the distributed filesystem; and a storage management mechanism; wherein two or more cloud controllers collectively manage a set of data stored in the distributed filesystem, wherein collectively managing the set of data comprises storing the set of data for the distributed filesystem in one or more cloud storage systems and maintaining metadata for a hierarchical namespace for the distributed filesystem to manage and access the files and directories in which the set of data is stored into a global namespace, wherein the cloud controllers cache and ensure data consistency for the set of data stored in the cloud storage systems; wherein the cloud controller is configured to receive a request from a client, wherein the request specifies a namespace operation for a namespace target in the global namespace of the distributed filesystem, wherein the requested namespace operation changes at least one of the presence or the location of the namespace target in the hierarchical namespace but does not modify the contents of a filesystem object that is associated with the namespace target, wherein the namespace operation is a rename operation and the filesystem object is a directory; and
;wherein the storage management mechanism contacts a second, distinct cloud controller that manages a portion of the global namespace for the distributed filesystem that includes the namespace target, wherein the second cloud controller detects and blocks any additional namespace operations that are initiated for the namespace target by any other cloud controllers in the distributed filesystem during a time interval in which the namespace operation is being executed to preserve global namespace consistency for the distributed filesystem; wherein upon determining that the rename operation is valid and allowed, the second cloud controller commits the namespace operation by broadcasting notification of the rename operation to the other cloud controllers for the distributed filesystem and sending an incremental metadata snapshot to the other cloud controllers for the distributed filesystem to notify the other cloud controllers of the renamed directory; wherein the second cloud controller receives multiple rename requests for the directory in a short time interval; wherein the second cloud controller conveys the order of the resulting multiple rename operations for the directory in the incremental metadata snapshot to preserve the sequential nature of the rename operations; and wherein a third cloud controller receiving the incremental metadata snapshot replays the rename operations in the order conveyed in the incremental metadata snapshot to match the global namespace of the second cloud controller consistently.
-
Specification