Dynamic update to views of a file system backed by object storage

• US 10,740,302 B2
• Filed: 05/25/2016
• Issued: 08/11/2020
• Est. Priority Date: 03/02/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Slik, David, Altman, Tym, Ciapponi, Adam F.
• Primary Examiner(s): Pham, Tuan A
• Application Number:
  US15/164,465
• Publication Number:
  US 20160267103A1
• Time in Patent Office: 1,539 Days
• Field of Search: None
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system can apply file placement rules to dynamically place files and directories within file system views backed by objects in an object storage system. After detection of an update to a first file system view that causes an update of an object in a storage grid, an object manager begins evaluation of file placement rules against metadata of the object. For each file placement rule that is triggered, the object manager determines identifies gateways that export the first file system view. The object manager then instructs the gateways to update their representations of the first file system view. The disclosed embodiments may be able to scale to managing hundreds of billions of files spanning thousands of file system views, especially in the presence of disconnected operation.

Claims

1. A method comprising:

detecting an object storage operation for a first object in a distributed object storage system;

identifying a first file placement rule for a first set of gateways that maintain local representations of a first namespace, wherein the first set of gateways is associated with the distributed object storage system;

determining a first action to be performed based, at least in part, on the first namespace and the detected object storage operation for the first object; and

causing the first set of gateways to update local representations of the first name space to include a path for a file to be created, including communicating a notification that indicates an object identifier of the first object and that indicates the path, wherein the first action comprises a conflict resolution action to be performed by a first gateway, of the first set of gateways, that detects a local representation conflict from the update.

2. The method of claim 1, further comprising identifying the first namespace from metadata of the first object.

3. The method of claim 1, wherein the first namespace is a sub-space of a file system namespace.

4. The method of claim 1, wherein the first object represents either a file system file or a file system directory.

5. The method of claim 1, further comprising:

detecting an update to a file system, wherein the object storage operation corresponds to the update to the file system and the first namespace is a sub-space of the file system'"'"'s namespace.

6. The method of claim 1, wherein generating the notification comprises generating the notification with an indication of a hierarchical change in the first namespace based on the detected object storage operation.

7. The method of claim 1, wherein generating the notification comprises generating the notification also with an object identifier of a second object, wherein the second object represents a directory and the first object represents a file within the directory.

8. The method of claim 1, further comprising:

identifying a second file placement rule defined for the first namespace, wherein the second file placement rule indicates that an object created in the first namespace and matching a criterion is to be replicated to a second namespace.

9. The method of claim 1, further comprising:

determining that the object storage operations correspond to the first action and do not correspond to a second action.

10. A non-transitory machine readable medium having stored thereon instructions for performing a method comprising machine executable code which when executed by at least one machine, causes the machine to:

detect an update to a first share that causes an update of a first object in the storage grid;

begin evaluation of a plurality of file placement rules against metadata of the first object, wherein the first object corresponds to a file or a directory;

based on the evaluation, determine that at least a first file placement rule is triggered by the update of the first object;

identify a first set of gateways that export the first share to clients, wherein the first set of gateways is associated with the storage grid; and

communicate an instruction to the first set of gateways to perform a first action defined in the first file placement rule, the first action including creating a file according to a path in the first share, and to perform a second action including a conflict resolution action defined in the first file placement rule.

11. The non-transitory computer readable medium of claim 10, wherein the update to the first share comprises a create, delete, or modify of a file or a directory.

12. The non-transitory computer readable medium of claim 10, further comprising instructions to detect the update to the first share and to notify the first set of gateways of the update based on the first set of gateways previously subscribing for notification of updates to the first share.

13. The non-transitory computer readable medium of claim 10, wherein the instructions to evaluate comprise the program instructions to periodically evaluate the plurality of file placement rules.

14. The non-transitory computer readable medium of claim 10, wherein the instructions to evaluate comprise the program instructions to evaluate the plurality of file placement rules in response to detection of object creation or object destruction in the storage grid.

15. The non-transitory computer readable medium of claim 10, wherein a plurality of shares corresponds to a file system namespace and the plurality of shares comprise the first share.

16. The non-transitory computer readable medium of claim 10, wherein the instructions to begin evaluation of the plurality of file placement rules against the first object comprise instructions to determine whether metadata of the first object identifies a share specified by a criterion of the file placement rule.

17. A computing device comprising:

a network interface;

a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of synchronizing shares across a storage grid; and

a processor coupled to the memory and the network interface, the processor configured to execute the machine executable code to cause the processor to;

detect an update to a first representation of a first shared view of a file system;

begin evaluation of a plurality of file placement rules against metadata of a first object, wherein the update to the first representation causes an update in an object storage system for the first object;

based on the evaluation, determine that at least a first file placement rule and a second file placement rule are triggered;

identify a first set of gateways that maintain a respective one of a set of representations of the first shared view; and

communicate, via the network interface, an instruction to the first set of gateways to perform a first action comprising creating a file according to a path, the first action defined in the first file placement rule to synchronize the set of representations and to perform a second action comprising a conflict resolution action at one or more gateways of the first set of gateways in response to detecting a conflict resulting from the synchronization in a corresponding representation of the set of representations maintained at the one or more gateways, the second action defined in the second file placement rule.

18. The computing device of claim 17, wherein the one or more gateways comprises a first gateway and a second gateway, and the conflict resolution action comprises a first conflict resolution action at the first gateway and a second conflict resolution action, different from the first conflict resolution action, at the second gateway.