Systems and methods for managing files in a content storage system

• US 9,355,120 B1
• Filed: 03/01/2013
• Issued: 05/31/2016
• Est. Priority Date: 03/02/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Altman, Tym, Ciapponi, Adam F., Slik, David
• Primary Examiner(s): Pham, Tuan A
• Application Number:
  US13/782,056
• Time in Patent Office: 1,187 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems and methods that allow operators to configure how files and directories are placed within file system views into a storage system, and how these configured file placements are performed by the distributed system. Possible features include a mechanism by which the desired placement of files and directories can be specified, a mechanism by which the placement goals can be realized in a given topology of a distributed system, and a mechanism by which changes to the placement of objects can applied retroactively. The disclosed embodiments may be able to scale to managing hundreds of billions of files spanning thousands of file system views, especially in the presence of disconnected operation.

Claims

1. A non-transitory computer readable medium having instructions stored thereon, the non-transitory computer readable medium comprising instructions for:

receiving a placement rule comprising selection criteria and a placement action, the placement action comprising information relating to locations in a hierarchical directory structure to place data objects that corresponds to the selection criteria and the placement action comprising a configurable conflict resolution mechanism;

after receiving the placement rule, identifying a previous data object that was previously stored to the storage system and corresponds to the selection criteria of the placement rule; and

modifying a location in the hierarchical directory structure corresponding to the previous data object, the modified location being determined based on the placement action of the placement rule and at least one attribute of the previous data object;

applying the configurable conflict resolution mechanism to enable user-defined methods of resolving conflicts, when multiple clients write to the data object without locking the hierarchical directory structure;

receiving from an object store device, a notification comprising information indicating receipt of, at a first gateway device, a data object to be stored in a storage system, wherein the storage system comprises a distributed storage system comprising a plurality of administrative domains, wherein the placement rule is associated with one of the plurality of administrative domains;

determining that the data object corresponds to the selection criteria of the placement rule by;

determining whether the data object is associated with a rules identifier identifying the placement rule; and

evaluating the selection criteria, when the rules identifier is not present or upon determining the rules identifier is associated with a previous version of the placement rule, to determine whether the data object corresponds to the selection criteria of the placement rule;

determining that the data object is associated with the same administrative domain as the placement rule based, at least in part, on a determination that the data object corresponds to the selection criteria of the placement rule;

determining a location in the hierarchical directory structure to place the data object, the determined location being based on the placement action of the placement rule and at least one attribute of the data object; and

transmitting in accordance with the placement rule, to a second gateway device, a notification comprising information to enable the second gateway device to update a representation of the hierarchical directory structure to include the determined location by creating a new reference to the data object and according to said modifying.

2. The non-transitory computer readable medium of claim 1,

wherein:

the selection criteria comprise a plurality of conditions; and

determining that the data object corresponds to the selection criteria of the placement rule comprises determining that the data object satisfies at least one of the plurality of conditions.

3. The non-transitory computer readable medium of claim 1, further comprising instructions for:

determining that the second gateway device is not accessible; and

queuing the notification for later transmission.

4. The non-transitory computer readable medium of claim 1, wherein receiving the data object at the first gateway device comprises placing the data object on a processing queue, the non-transitory computer readable medium further comprising instructions for:

transferring the data object from the processing queue to an evaluation queue prior to determining that the data object corresponds to the selection criteria of the placement rule.

5. The non-transitory computer readable medium of claim 1, further comprising instructions for:

caching, to the storage system, information relating to a determination that the data object corresponds to the selection criteria of the placement rule and the determined location of the data object in the hierarchical directory structure, the storage system being configured for using the cached information and not repeating the determination that the data object corresponds to the selection criteria and the determined location of the data object in the hierarchical directory structure.

6. The non-transitory computer readable medium of claim 1, further comprising instructions for:

receiving, through a graphical user interface, information to construct the selection criteria of the placement rule; and

receiving, through the graphical user interface, information to construct the placement action of the placement rule.

7. The non-transitory computer readable medium of claim 1, further comprising instructions for:

before transmitting the notification to the second gateway device, producing information that is descriptive of changes to the hierarchical directory structure resulting from updating the hierarchical directory structure to include the determined location.

8. A system comprising:

a processor;

a computer readable medium comprising instructions executable by the processor to cause the system to perform operations comprising,

receiving a placement rule comprising selection criteria and a placement action, the placement action comprising information relating to locations in a hierarchical directory structure to place data objects that corresponds to the selection criteria and the placement action comprising a configurable conflict resolution mechanism;

after receiving the placement rule, identifying a previous data object that was previously stored to the storage system and corresponds to the selection criteria of the placement rule; and

modifying a location in the hierarchical directory structure corresponding to the previous data object, the modified location being determined based on the placement action of the placement rule and at least one attribute of the previous data object;

applying the configurable conflict resolution mechanism to enable user-defined methods of resolving conflicts, when multiple clients write to the data object without locking the hierarchical directory structure;

receiving from an object store device, a notification comprising information indicating receipt of, at a first gateway device, a data object to be stored in a storage system, wherein the storage system comprises a distributed storage system comprising a plurality of administrative domains, wherein the placement rule is associated with one of the plurality of administrative domains;

determining that the data object corresponds to the selection criteria of the placement rule by;

determining whether the data object is associated with a rules identifier associated with the placement rule; and

evaluating the selection criteria, when the rules identifier is not present or upon determining the rules identifier is associated with a previous version of the placement rule, to determine whether the data object corresponds to the selection criteria of the placement rule;

determining that the data object is associated with the same administrative domain as the placement rule based, at least in part, on a determination that the data object corresponds to the selection criteria of the placement rule;

determining a location in the hierarchical directory structure to place the data object, the determined location being based on the placement action of the placement rule and at least one attribute of the data object; and

transmitting in accordance with the placement rule, to a second gateway device, a notification comprising information to enable the second gateway device to update a representation of the hierarchical directory structure to include the determined location by creating a new reference to the data object and according to said modifying.

9. The system of claim 8, wherein:

the selection criteria comprise a plurality of conditions; and

determining that the data object corresponds to the selection criteria of the placement rule comprises determining that the data object satisfies all of the plurality of conditions.

10. The system of claim 8, wherein the first gateway device is configured to provide a first file system representation of the hierarchical directory structure, the second gateway device is configured to provide a second file system representation of the hierarchical directory structure, and the data object is referenced by a directory of the first file system representation different than a directory of the second file system representation.

11. The system of claim 8, wherein the computer readable medium further comprises instructions executable by the processor cause the system to determine that the second gateway device is not accessible;

and

queue the notification for later transmission.

12. The system of claim 8, wherein receiving the data object at the first gateway device comprises placing the data object on a processing queue, wherein the computer readable medium further comprises instructions executable by the processor to cause the system to:

transfer the data object from the processing queue to an evaluation queue prior to a determination that the data object corresponds to the selection criteria of the placement rule.

13. The system of claim 8, wherein the computer readable medium further comprises instructions executable by the processor to cause the system to:

cache, to the storage system, information relating to the determination that the data object corresponds to the selection criteria of the placement rule and the determined location of the data object in the hierarchical directory structure, the storage system being configured for using the cached information and not repeating the determination that the data object corresponds to the selection criteria and the determined location of the data object in the hierarchical directory structure.

14. The system of claim 8, wherein the computer readable medium further comprises instructions executable by the processor to cause the system to:

receive, through a graphical user interface, information to construct the selection criteria of the placement rule; and

receive, through the graphical user interface, information to construct the placement action of the placement rule.

15. The system of claim 8, wherein the computer readable medium further comprises instructions executable by the processor to cause the system to:

before transmitting the notification to the second gateway device, produce information that is descriptive of changes to the hierarchical directory structure resulting from updating the hierarchical directory structure to include the determined location.