Multiple node quota filter

• US 8,131,784 B1
• Filed: 09/26/2007
• Issued: 03/06/2012
• Est. Priority Date: 09/26/2007
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Zhuge, John, Mu, Paul Yuedong
• Primary Examiner(s): Xue, Belinda
• Application Number:
  US11/862,136
• Time in Patent Office: 1,623 Days
• Field of Search: None
• US Class Current: 707/823
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

An apparatus and a method that allow a quota to be implemented over a data storage system having data associated with the user stored in multiple volumes or directories across multiple storage servers. The quota limits a user or a group of users to a defined amount (e.g., quota) of storage resources of the storage system. The method may include monitoring input-output (IO) data access requests on each of the multiple storage servers, for example, using a pluggable quota filter on each of the storage servers, determining a total usage of storage resources for a particular user over the multiple storage servers, and limiting the user to the quota over the multiple storage servers.

Claims

1. A method, comprising:

monitoring, by a quota filter executing on a storage server, input-output (IO) data access requests, received at the storage server of a plurality of storage servers clustered in a data storage system, for a quota-sensitive data access request;

requesting, by the quota filter, a total usage of storage resources for a user across the plurality of storage servers, wherein said requesting comprises

performing a lookup in a local quota database that stores the total usage of storage resources for the user and a quota limit of an amount of storage resources allocated to the user,

wherein the local quota database is a cached copy of at least a portion of a remote quota database that stores amounts of storage resources and quota limits of amounts of storage resources allocated to a plurality of users across the plurality of storage servers clustered in the data storage system, and

wherein the remote quota database tracks amounts of storage resources used by the plurality of users across the plurality of storage servers clustered in the data storage system; and

limiting the user to the quota limit of storage resources across the plurality of storage servers based on the total usage of storage resources, wherein said limiting comprises determining, by the quota filter, that the quota-sensitive data access request would make the total usage of storage resources exceed the quota limit for the user, and allowing the user to exceed the quota limit by storing data in a directory designated as garbage collectable, wherein the data in the garbage collectable directory is moved or deleted according to a cleaning policy or moved between a high cost storage medium and a low cost storage medium under a hierarchical storage management scheme.

2. The method of claim 1, wherein limiting the user comprises limiting the user to the quota limit based on at least one of a volume set of a plurality of volumes of the storage system, or a directory set of a plurality of directories of the storage system.

3. The method of claim 2, wherein limiting the user to the quota limit based on the directory set comprises designating a directory as garbage collectable, and wherein the method further comprises:

performing an operation on the directory designated as garbage collectable according to the cleaning policy.

4. A storage server of a cluster of storage servers in a data storage system, comprising:

a processor;

a first communication interface through which to communicate with a quota manager that manages a total usage of storage resources by a user across a plurality of storage servers using a remote quota database, wherein the remote quota database stores amounts of storage resources and quota limits for a plurality of users across the plurality of storage servers clustered in the data storage system;

a second communication interface through which to communicate with a client processing system coupled to the storage server;

a third communication interface through which to communicate with a plurality of storage devices; and

a memory storing instructions which configured the processor to execute a quota filter to limit a user to a quota limit of storage resources in the data storage system wherein the quota filter is configured to perform the following operations;

monitoring input-output (IO) data access requests received at the storage server from the client to check for a quota-sensitive data access request;

performing a lookup in a local quota database that stores a total usage of storage resources for the user across the plurality of storage server and the quota limit of an amount of storage resources allocated to the user, wherein the local quota database is a cached copy of at least a portion of the remote quota database;

determining that the quota-sensitive data access request would make the total usage of storage resources exceed the quota limit for the user; and

allowing the user to exceed the quota limit by storing data in a directory designated as garbage collectable, wherein the data in the garbage collectable directory is moved or deleted according to a cleaning policy or moved between a high cost storage medium and a low cost storage medium under a hierarchical storage management scheme.

5. The storage server of claim 4, further comprising:

determining a local usage of storage resources on the storage server by the user;

sending the local usage of storage resources to the quota manager.

6. The storage server of claim 4, wherein the quota-sensitive data access request comprises at least one of creating a new file, deleting an existing file, or writing to an existing file to append the data of the existing file.

7. The storage server of claim 4, wherein the quota filter is a protocol layer filter.

8. The storage server of claim 4, further comprising additional instructions which configured the processor to perform the following:

deleting the directory designated as garbage collectable according to the cleaning policy.

9. A method, comprising:

setting a quota policy for a user for a set of logical data containers across a plurality of storage servers clustered in a data storage system;

storing the quota policy in a first quota data store that stores quota limits for a plurality of users across the plurality of storage servers clustered in the data storage system, wherein each quota limit is an amount of storage resources allocated to each user;

maintaining a total usage of storage resources by the user in a second quota data store, which is accessible by each of the plurality of storage servers, wherein the second quota data store stores amounts of storage resources used by the plurality of users across the plurality of storage servers clustered in the data storage system;

receiving information regarding a quota-sensitive data access request received at a quota filter executing on a particular storage server of the plurality of storage servers;

performing, by the quota filter, a lookup in a local, cached copy of at least a portion of the second data store to determine a total amount of resources used by the user across the plurality of storage servers clustered in the data storage system, wherein the local, cached copy is stored in the particular storage server;

calculating, by the quota filter, affected storage resources based on the received information and the total amount of resources used by the user;

comparing, by the quota filter, the affected storage resources against the quota policy; and

allowing the user to exceed the quota limit by storing data in a directory designated as garbage collectable, wherein the data in the garbage collectable directory is moved or deleted according to a cleaning policy or moved between a high cost storage medium and a low cost storage medium under a hierarchical storage management scheme.

10. The method of claim 9, wherein the quota policy comprises at least one of a block quota, a storage server quota, a file quota, or an inode quota.

11. The method of claim 9, wherein said initiating execution of the system rule comprises sending a command to execute the system rule that denies the quota-sensitive data access request that would cause the quota policy to be violated.

12. A quota management system, comprising:

a computing device having multiple communication interfaces, wherein each communication interface communicates with corresponding ones of a plurality of storage servers clustered in a data storage system, wherein the computing device comprises

a quota data store for maintaining a total usage of storage resources by a user across the plurality of storage servers clustered in the data storage system and a quota limit of an amount of storage resources allocated to the user; and

a quota manager, executing on the computing device, configured to receive from a quota filter executing on a particular storage server information regarding a quota-sensitive data access request directed to the particular storage server; and

the plurality of storage servers clustered in the data storage system each communicatively coupled to the quota management system, wherein at least one of the plurality of storage servers executes the quota filter configured to perform the following operations;

monitoring input-output (IO) data access requests received at the particular storage server from a client processing system to detect the quota-sensitive data access request;

performing a lookup in a local, cached copy of at least a portion of the quota data store to determine the total usage of storage resources for the user and the quota limit allocated to the user;

determining that the quota-sensitive data access request would make the total usage of storage resources exceed the quota limit for the user; and

allowing the user to exceed the quota limit by storing data in a directory designated as garbage collectable, wherein the data in the garbage collectable directory is moved or deleted according to a cleaning policy or moved between a high cost storage medium and a low cost storage medium under a hierarchical storage management scheme.

13. The quota management system of claim 12, wherein the quota manager is configured to maintain the total usage by determining local usage of storage resources by the user on each of the plurality of storage servers, and summing the local usage on the plurality of storage servers to determine the total usage of storage resources by the user.

14. The quota management system of claim 13, wherein the quota manager determines the local usage by accessing a local accounting data store corresponding to a logical data container on each of the plurality of storage servers.

15. The quota management system of claim 12, wherein the quota manager maintains the total usage of storage resources over a logical data container set, including a plurality of logical data containers across the plurality of storage servers.

16. The quota management system of claim 15, wherein the logical data container set is at least one of a volume set having multiple volumes across the plurality of storage servers or a directory set having multiple directories across the plurality of storage servers.

17. The quota management system of claim 15, wherein each of the logical data containers corresponds to a file system and comprises local volume accounting, and wherein the quota manager is configured to use the local volume accounting to determine the total usage of storage resources for the user across the plurality of storage servers.