Resource efficient scale-out file systems

• US 9,916,258 B2
• Filed: 03/31/2011
• Issued: 03/13/2018
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Mitra, Soumyadeb, Hsu, Windsor W.
• Primary Examiner(s): Featherstone, Mark D
• Assistant Examiner(s): McQuitery, Diedra
• Application Number:
  US13/077,779
• Publication Number:
  US 20120254257A1
• Time in Patent Office: 2,539 Days
• CPC Class Codes: G06F 12/0846 Cache with multiple tag or data arrays being simultaneously accessible
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 16/17 Details of further file system functions
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: G06F 9/5022 Mechanisms to release resources

Abstract

According to one embodiment, a file system (FS) of a storage system is partitioned into a plurality of FS partitions, where each FS partition stores segments of data files. In response to an input and output (IO) request for accessing a first of the FS partitions, a second of the FS partitions is selected that is currently in a ready state for access. The second FS partition is then removed from the ready state for access. The first FS partition is brought into a ready state for access.

Claims

1. A computer-implemented method for providing a file system, the method comprising:

partitioning a file system (FS) of a storage system into a plurality of FS partitions by a FS partition manager executed by a processor of the storage system, each FS partition storing a plurality of files, wherein the plurality of FS partitions form a single logical file system using a single name space, wherein each of the FS partitions stores files having similar access patterns;

receiving an input and output (IO) request for accessing a first of the FS partitions, the first FS partition providing access to a first set of files;

in response to the IO request, selecting by the FS partition manager a second of the FS partitions that is currently in a ready state for access and that has no pending IO request for the second FS partition, the second FS partition providing access to a second set of files;

removing by the FS partition manager the second FS partition from the ready state for access, including unloading second FS metadata associated with the second FS partition from a system memory of the storage system, wherein the second FS metadata includes information for accessing the second set of files;

bringing the first FS partition into a ready state for access, including loading first FS metadata associated with the first FS partition into the system memory for servicing the IO request, wherein the first FS metadata includes information for accessing the first set of files, including

enqueuing the request in a pending IO queue of the file system if the first FS partition is not in the ready state for access,

scheduling an execution of the IO request if the first FS partition is in the ready state for access, and

incrementing a pending IO count associated with the first FS partition to indicate a number of IO requests currently pending for execution in the first FS partition; and

providing access to the first set of files via the first FS partition of the storage system and via the first FS metadata associated with the first FS partition loaded into the system memory, while the second FS metadata associated with the second FS partition is unloaded from the system memory, such that a memory space of the system memory required to store FS metadata is less than the system memory required to concurrently store both the first and the second FS metadata, wherein the first FS metadata includes information for accessing the first set of files, wherein the second FS metadata includes information for accessing the second set of files.

2. The method of claim 1, wherein the first FS metadata includes name and directory information for accessing the files associated with the first FS partition, and wherein the second FS metadata includes name and directory information for accessing the files associated with the second FS partition.

3. The method of claim 1, wherein removing the second FS partition from the ready state for access includes powering down a storage device associated with the second FS partition, and wherein bringing the first FS partition into the ready state for access includes powering up a storage device associated with the first FS partition.

4. The method of claim 1, wherein the files stored in each FS partition are deduplicated within the FS partition.

5. The method of claim 1, further comprising decrementing the pending IO count in response to a signal indicating that execution of the IO request has been completed.

6. The method of claim 1, further comprising:

assigning a token having an initial value to the first FS partition when the first FS partition is being loaded; and

decrementing the value of the token associated with the first FS partition in response to the signal indicating a complete execution of the IO request.

7. The method of claim 6, further comprising periodically decrementing the value of the token associated with the first FS partition.

8. The method of claim 7, further comprising removing the first FS partition from the ready state for access if at least one of the pending IO count and the value of the token reaches a predetermined value.

9. The method of claim 1, wherein the second FS partition is removed from the ready state for access only if there is no IO request pending for the second FS partition and the second FS partition has been in the ready state for access for a predetermined period of time.

10. The method of claim 1, wherein files with similar access patterns include files with similar last modified times such that FS partitions of the storage system are accessed sequential so relevant files from one partition is accessed before moving on to the next partition.

11. The method of claim 1, further comprising writing a file to a FS partition in a ready state, wherein the FS partition is selected based on a likely future access pattern of the file.

12. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform operations for managing resources of a file system, the operations comprising:

partitioning a file system (FS) of a storage system into a plurality of FS partitions by a FS partition manager executed by a processor of the storage system, each FS partition storing a plurality of files, wherein the plurality of FS partitions form a single logical file system using a single name space, wherein each of the FS partitions stores files having similar access patterns;

receiving an input and output (IO) request for accessing a first of the FS partitions, the first FS partition providing access to a first set of files;

in response to the IO request, selecting a second of the FS partitions that is currently in a ready state for access and that has no pending IO request for the second FS partition, the second FS partition providing access to a second set of files;

removing the second FS partition from the ready state for access, including unloading second FS metadata associated with the second FS partition from a system memory of the storage system, wherein the second FS metadata includes information for accessing the second set of files;

bringing the first FS partition into a ready state for access, including loading first FS metadata associated with the first FS partition into the system memory for servicing the IO request, wherein the first FS metadata includes information for accessing the first set of files, including

enqueuing the request in a pending IO queue of the file system if the first FS partition is not in the ready state for access,

scheduling an execution of the IO request if the first FS partition is in the ready state for access, and

incrementing a pending IO count associated with the first FS partition to indicate a number of IO requests currently pending for execution in the first FS partition; and

providing access to the first set of files via the first FS partition of the storage system and via the first FS metadata associated with the first FS partition loaded into the system memory, while the second FS metadata associated with the second FS partition is unloaded from the system memory, such that a memory space of the system memory required to store FS metadata is less than the system memory required to concurrently store both the first and the second FS metadata, wherein the first FS metadata includes information for accessing the first set of files, wherein the second FS metadata includes information for accessing the second set of files.

13. The computer-readable storage medium of claim 12, wherein the first FS metadata includes name and directory information for accessing the files associated with the first FS partition, and wherein the second FS metadata includes name and directory information for accessing the files associated with the second FS partition.

14. The computer-readable storage medium of claim 12, wherein removing the second FS partition from the ready state for access includes powering down a storage device associated with the second FS partition, and wherein bringing the first FS partition into the ready state for access includes powering up a storage device associated with the first FS partition.

15. The computer-readable storage medium of claim 12, wherein the files stored in each FS partition are deduplicated within the FS partition.

16. The computer-readable storage medium of claim 12, wherein the operations further comprise decrementing the pending IO count in response to a signal indicating that execution of the IO request has been completed.

17. The computer-readable storage medium of claim 12, wherein the operations further comprise:

assigning a token having an initial value to the first FS partition when the first FS partition is being loaded; and

decrementing the value of the token associated with the first FS partition in response to the signal indicating a complete execution of the IO request.

18. The computer-readable storage medium of claim 16, wherein the operations further comprise periodically decrementing the value of the token associated with the first FS partition.

19. The computer-readable storage medium of claim 18, wherein the operations further comprise removing the first FS partition from the ready state for access if at least one of the pending IO count and the value of the token reaches a predetermined value.

20. The computer-readable storage medium of claim 12, wherein the second FS partition is removed from the ready state for access only if there is no IO request pending for the second FS partition and the second FS partition has been in the ready state for access for a predetermined period of time.

21. A storage system, comprising:

a system memory;

a processor;

a file system (FS) executed in the system memory by the processor, the file system having a plurality of FS partitions, each FS partition storing a plurality of files, wherein the plurality of FS partitions form a single logical file system using a single name space, wherein each of the FS partitions stores files having similar access patterns;

a partition manager coupled to the file system and the system memory, wherein the partition manager is configured to

receive an input and output (IO) request for accessing a first of the FS partitions, the first FS partition providing access to a first set of files,

in response to the IO request, select a second of the FS partitions that is currently in a ready state for access and that has no pending IO request for the second FS partition, the second FS partition providing access to a second set of files,

remove the second FS partition from the ready state for access, including unloading second FS metadata associated with the second FS partition from a system memory of the storage system, wherein the second FS metadata includes information for accessing the second set of files,

bring the first FS partition into a ready state for access, including loading first FS metadata associated with the first FS partition into the system memory for servicing the IO request, wherein the first FS metadata includes information for accessing the first set of files, including enqueuing the request in a pending IO queue of the file system if the first FS partition is not in the ready state for access,

scheduling an execution of the IO request if the first FS partition is in the ready state for access, and

incrementing a pending IO count associated with the first FS partition to indicate a number of IO requests currently pending for execution in the first FS partition, and

provide access to the first set of files via the first FS partition of the storage system and via the first FS metadata associated with the first FS partition loaded into the system memory, while the second FS metadata associated with the second FS partition is unloaded from the system memory, such that a memory space of the system memory required to store FS metadata is less than the system memory required to concurrently store both the first and the second FS metadata, wherein the first FS metadata includes information for accessing the first set of files, wherein the second FS metadata includes information for accessing the second set of files.

22. The system of claim 21, wherein the first FS metadata includes name and directory information for accessing the files associated with the first FS partition, and wherein the second FS metadata includes name and directory information for accessing the files associated with the second FS partition.

23. The system of claim 21, wherein in removing the second FS partition from the ready state for access the partition manager is to power down a storage device associated with the second FS partition, and wherein in bringing the first FS partition into the ready state for access the partition manager is to power up a storage device associated with the first FS partition.

24. The system of claim 21, wherein the files stored in each FS partition are deduplicated within the FS partition.

25. The system of claim 21, wherein the partition manager is further to decrement the pending IO count in response to a signal indicating that execution of the IO request has been completed.

26. The system of claim 21, wherein the partition manager is further to:

assign a token having an initial value to the first FS partition when the first FS partition is being loaded, and

decrement the value of the token associated with the first FS partition in response to the signal indicating a complete execution of the IO request.

27. The system of claim 26, wherein the partition manager is further to periodically decrement the value of the token associated with the first FS partition.