Time-based data partitioning

• US 9,619,474 B2
• Filed: 03/31/2011
• Issued: 04/11/2017
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Mitra, Soumyadeb, Hsu, Windsor W.
• Primary Examiner(s): Lin, Shew-Fen
• Assistant Examiner(s): McQuitery, Diedra
• Application Number:
  US13/077,775
• Publication Number:
  US 20120254174A1
• Time in Patent Office: 2,203 Days
• Field of Search: None
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 12/121 using replacement algorithms
• CPC Class Codes: G06F 12/126 with special data handling, e.g. priority of data or instructions, handling errors or pinning
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/18 File system types
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 2212/1028 Power efficiency

Abstract

According to one embodiment, a file system (FS) of a storage system is partitioned into a plurality of FS partitions, where each FS partition stores segments of data files. In response to a request for writing a file to the storage system, the file is stored in a first of the FS partitions that is selected based on a time attribute of the file, such that files having similar time attributes are stored in an identical FS partition.

Claims

1. A computer-implemented method for providing a file system, the method comprising:

providing a plurality of FS (file system) partitions of an FS of a storage system, wherein each of the FS partitions is associated with a specific time period, including a first FS partition corresponding to a first time period and a second FS partition corresponding to a second time period;

in response to a request for writing a file to the storage system, examining a last modified time attribute of the file to determine that the last modified time attribute of the file is associated with the first time period;

storing the file in the first FS partition, such that files having similar time attributes are stored in an identical FS partition, wherein the first FS partition includes first FS metadata having name and directory information for accessing a first set of files each having a time attribute associated with the first time period, wherein the second FS partition includes second FS metadata having name and directory information for accessing a second set of files each having a time attribute associated with the second time period, and wherein the first FS metadata of the first FS partition can be independently loaded in a memory of the storage system for accessing the first set of files stored in the first FS partition, without having to load the second FS metadata the second FS partition in the memory to reduce a memory usage requirement;

determining that data stored in the first FS partition has reached a predetermined percentage of a storage capacity of the first FS partition;

allocating a third FS partition as an active FS partition to store subsequent files, while transforming the first FS partition into a read-only FS partition as an archive FS partition;

scanning a plurality of files stored in the first FS partition to identify a third set of files having a last modified time attribute within a third period of time; and

migrating the third set of files to a fourth FS partition, such that the fourth FS partition stores similar files having the last modified time attribute within the third period of time.

2. The method of claim 1, further comprising:

determining that the second FS partition does not have a pending input/output (IO) request pending for accessing files stored in the second FS partition;

unloading the second FS metadata of the second FS partition from a system memory of the storage system;

loading the first FS metadata of the first FS partition into the system memory for accessing files stored in the first FS partition.

3. The method of claim 2, further comprising:

assigning a token having an initial value to the first FS partition when the first FS partition is being loaded; and

decrementing the token associated with the first FS partition in response to a completion of each IO request accessing data stored in the first FS partition.

4. The method of claim 3, further comprising periodically decrementing the token associated with the first FS partition.

5. The method of claim 4, further comprising unloading the first FS metadata of the first FS partition from the system memory when the token drops below a predetermined threshold.

6. The method of claim 1, further comprising:

determining whether the first FS partition is in a ready state;

enqueuing the request in a pending IO queue of the file system in response to determining that the first FS partition is not in a ready state; and

incrementing a pending IO count associated with the first FS partition to indicate a number of IO requests currently pending for execution in the first FS partition.

7. The method of claim 6, further comprising decrementing the pending IO count in response to a completion of an execution of an IO request.

8. The method of claim 1, further comprising:

removing a second of the FS partitions from a ready state for access when the second FS partition is not being accessed; and

bringing in a third of the FS partitions into a ready state for access when the third FS partition is to be accessed.

9. The method of claim 8, wherein removing the second FS partition from the ready state for access includes unloading metadata associated with the second FS partition from a system memory of the storage system, the system memory being shared amongst the FS partitions, and wherein bringing the first FS partition into the ready state for access includes loading metadata associated with the first FS partition into the system memory.

10. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform operations of managing resources of a file system, the operations comprising:

providing a plurality of FS (file system) partitions of an FS of a storage system, wherein each of the FS partitions is associated with a specific time period, including a first FS partition corresponding to a first time period and a second FS partition corresponding to a second time period;

in response to a request for writing a file to the storage system, examining a last modified time attribute of the file to determine that the last modified time attribute of the file is associated with the first time period;

storing the file in the first FS partition, such that files having similar time attributes are stored in an identical FS partition, wherein the first FS partition includes first FS metadata having name and directory information for accessing a first set of files each having a time attribute associated with the first time period, wherein the second FS partition includes second FS metadata having name and directory information for accessing a second set of files each having a time attribute associated with the second time period, and wherein the first FS metadata of the first FS partition can be independently loaded in a memory of the storage system for accessing the first set of files stored in the first FS partition, without having to load the second FS metadata the second FS partition in the memory to reduce a memory usage requirement;

determining that data stored in the first FS partition has reached a predetermined percentage of a storage capacity of the first FS partition;

allocating a third FS partition as an active FS partition to store subsequent files, while transforming the first FS partition into a read-only FS partition as an archive FS partition;

scanning a plurality of files stored in the first FS partition to identify a third set of files having a last modified time attribute within a third period of time; and

migrating the third set of files to a fourth FS partition, such that the fourth FS partition stores similar files having the last modified time attribute within the third period of time.

11. The computer-readable storage medium of claim 10, wherein the operations comprise:

determining that the second FS partition does not have a pending input/output (IO) request pending for accessing files stored in the second FS partition;

unloading the second FS metadata of the second FS partition from a system memory of the storage system;

loading the first FS metadata of the first FS partition into the system memory for accessing files stored in the first FS partition.

12. The computer-readable storage medium of claim 11, wherein the operations further comprise:

assigning a token having an initial value to the first FS partition when the first FS partition is being loaded; and

decrementing the token associated with the first FS partition in response to a completion of each IO request accessing data stored in the first FS partition.

13. The computer-readable storage medium of claim 11, wherein the operations further comprise periodically decrementing the token associated with the first FS partition.

14. The computer-readable storage medium of claim 13, wherein the operations further comprise unloading the first FS metadata of the first FS partition from the system memory when the token drops below a predetermined threshold.

15. The computer-readable storage medium of claim 10, wherein the operations further comprise:

determining whether the first FS partition is in a ready state;

enqueuing the request in a pending IO queue of the file system in response to determining that the first FS partition is not in a ready state; and

incrementing a pending IO count associated with the first FS partition to indicate a number of IO requests currently pending for execution in the first FS partition.

16. The computer-readable storage medium of claim 15, wherein the operations further comprise decrementing the pending IO count in response to a completion of an execution of an IO request.

17. The computer-readable storage medium of claim 10, wherein the operations further comprise:

removing a second of the FS partitions from a ready state for access when the second FS partition is not being accessed; and

bringing in a third of the FS partitions into a ready state for access when the third FS partition is to be accessed.

18. The computer-readable storage medium of claim 17, wherein removing the second FS partition from the ready state for access includes unloading metadata associated with the second FS partition from a system memory of the storage system, the system memory being shared amongst the FS partitions, and wherein bringing the first FS partition into the ready state for access includes loading metadata associated with the first FS partition into the system memory.

19. A storage system, comprising:

a system memory;

a file system (FS) executed in the system memory;

a partition manager coupled to the file system and the system memory, wherein the partition manager is configured to perform operations, the operations including

providing a plurality of FS partitions of the FS, wherein each of the FS partitions is associated with a specific time period, including a first FS partition corresponding to a first time period and a second FS partition corresponding to a second time period,

in response to a request for writing a file to the storage system, examining a last modified time attribute of the file to determine that the last modified time attribute of the file is associated with the first time period,

storing the file in the first FS partition, such that files having similar time attributes are stored in an identical FS partition, wherein the first FS partition includes first FS metadata having name and directory information for accessing a first set of files each having a time attribute associated with the first time period, wherein the second FS partition includes second FS metadata having name and directory information for accessing a second set of files each having a time attribute associated with the second time period, and wherein the first FS metadata of the first FS partition can be independently loaded in a memory of the storage system for accessing the first set of files stored in the first FS partition, without having to load the second FS metadata the second FS partition in the memory to reduce a memory usage requirement,

determining that data stored in the first FS partition has reached a predetermined percentage of a storage capacity of the first FS partition,

allocating a third FS partition as an active FS partition to store subsequent files, while transforming the first FS partition into a read-only FS partition as an archive FS partition,

scanning a plurality of files stored in the first FS partition to identify a third set of files having a last modified time attribute within a third period of time, and

migrating the third set of files to a fourth FS partition, such that the fourth FS partition stores similar files having the last modified time attribute within the third period of time.

20. The system of claim 19, wherein the operations further comprise:

determining that the second FS partition does not have a pending input/output (IO) request pending for accessing files stored in the second FS partition;

unloading the second FS metadata of the second FS partition from a system memory of the storage system;

loading the first FS metadata of the first FS partition into the system memory for accessing files stored in the first FS partition.

21. The system of claim 20, wherein the operations further comprise:

assigning a token having an initial value to the first FS partition when the first FS partition is being loaded; and

decrementing the token associated with the first FS partition in response to a completion of each IO request accessing data stored in the first FS partition.

22. The system of claim 21, wherein the operations further comprise periodically decrementing the token associated with the first FS partition.