Filesystem-Aware Block Storage System, Apparatus, and Method

• US 20070266037A1
• Filed: 05/03/2007
• Published: 11/15/2007
• Est. Priority Date: 11/05/2004
• Status: Active Grant

• Current Assignee: Storcentric DIP Lender LLC
• Original Assignee: Data Robotics Incorporated
• Inventors: Barrall, Geoffrey, Clarkson, Neil, Terry, Julian
• Granted Patent: US 7,873,782 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1084 Degraded mode, e.g. caused by single or multiple storage removals or disk failures
• CPC Class Codes: G06F 11/1096 Parity calculation or recalculation after configuration or reconfiguration of the system
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2087 with a common controller
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/325 by lamps or LED's
• CPC Class Codes: G06F 2211/1004 Adaptive RAID, i.e. RAID system adapts to changing circumstances, e.g. RAID1 becomes RAID5 as disks fill up
• CPC Class Codes: G06F 2211/1023 Different size disks, i.e. non uniform size of disks in RAID systems with parity
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G11B 2220/415 Redundant array of inexpensive disks [RAID] systems
• CPC Class Codes: G11B 2220/65 wherein solid state memory is used for storing indexing information or metadata
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/329 on a disc [VTOC]

Abstract

A filesystem-aware storage system locates and analyzes host filesystem data structures in order to determine storage usage of the host filesystem. To this end, the storage system might locate an operating system partition, parse the operating system partion to locate its data structures, and parse the operating system data structures to locate the host filesystem data structures. The storage system manages data storage based on the storage usage of the host file system. The storage system can use the storage usage information to identify storage areas that are no longer being used by the host filesystem and reclaim those areas for additional data storage capacity. Also, the storage system can identify the types of data stored by the host filesystem and manage data storage based on the data types, such as selecting a storage layout and/or an encoding scheme for the data based on the data type.

Claims

1. A method of managing storage space by a block-level storage system that stores data under control of a host filesystem, the method comprising:

storing data for the host filesystem in the block-level storage system, the data including host filesystem data structures;

locating the host filesystem data structures stored in the block-level storage system;

analyzing the host filesystem data structures to determine storage usage of the host filesystem; and

managing physical storage based on the storage usage of the host filesystem.

2. A method according to claim 1, wherein locating host filesystem data structures in storage comprises:

maintaining a partition table;

parsing the partition table to locate an operating system partition;

parsing the operating system partition to identify the operating system and locate operating system data structures; and

parsing the operating system data structures to identify the host filesystem and locate the host filesystem data structures.

3. A method according to claim 2, wherein the operating system data structures include a superblock, and wherein parsing the operating system data structures includes parsing the superblock.

4. A method according to claim 2, wherein the host filesystem data structures include at least one free block bitmap, and wherein analyzing the host filesystem data structures includes parsing the at least one free block bitmap.

5. A method according to claim 2, wherein analyzing the host filesystem data structures comprises:

making a working copy of a host filesystem data structure; and

parsing the working copy.

6. A method according to claim 1, wherein managing physical storage based on the storage usage of the host filesystem comprises:

identifying a storage area no longer being used by the host filesystem; and

reclaiming the storage area for subsequent data storage.

7. A method according to claim 6, wherein reclaiming the storage area comprises:

determining a logical sector address for the storage area;

identifying a cluster access table entry for the storage area based on the logical sector address;

decrementing a reference count associated with the cluster access table entry; and

freeing the storage area if the decremented reference count is zero.

8. A method according to claim 6, wherein identifying a storage area no longer being used by the host filesystem comprises:

identifying a free block bitmap associated with the host filesystem; and

parsing the free block bitmap to identify clusters that are no longer being used by the filesystem.

9. A method according to claim 8, wherein identifying the free block bitmap associated with the host filesystem comprises:

maintaining a partition table;

parsing the partition table to locate an operating system partition;

parsing the operating system partition to locate a superblock; and

identifying the host filesystem based on the superblock.

10. A method according to claim 8, further comprising:

making a working copy of the free block bitmap, wherein parsing the free block bitmap to identify clusters that are no longer being used by the filesystem comprises parsing the working copy of the free block bitmap to identify clusters that are no longer being used by the filesystem.

11. A method according to claim 1, wherein managing physical storage based on the storage usage of the host filesystem comprises:

identifying a data type associated with data provided by the host filesystem, and

storing the data using a storage scheme selected based on the data type, whereby data having different data types can be stored using different storage schemes selected based on the data types.

12. A method according to claim 11, wherein storing the data using a storage scheme selected based on the data type comprises:

storing the data using a storage layout selected based on the data type.

13. A method according to claim 12, wherein storing the data using a storage layout selected based on the data type comprises:

storing frequently accessed data so as to provide enhanced accessibility.

14. A method according to claim 13, wherein storing frequently accessed data so as to provide enhanced accessibility comprises:

storing the frequently accessed data uncompressed and in sequential storage.

15. A method according to claim 12, wherein storing the data using a storage layout selected based on the data type comprises:

storing infrequently accessed data so as to provide enhanced storage efficiency.

16. A method according to claim 15, wherein storing infrequently accessed data so as to provide enhanced storage efficiency comprises:

storing the infrequently accessed data using at least one of data compression and non-sequential storage.

17. A method according to claim 11, wherein storing the data using a storage scheme selected based on the data type comprises:

storing the data using an encoding scheme selected based on the data type.

18. A method according to claim 17, wherein the encoding scheme comprises at least one of:

data compression; and

encryption.

19. A method according to claim 1, wherein managing physical storage based on the storage usage of the host filesystem comprises:

maintaining information regarding usage of the physical storage; and

comparing the storage usage of the host filesystem to the information regarding usage of the physical storage.

20. A method according to claim 1, wherein such analyzing is performed at least in part asynchronously with respect to storage requests.

21. A method according to claim 20, wherein such asynchronous analyzing comprises:

storing information regarding the storage requests; and

subsequently processing the stored information to determine the storage usage of the host filesystem.

22. A method according to claim 21, wherein the information is stored using at least one of:

a queue; and

a bitmap corresponding to a bitmap of the host filesystem.

23. A method according to claim 20, wherein a frequency of such asynchronous analyzing is selected based on at least one of available storage space and storage system load.

24. A method according to claim 20, wherein a frequency of such asynchronous analyzing is selected so as to allow sufficient time for host filesystem metadata to be updated to reflect a storage request.

25. A method according to claim 20, wherein such analyzing is performed partly synchronously and partly asynchronously with respect to storage requests.

26. A method according to claim 1, wherein such analyzing is performed purely synchronously with respect to storage requests.

27. A block-level storage system that stores data under control of a host filesystem, the system comprising:

a block-level storage in which data is stored for the host filesystem, the data including host filesystem data structures; and

a storage controller operably coupled to the block-level storage for storing the data including the host filesystem data structures, locating the host filesystem data structures stored in the block-level storage system, analyzing the host filesystem data structures to determine storage usage of the host filesystem and managing physical storage based on the storage usage of the host filesystem.

28. A system according to claim 27, wherein the storage controller is operably coupled to maintain a partition table, parse the partition table to locate an operating system partition, parse the operating system partition to identify the operating system and locate operating system data structures, parse the operating system data structures to identify the host filesystem and locate the host filesystem data structures, and parse the host filesystem data structures to determine storage usage of the host filesystem.

29. A system according to claim 28, wherein the operating system data structures include a superblock, and wherein the storage processor is operably coupled to parse the superblock.

30. A system according to claim 28, wherein the host filesystem data structures include at least one free block bitmap, and wherein the storage controller is operably coupled to parse the at least one free block bitmap.

31. A system according to claim 28, wherein the storage controller is operably coupled to make a working copy of a host filesystem data structure and parse the working copy.

32. A system according to claim 27, wherein the storage controller is operably coupled to identify a storage area no longer being used by the host filesystem and reclaim the storage area for subsequent data storage.

33. A system according to claim 32, wherein the storage controller is operably coupled to identify determine a logical sector address for the storage area, identify a cluster access table entry for the storage area based on the logical sector address, decrement a reference count associated with the cluster access table entry, and free the storage area if the decremented reference count is zero.

34. A system according to claim 32, wherein the storage controller is operably coupled to identify a free block bitmap associated with the host filesystem and parse the free block bitmap to identify clusters that are no longer being used by the filesystem.

35. A system according to claim 34, wherein the storage controller is operably coupled to maintain a partition table, parse the partition table to locate an operating system partition, parse the operating system partition to locate a superblock, and identify the host filesystem based on the superblock.

36. A system according to claim 34, wherein the storage controller is operably coupled to make a working copy of the free block bitmap and parse the working copy of the free block bitmap to identify clusters that are no longer being used by the filesystem.

37. A system according to claim 27, wherein the storage controller is operably coupled to identify a data type associated with data provided by the host filesystem and store the data using a storage scheme selected based on the data type, whereby data having different data types can be stored using different storage schemes selected based on the data types.

38. A system according to claim 37, wherein the storage controller is operably coupled to select a storage layout for the data based on the data type and store the data using the selected storage layout.

39. A system according to claim 38, wherein the storage controller is operably coupled to store frequently accessed data so as to provide enhanced accessibility.

40. A system according to claim 39, wherein the storage controller is operably coupled to store the frequently accessed data uncompressed and in sequential storage.

41. A system according to claim 38, wherein the storage controller is operably coupled to store infrequently accessed data so as to provide enhanced storage efficiency.

42. A system according to claim 41, wherein the storage controller is operably coupled to store the infrequently accessed data using at least one of data compression and non-sequential storage.

43. A system according to claim 37, wherein the storage controller is operably coupled to store the data using an encoding scheme selected based on the data type.

44. A system according to claim 43, wherein the encoding scheme comprises at least one of:

data compression; and

encryption.

45. A system according to claim 27, wherein the storage controller is operably coupled to maintain information regarding usage of the physical storage and compare the storage usage of the host filesystem to the information regarding usage of the physical storage.

46. A system according to claim 27, wherein such analyzing is performed at least in part asynchronously with respect to storage requests.

47. A system according to claim 46, wherein such asynchronous analyzing comprises:

storing information regarding the storage requests; and

subsequently processing the stored information to determine the storage usage of the host filesystem.

48. A system according to claim 47, wherein the information is stored using at least one of:

a queue; and

a bitmap corresponding to a bitmap of the host filesystem.

49. A system according to claim 46, wherein a frequency of such asynchronous analyzing is selected based on at least one of available storage space and storage system load.

50. A system according to claim 46, wherein a frequency of such asynchronous analyzing is selected so as to allow sufficient time for host filesystem metadata to be updated to reflect a storage request.

51. A system according to claim 46, wherein such analyzing is performed partly synchronously and partly asynchronously with respect to storage requests.

52. A system according to claim 27, wherein such analyzing is performed purely synchronously with respect to storage requests.