Method and apparatus for tiered storage

• US 20110246491A1
• Filed: 04/01/2010
• Published: 10/06/2011
• Est. Priority Date: 04/01/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Avere Systems, Inc. (Microsoft Corporation)
• Inventors: Hosteny, Joseph IV, Butler, Jeffrey, Boyles, John R., Clash, Daniel, Kazar, Michael L., Nydick, Daniel S.
• Granted Patent: US 9,342,528 B2
• Time in Patent Office: Days
• US Class Current: 707/755
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/163 Server or database system
• CPC Class Codes: G06F 2212/60 Details of cache memory
• CPC Class Codes: G06F 2212/62 Details of cache specific to multiprocessor cache arrangements
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system for storing file data and directory data received over a network includes a network interface in communication with the network which receives NAS requests containing data to be written to files from the network. The system includes a first type of storage. The system includes a second type of storage different from the first type of storage. The system includes a policy specification n which specifies a first portion of one or more files'"'"' data which is less than all of the files'"'"' data is stored in the first type of storage and a second portion of the data which is less than all of the data of the files is stored in the second type of store. The system comprises a processing unit which executes the policy and causes the first portion to be stored in the first type of storage and a second portion to be stored in the second type of storage. A method for storing file data and directory data received over a network.

Claims

1. A system for storing file data and directory data received over a network comprising:

a network interface in communication with the network which receives NAS requests containing data to be written to files from the network;

a first type of storage;

a second type of storage different from the first type of storage;

a policy specification which specifies a first portion of one or more files'"'"' data which is less than all of the files'"'"' data is stored in the first type of storage and a second portion of the data which is less than all of the data of the files is stored in the second type of store; and

a processing unit which executes the policy and causes the first portion to be stored in the first type of storage and a second portion to be stored in the second type of storage.

2. A system as described in claim 1 where the policy specification is stored in a file within the first type of storage or the second type of storage.

3. A system as described in claim 1 including a management database outside of the first type of storage or the second type of storage where the policy specification is stored in the management database.

4. The system as described in claim 1 wherein the policy specification specifies the first portion and the second portion of one or more files'"'"' data, where either portion may include file system meta-data.

5. The system as described in claim 4 wherein the policy specification for the first portion or the second portion of one or more files'"'"' data includes meta-data containing block addressing information.

6. The system as described in claim 1 including a buffer module having buffers and which reads and writes data into the buffers.

7. The system as described in claim 6 including an inode attribute manager which updates attributes in an inode.

8. The system as described in claim 7 including a directory manager which treats a directory object as a set of mappings between file names and inode identifiers.

9. The system as described in claim 7 including a data manager which copies data to and from the buffers.

10. The system as described in claim 9 including an inode object allocator which allocates inodes and tags them with policy specifications.

11. A system as described in claim 10 including a NAS server operations module which receives incoming NAS requests and invokes local storage operations interfacing with the inode attribute manager, the directory manager, the data manager and the inode object allocator for reading and writing file and directory attributes, reading and writing file data, and performing directory operations.

12. The system as described in claim 10 including a NAS cache operations module which acts as a cache of data storage in one or external NFS servers and which creates and maintains cached versions of actively accessed directories and files stored at the external NFS server.

13. A system as described in claim 1 wherein at least two files are stored within a directory, where a Block Allocator allocates some blocks to a first file in the directory from a first type of storage, and allocates some blocks to a second file in the directory from a second type of storage different from the first type of storage.

14. A system as described in claim 13 including a Block Allocator that allocates blocks for a file from a first type of storage and additional blocks for the same file from a second type of storage different from the first type of storage.

15. A system as described in claim 14 where the Block Allocator determines the blocks to be allocated to a file from a first type of storage, and the blocks to be allocated from a second type of storage different from the first type of storage, based upon the policy associated with the file.

16. A method for storing file data and directory data received over a network comprising the steps of:

receiving NAS requests containing data to be written to files from the network at a network interface;

executing with a processing unit the policy specification which specifies a first portion of one or more files'"'"' data which is less than all of the files'"'"' data is stored in a first type of storage and a second portion of the data which is less than all of the data of the files stored in the second type of data which is different from the first type of storage; and

causing with the processing unit the first portion to be stored in the first type of storage and the second portion to be stored in the second type of storage.

17. The method as described in claim 16 including the step of writing data to a file located in a directory which is a policy root directory and has the policy specification.

18. The method as described in claim 17 including the steps of looking up a subdirectory D, which is a policy root directory having an associated policy specification, and associating the policy specification with subdirectory D.

19. The method as described in claim 18 including the step of looking up a subdirectory D of a parent directory having an associated policy specification, and associating the parent directory'"'"'s policy specification with subdirectory D.

20. The method as described in claim 19 including the step of looking up a file within a directory having an associated policy specification, and associating the directory'"'"'s policy specification with file F.

21. The method as described in claim 20 including the step of writing data to a file F located in a directory having an associated policy specification, and the step of the Inode Space Allocator allocating space to hold the written data according to the associated policy specification.

22. The method as described in claim 18 wherein the looking up directory D step includes the steps of consulting a cache of a management database, determining that the directory D is a policy root, and augmenting directory D'"'"'s file handle with a policy tag indicating a policy to be applied to files under directory D in the file system'"'"'s namespace.

23. The method as described in claim 22 including the step of looking up a subdirectory D of a parent directory having an associated policy specification, and associating the parent directory'"'"'s policy specification with subdirectory D.

24. The method as described in claim 23 including the step of looking up a file within a directory having an associated policy specification, and associating the directory'"'"'s policy specification with file F.

25. The method as described in claim 24 including the steps of receiving the lookup for file F within directory D, and propagating the policy tag from directory D'"'"'s file handle to file F'"'"'s file handle.

26. The method as described in claim 25 including the step of storing the data in buffers of a memory until a buf clean module collects modified data to write the modified data to persistent storage;

determining with the buf clean module which portions of the file F have new data; and

calling from the buf clean module into an inode space allocator to ensure that blocks have been allocated in persistent storage for an appropriate block range in the file F.

27. The method as described in claim 26 including the step of identifying with the inode space allocator a policy associated with the file F from the policy tag of file F'"'"'s file handle;

calling a block allocator module with the inode space allocator;

obtaining with the block allocator module storage blocks of an appropriate type in persistent storage; and

inserting with the inode space allocator block addresses associated with the appropriate storage into an indirect block tree associated with a file F.

28. The method as described in claim 27 including the step of starting IO operations with the buf clean module from buffers in memory having respective data written to the appropriate storage blocks in persistent storage.

29. A system for storing file data and directory data received over a network comprising:

a network interface in communication with the network which receives the NAS requests from the network, including the NAS requests containing data to be written to files;

a first type of storage;

a second type of storage different from the first type of storage;

a policy specification which specifies a first portion of one or more directories'"'"' data which is less than all of the directories'"'"' data is stored in the first type of storage and a second portion of the data which is less than all of the data of the directories is stored in the second type of store; and

a processing unit which executes the policy and causes the first portion to be stored in the first type of storage and a second portion to be stored in the second type of storage.