System and method of handling file metadata

• US 20060248038A1
• Filed: 04/29/2005
• Published: 11/02/2006
• Est. Priority Date: 04/29/2005
• Status: Active Grant

• Current Assignee: Overland Storage Incorporated (Silicon Valley Technology Partners LLC)
• Original Assignee: Overland Storage Incorporated (Silicon Valley Technology Partners LLC)
• Inventors: Kaplan, Marc, Burrows, Dan, Wijayaratne, Ravi, Osborn, Kevin
• Granted Patent: US 7,454,406 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/164 File meta data generation
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99939 Privileged access
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A network storage system comprises data storage, one or more file system APIs, and a metadata handler. The data storage can comprise file data and associated metadata. The file system APIs can be configured to read and write file data and metadata to and from the data storage. The metadata handler can have a library of functions for handling the metadata. The library can include a metadata blob creation function and a metadata blob extraction function. The metadata blob creation function cooperates with the file system APIs to read source metadata associated with specified file data and creates and populates a metadata blob from which a substantial copy of the source metadata can be generated. The metadata blob extraction function receives at least a metadata blob, extracts information from the metadata blob, and cooperates with the file system APIs to generate destination metadata, a substantial copy of the source metadata.

Claims

1. A method of accessing a data file and associated metadata from data storage, the method comprising:

receiving a file access request for a file;

accessing the the file and metadata associated with the file from data storage; and

generating a metadata object that encapsulates information from which the metadata associated with the file can be substantially restored, wherein the metadata object has a format different from the retrieved metadata'"'"'s native format.

2. The method of claim 1, wherein the metadata object is a binary large object.

3. The method of claim 1, wherein generating a metadata object comprises populating at least a portion of the metadata object with information corresponding to metadata fields used by the retrieved metadata'"'"'s native file system.

4. The method of claim 3, wherein populating at least a portion of the metadata object comprises determining a native file system of the retrieved metadata and populating a portion of the metadata object configured to store the metadata fields used by the retrieved metadata'"'"'s native file system.

5. The method of claim 1, wherein generating a metadata object comprises generating a metadata object configured to be stored on a file system different from the retrieved metadata'"'"'s native file system without requiring the storing file system to be able to read underlying metadata stored in the metadata object.

6. An apparatus for accessing a data file and associated metadata, the apparatus comprising:

at least one processor configured to access a requested file and metadata associated with the file; and

at least one processor configured to generate a metadata object that encapsulates information from which the metadata can be substantially restored, wherein the metadata object has a format different from the native format of the metadata.

7. The apparatus of claim 6, wherein the processor configured to access the requested file and metadata and the processor configured to generate a metadata object are the same processor.

8. The apparatus of claim 6, wherein the metadata object is a binary large object.

9. The apparatus of claim 6, wherein at least one of the processors is configured to perform at least one of its functions by executing a metadata handler configured to communicate with data storage configured to store file data and metadata associated with the file data, at least a first portion of the metadata being metadata native to a first file system and at least a second portion of the metadata being metadata native to a second file system, the metadata handler having a library of functions configured to handle at least the first portion of metadata and the second portion of metadata, the library comprising:

a metadata object creation function configured to, when called by a first calling application, receive, from the data storage, source metadata associated with specified file data, and to create a metadata object and to populate at least a portion of the metadata object with information from which the source metadata can be substantially restored, the populated portion corresponding to metadata fields used by the source metadata'"'"'s native file system; and

a metadata object extraction function configured to, when called by a second calling application, receive at least a metadata object, extract information from the received metadata object, and generate destination metadata, for storage in the data storage, from the extracted information, wherein if the received metadata object was generated by the metadata object creation function from source metadata, the destination metadata is a substantial copy of the source metadata.

10. The apparatus of claim 9, wherein the metadata handler is further configured to perform at least some of its functions at least partially by invoking one or more file system APIs configured to read and write the file data and the metadata.

11. The apparatus of claim 9, wherein the metadata object creation function is further configured to recognize native metadata of a plurality of file systems and to determine which portion of the metadata object to populate based at least in part on which file system is native to the source metadata.

12. The apparatus of claim 9, wherein the metadata object creation function is further configured to recognize native metadata of a plurality of file systems and to determine a format for the metadata object based at least in part on which file system is native to the source metadata.

13. The apparatus of claim 9, wherein the metadata object extraction function is further configured to determine a version of the received metadata object and to determine how to process the metadata object based at least in part on the version of the received metadata object.

14. A network storage system comprising:

data storage configured to store file data and metadata associated with the file data, at least a first portion of the metadata being metadata native to a first file system and at least a second portion of the metadata being metadata native to a second file system;

one or more file system APIs configured to define operations for writing file data and metadata to the data storage and reading file data and metadata from the data storage; and

a metadata handler having a library of functions configured to handle at least the first portion of metadata and the second portion of metadata, the library comprising;

a metadata object creation function configured to, when called by a first calling application, cooperate with at least one of the file system APIs to read, from the data storage, source metadata associated with specified file data, and to create a metadata object and to populate at least a portion of the metadata object with information from which a substantial copy of the source metadata can be generated, the populated portion corresponding to metadata fields used by the source metadata'"'"'s native file system; and

a metadata object extraction function configured to, when called by a second calling application, receive at least a metadata object, extract information from the received metadata object, cooperate with at least one of the file system APIs in order to generate destination metadata from the extracted information, and to store the destination metadata in the data storage, wherein if the received metadata object was generated by the metadata object creation function from source metadata, the destination metadata is a substantial copy of the source metadata.

15. The system of claim 14, wherein the metadata object is a binary large object.

16. The system of claim 14, wherein at least one of the first portion and the second portion are stored in the data storage in a format different from the first portion'"'"'s or second portion'"'"'s native format.

17. The system of claim 14, wherein the data storage, the file system APIs, and the metadata handler reside within or in communication with a network attached storage device.

18. The system of claim 14, further comprising at least one client application configured to invoke the metadata handler'"'"'s library.

19. The system of claim 18, wherein the data storage, the file system APIs, the metadata handler, and the client application reside within or in communication with a network attached storage device.

20. The system of claim 18, wherein the client application comprises a wrapper library that includes at least references to the metadata handler'"'"'s library functions which provide a public interface to the metadata handler'"'"'s library functions.

21. The system of claim 18, wherein the client application is configured to perform at least one of a backup operation, a restore operation, and a data migration operation.

22. The system of claim 21, wherein the backup operation can store metadata objects on at least one external backup storage device that stores files using a file system that is different from a native file system of at least a portion of underlying metadata stored in the metadata objects and the restore operation can restore underlying metadata, neither operation requiring the client application to be able to read the metadata object or the underlying metadata independently of the metadata handler.

23. A method of storing a data file and associated metadata, the method comprising:

requesting a file at least partially by invoking a metadata handler configured to receive file data and metadata and to encapsulate within a metadata object information from which the metadata can be substantially restored;

receiving the requested file and a metadata object that encapsulates information from which metadata associated with the file can be substantially restored, wherein the metadata object has a format different from the received metadata'"'"'s native format; and

storing the requested file and the metadata object.

24. The method of claim 23, wherein the metadata object is a binary large object.

25. The method of claim 23, wherein storing the file and the metadata object comprises storing the file and the metadata object on a backup file system different from the associated metadata'"'"'s native file system without requiring the backup file system to be able to read the associated metadata stored in the metadata object.

26. The method of claim 25, further comprising providing the stored file and the metadata object as input to a metadata handler in order to substantially restore the stored file and underlying metadata encapsulated in the metadata object, without requiring the backup file'"'"'system to be able to read the underlying metadata stored in the metadata object.