Technique for virtualizing storage using stateless servers

• US 8,209,365 B2
• Filed: 07/17/2008
• Issued: 06/26/2012
• Est. Priority Date: 07/23/2007
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Manjunath, Geetha
• Primary Examiner(s): Ehichioya, Fred I
• Assistant Examiner(s): OBISESAN, AUGUSTINE KUNLE
• Application Number:
  US12/175,283
• Publication Number:
  US 20090030957A1
• Time in Patent Office: 1,440 Days
• Field of Search: None
• US Class Current: 707/827
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: G06F 16/188 Virtual file systems

Abstract

A system and method for virtualizing networked heterogeneous storage devices in a computing system using stateless servers is disclosed. In one embodiment, the method includes self discovery of mapping information between a plurality of virtual file systems to a plurality of physical file systems upon startup by a virtual data store (VDS) server. The networked heterogeneous storage devices include the plurality of physical file systems hosted on associated file servers. Further, each physical file system includes a storage space in which a plurality of data files may be stored.

Claims

1. A method of virtualizing networked heterogeneous storage devices in a computing system, wherein the networked heterogeneous storage devices includes a plurality of physical file systems hosted on associated file servers, and wherein each physical file system includes a storage space in which a plurality of data files may be stored comprising:

self discovering mapping information between a plurality of virtual file systems to the plurality of physical file systems upon startup by a virtual data store (VDS) server, comprising;

determining available physical file systems to be virtualized based on configuration information;

communicating with each available physical file system via an associated protocol gateway to determine list of subdirectories in each root directory, wherein each root directory comprises top level directories of each physical file system;

mapping each subdirectory of each physical file system to a protocol gateway object and creating mapping information; and

storing the mapping information in a directory cache by the VDS server.

2. The method of claim 1, wherein self discovering the mapping information between the plurality of virtual file systems to the plurality of physical file systems upon startup comprises:

mapping a plurality of files and associated folders in the plurality of virtual file systems to the plurality of files and associated folders in the plurality of physical file systems based on a current state of the plurality of physical file systems to discover and create mapping information upon startup; and

storing the mapping information in the directory cache by the VDS server.

3. The method of claim 2, wherein mapping the plurality of files and the associated folders in the plurality of virtual file systems to the plurality of files and the associated folders in the plurality of physical file systems comprises:

mapping root level folders associated with the plurality of virtual file systems to one or more protocol gateway objects associated with the plurality of physical file systems based on a current state of the plurality of physical file systems to obtain the mapping information upon startup; and

storing the mapping information in the directory cache by the VDS server.

4. The method of claim 1, further comprising:

obtaining an associated physical file based on the stored mapping information upon receiving a request for a virtual file from a client by the VDS server.

5. The method of claim 4, wherein obtaining the associated physical file comprises:

determining the physical repository gateway object upon receiving a request for a virtual file from a client by determining the subdirectory name of the requested file and locating the subdirectory name in the mapping information; and

obtaining the associated physical file via the determined physical repository gateway object to obtain the requested virtual file and to perform the requested file operation on the obtained file.

6. The method of claim 3, further comprising:

refreshing the directory cache located in the VDS server for a subset of entries upon creating a new subdirectory in a root directory in anyone of the file systems hosted on associated one of the plurality of physical file systems.

7. The method of claim 3, further comprising:

if the requested file is located in the plurality of physical file systems, then contacting each associated physical file system to check to obtain the requested file.

8. The method of claim 3, further comprising:

determining whether any of the subdirectories in each root directory are split directories;

if so, retrieving files from each of the split directories; and

combining the retrieved files to form the requested file.

9. The method of claim 3, further comprising:

determining whether any of the subdirectories are replicated; and

if so, obtaining primary and replica files from replicated directories and providing a file that is substantially closer to a primary file and maintaining consistency between the primary and replica files.

10. The method of claim 1, wherein the associated file servers comprise heterogeneous file servers providing different file system protocols, wherein the heterogeneous file servers are selected from the group consisting of Grid file servers, Unix file servers, and Windows file servers, Web-based file server, and wherein the file system protocols are selected from the group consisting of Grid file transfer protocol (GridFTP), network-attached storage file system (NFS)

protocol, common Internet file system (CIFS) protocol, and Web-based distributed authoring and versioning (WebDAV) protocol, respectively.

11. A non-transitory computer readable storage medium having instructions, that when executed by a computing platform, result in execution of a method of virtualizing networked heterogeneous storage devices in a computing system comprising:

self discovering mapping information between a plurality of virtual file systems to a plurality of physical file systems upon startup by a virtual data store (VDS) server, wherein the networked heterogeneous storage devices includes the plurality of physical file systems hosted on associated file servers, and wherein each physical file system includes a storage space in which a plurality of data files may be stored, comprising;

determining available physical file systems to be virtualized based on configuration information;

communicating with each available physical file system via an associated protocol gateway to determine list of subdirectories in each root directory, wherein each root directory comprises top level directories of each physical file system;

mapping each subdirectory of each physical file system to a protocol gateway object and creating mapping information; and

storing the mapping information in a directory cache by the VDS server.

12. The non-transitory computer readable storage medium of claim 11, wherein self discovering the mapping information between the plurality of virtual file systems to the plurality of physical file systems upon startup comprises:

mapping a plurality of files and associated folders in the plurality of virtual file systems to the plurality of files and associated folders in the plurality of physical file systems based on the current state of the plurality of physical file systems to discover and create mapping information upon startup; and

storing the mapping information in the directory cache by the VDS server.

13. The non-transitory computer readable storage medium of claim 11, wherein mapping the plurality of files and the associated folders in the plurality of virtual file systems to the plurality of files and the associated folders in the plurality of physical file systems comprises:

mapping root level folders associated with the plurality of virtual file systems to one or more protocol gateway objects associated with the plurality of physical file systems based on a current state of the plurality of physical file systems to obtain the mapping information upon startup; and

storing the mapping information in the directory cache by the VDS server.

14. The non-transitory computer readable storage medium of claim 11, further comprising:

obtaining an associated physical file based on the stored mapping information upon receiving a request for a virtual file from a client by the VDS server.

15. A virtual enterprise file system comprising:

a plurality of clients;

a plurality of file systems located in associated physical repositories and wherein each physical file system includes a storage space in which a plurality of data files may be stored; and

a virtual data store (VDS) server hosting a VDS component, wherein the VDS component self discovers mapping information from a plurality of virtual file systems to the plurality of physical file systems upon startup, wherein the VDS component comprises;

a protocol gateways layer;

a presentation protocol layer; and

a file virtualization layer, wherein the file virtualization layer determines available physical file systems to be virtualized based on a configuration information upon startup, wherein the file virtualization layer reviews the configuration information and determines available physical file systems to be virtualized, wherein the file virtualization layer communicates with each available physical file system via an associated gateway in the protocol gateways layer to determine list of subdirectories in each root directory, wherein each root directory comprises top level directories of each physical file system, and wherein the file virtualization layer maps each subdirectory of each physical file system to protocol gateway object and creates mapping information and stores the mapping information in a directory cache.

16. The system of claim 15, wherein the presentation protocol layer receives a request for a virtual file from a client, and wherein the protocol gateways layer obtains an associated physical file via the determined physical repository gateway object based on the stored mapping information to obtain the requested virtual file to perform a requested file operation.

17. The system of claim 15, wherein the VDS component further comprises:

a policy engine for configuring administrative control parameters of the file virtualization layer and the physical file systems, wherein the administrative control parameters are selected from the group consisting of source and destination tiers of file systems, disk/user quota, data migration policies and actions based on file and storage attributes.

18. The system of claim 15, wherein the file virtualization layer refreshes the directory cache located in the VDS server for a subset of entries upon creating a new subdirectory in a new root directory in anyone of the plurality of physical file systems hosted on associated one of the physical repositories.

19. A virtual enterprise file system comprising:

a plurality of clients;

a plurality of file systems located in associated physical repositories and wherein each physical file system includes a storage space in which a plurality of data files may be stored; and

multiple virtual data store (VDS) servers, wherein each VDS server hosts a VDS component, wherein the VDS component in one of the multiple VDS servers self discovers the mapping of the files and the folders of a plurality of virtual file systems to the files and folders on plurality of physical file systems upon startup, wherein the VDS component comprises;

a protocol gateways layer;

a client interface layer; and

a file virtualization layer, wherein the file virtualization layer determines available physical file systems to be virtualized based on a configuration information upon startup, wherein the file virtualization layer reviews the configuration information and determines available physical file systems to be virtualized, wherein the file virtualization layer communicates with each available physical file system via an associated gateway in the protocol gateways layer to determine list of subdirectories in each root directory, wherein each root directory comprises top level directories of each physical file system, and wherein the file virtualization layer maps each subdirectory of each physical file system to protocol gateway object and creates mapping information and stores in a directory cache.

20. The system of claim 19, wherein the client interface layer receives a request for a virtual file from a client and wherein the file virtualization layer determines the physical repository gateway object to be used to process the request and the protocol gateways layer obtains an associated physical file via a determined physical repository gateway object based on the stored mapping information to obtain the requested virtual file to perform a requested file operation.