Transferring data between block and file storage systems

• US 10,257,258 B2
• Filed: 10/31/2016
• Issued: 04/09/2019
• Est. Priority Date: 10/31/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gao, Xue Dong, Nielsen, Karl A., Peng, Yao, Rinaldi, Brian A., Yao, Tang Ren
• Primary Examiner(s): Zand, Davoud A
• Application Number:
  US15/338,642
• Publication Number:
  US 20180124152A1
• Time in Patent Office: 890 Days
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A processor may receive a request to transmit a logical volume to a cloud-based server. The logical volume may be stored in a data storage subsystem that includes one or more ranks. Each rank may include one or more extents, which may include one or more stride. The processor may determine an extent of the one or more extents that includes data of the logical volume. The processor may determine a set of strides associated with the extent. The processor may copy the set of strides into a stride buffer and combine the set of strides into a block extent file. The processor may transmit the block extent file to the cloud server.

Claims

1. A method comprising:

receiving a request to transmit a logical volume stored on a host server to a remote server, the logical volume being stored in a data storage system that includes one or more ranks, each rank having one or more extents, each extent having one or more strides;

determining an extent of the one or more extents that includes data of the logical volume;

determining a set of strides associated with the extent;

copying the set of strides for the extent into a stride buffer;

combining the set of strides for the extent into a block extent file; and

transmitting the block extent file to the remote server.

2. The method of claim 1, the method further comprising determining whether the logical volume is stored using block-level storage, wherein the determining the extent is performed in response to determining that the logical volume is stored using block-level storage, wherein the data storage system is a storage attached network (SAN) device, and wherein each rank of the one or more ranks is a redundant array of independent disks (RAID) array.

3. The method of claim 2, wherein the determining an extent of the one or more extents that includes data of the logical volume includes:

determining one or more logical block address (LBAs) associated with the logical volume;

determining, using the LBAs and a rank segment table, one or more ranks that are associated with the LBAs; and

mapping the one or more LBAs associated with the logical volume to one or more extent ranges using a logic configuration volume segment table.

4. The method of claim 1, wherein the data storage system is a Network-attached Storage (NAS) device that includes a storage processor, wherein each rank of the one or more ranks is a redundant array of independent disks (RAID) array, and wherein one or more of the operations are performed by the storage processor.

5. The method of claim 1, the method further comprising:

obtaining, in response to determining that the logical volume is stored using file-based storage, a filesystem mapping of the logical volume;

identifying, using the filesystem mapping of the logical volume, a file in the logical volume;

determining a set of strides associated with the file;

copying the set of strides associated with the file into the stride buffer;

combining the set of strides into a cloud file; and

transmitting the cloud file to the remote server.

6. The method of claim 5, wherein the determining the set of strides associated with the file comprises:

determining a set of LBAs for the file using the filesystem mapping;

determining, using the set of LBAs and a rank segment table, one or more ranks that are associated with the set of LBAs; and

mapping the set of LBAs associated with the file to one or more extent ranges using a logic configuration volume segment table.

7. The method of claim 5, wherein the obtaining the filesystem mapping of the logical volume comprises:

querying the host server for the filesystem mapping; and

receiving, from the host server, the filesystem mapping.

8. The method of claim 1, wherein the logical volume includes one or more distinct logical volumes, the method further comprising:

generating a master volume table file (MVTF) that stores metadata information of the one or more logical volumes, the MVTF having an entry for each of the one or more logical volume transferred;

generating, for each transferred logical volume, a volume extent meta file (VEMF) that stores metadata for the associated logical volume; and

transmitting the master volume table file and the one or more volume extent meta files to the remote server.

9. The method of claim 8, wherein the metadata information stored in the MVTF and the one or more VEMFs includes file names, sizes, access control lists, and identifications of the ranks and extents from which the one or more logical volumes were copied.

10. A system comprising:

a memory that includes one or more ranks, each rank having one or more extents, each extent having one or more strides; and

a processor communicatively coupled to the memory, the processor being configured to perform a method comprising;

receiving a request to transmit a logical volume stored in the memory to a remote server;

determining whether the logical volume is stored using block-level storage;

determining, in response to determining that the logical volume is stored using block-level storage, an extent of the one or more extents that includes data of the logical volume;

determining a set of strides associated with the extent;

copying the set of strides for the extent into a stride buffer;

combining the set of strides for the extent into a block extent file; and

transmitting the block extent file to the remote server.

11. The system of claim 10, the system further comprising a cloud connector configured to manage a connection between the system and a cloud container associated with remote server.

12. The system of claim 10, wherein the determining an extent of the one or more extents that includes data of the logical volume includes:

determining one or more logical block address (LBAs) associated with the logical volume;

determining, using the LBAs and a rank segment table, one or more ranks that are associated with the LBAs; and

mapping the one or more LBAs associated with the logical volume to one or more extent ranges using a logic configuration volume segment table.

13. The system of claim 10, wherein the method performed by the processor further comprises:

obtaining, in response to determining that the logical volume is stored using file-based storage, a filesystem mapping of the logical volume;

identifying, using the filesystem mapping of the logical volume, a file in the logical volume;

determining a set of strides associated with the file;

copying the set of strides associated with the file into the stride buffer;

combining the set of strides into a cloud file; and

transmitting the cloud file to the remote server.

14. The system of claim 13, wherein the determining the set of strides associated with the file comprises:

determining a set of LBAs for the file using the filesystem mapping;

determining, using the set of LBAs and a rank segment table, one or more ranks that are associated with the set of LBAs; and

mapping the set of LBAs associated with the file to one or more extent ranges using a logic configuration volume segment table.

15. The system of claim 10, wherein the logical volume includes one or more distinct logical volumes, and wherein the method performed by the processor further comprises:

generating a master volume table file (MVTF) that stores metadata information of the one or more logical volumes, the MVTF having an entry for each of the one or more logical volume transferred;

generating, for each transferred logical volume, a volume extent meta file (VEMF) that stores metadata for the associated logical volume; and

transmitting the master volume table file and the one or more volume extent meta files to the remote server.

16. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:

receiving a request to transmit a logical volume stored on a host server to a remote server, the logical volume being stored in a data storage system that includes one or more ranks, each rank having one or more extents, each extent having one or more strides;

determining whether the logical volume is stored using block-level storage;

determining, in response to determining that the logical volume is stored using block-level storage, an extent of the one or more extents that includes data of the logical volume;

determining a set of strides associated with the extent;

copying the set of strides for the extent into a stride buffer;

combining the set of strides for the extent into a block extent file; and

transmitting the block extent file to the remote server.

17. The computer program product of claim 16, wherein the determining an extent of the one or more extents that includes data of the logical volume includes:

determining one or more logical block address (LBAs) associated with the logical volume;

determining, using the LBAs and a rank segment table, one or more ranks that are associated with the LBAs; and

mapping the one or more LBAs associated with the logical volume to one or more extent ranges using a logic configuration volume segment table.

18. The computer program product of claim 16, wherein the method performed by the processor further comprises:

obtaining, in response to determining that the logical volume is stored using file-based storage, a filesystem mapping of the logical volume;

identifying, using the filesystem mapping of the logical volume, a file in the logical volume;

determining a set of strides associated with the file;

copying the set of strides associated with the file into the stride buffer;

combining the set of strides into a cloud file; and

transmitting the cloud file to the remote server.

19. The computer program product of claim 18, wherein the determining the set of strides associated with the file comprises:

determining a set of LBAs for the file using the filesystem mapping;

determining, using the set of LBAs and a rank segment table, one or more ranks that are associated with the set of LBAs; and

mapping the set of LBAs associated with the file to one or more extent ranges using a logic configuration volume segment table.

20. The computer program product of claim 16, wherein the logical volume includes one or more distinct logical volumes, and wherein the method performed by the processor further comprises:

generating a master volume table file (MVTF) that stores metadata information of the one or more logical volumes, the MVTF having an entry for each of the one or more logical volume transferred;

generating, for each transferred logical volume, a volume extent meta file (VEMF) that stores metadata for the associated logical volume; and

transmitting the master volume table file and the one or more volume extent meta files to the remote server.