Migrating data from legacy storage systems to object storage systems

• US 9,208,181 B2
• Filed: 10/24/2014
• Issued: 12/08/2015
• Est. Priority Date: 12/06/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Pandey, Vineet, Sharma, Chhavi, Kumar, Ranjit, Deshmukh, Parag, Voruganti, Kaladhar
• Primary Examiner(s): Ly, Anh
• Application Number:
  US14/523,801
• Publication Number:
  US 20150046502A1
• Time in Patent Office: 410 Days
• Field of Search: 707/809, 707/792, 707/797, 707/782, 707/828, 707/822, 707/827, 707/829, 707/825, 707/626, 707/638, 707/640, 707/692, 707/737, 707/736, 707/705, 707/770, 707/687, 707/793, 707/688, 707/694, 707/831, 707/E17.044, 707/E17.01, 707/E17.005, 707/E17.014, 707/E17.055, 711/114, 711/154, 711/202, 711/100, 711/112, 711/165, 711/170, 711/153, 711/156, 711/118, 711/162, 711/E12.002, 709/226, 709/211, 709/212, 709/213, 709/214, 709/223, 718/1
• US Class Current: 1/1
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/1794 Details of file format conversion
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/289 Object oriented databases
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

One or more techniques and/or systems are provided for migrating a dataset from a file storage system to an object storage system. That is, a snapshot of a file system may be received from the file storage system. The snapshot may comprise file data associated with a file of the file system. The file may be converted into an object using the file data. The object may be stored within a data constituent volume of the object storage system. A namespace volume, used to track objects, may be populated with a redirector that maps a front-end data path (e.g., a path used by clients to reference the object) to a back-end data path that specifies a path to the object within the data constituent volume. In this way, a dataset of one or more files may be migrated from the file storage system to the object storage system.

Claims

1. A method for migrating a dataset from a file storage system to an object storage system by converting a file to an object, comprising:

identifying file data associated with a file of a file system of a file storage system;

converting the file into an object using the file data;

responsive to determining that a prior object version of the object does not exist within at least one data constituent volume of an object storage system;

storing the object within a first data constituent volume of the object storage system, the storing comprising;

identifying a service level objective for the file; and

selecting the first data constituent volume for storage of the object based upon the first data constituent volume comprising one or more characteristics that correspond to the service level objective; and

creating an entry within a namespace volume associated with the object storage system, the entry comprising a redirector mapping a front-end data path for the object to a back-end data path, the back-end data path specifying a path to the object within the first data constituent volume; and

responsive to determining that the prior object version of the object does exist within a second data constituent volume of the object storage system;

receiving a data difference between the prior object version and the object;

creating a new object within the second data constituent volume based upon the prior object version and the data difference; and

creating a new redirector, within the namespace volume, for the new object.

2. The method of claim 1, comprising:

updating the redirector to map to a new location of the object within a third data constituent volume responsive to the object being moved from the first data constituent volume to the third data constituent volume.

3. The method of claim 1, the service level objective comprising at least one of:

a storage space efficiency objective;

a reliability objective;

a security objective;

a cost objective;

or

a performance objective.

4. The method of claim 1, comprising:

migrating a second file from a second file system into the first data constituent volume as a second object.

5. The method of claim 1, comprising:

migrating a second file from a second file system into the second data constituent volume as a second object.

6. The method of claim 1, the identifying file data comprising:

evaluating a snapshot of the file system to identify the file data.

7. The method of claim 1, the front-end data path comprising at least one of a storage cluster identifier, a logical partition identifier, a file system identifier, or a snapshot identifier.

8. The method of claim 1, the creating an entry within a namespace volume comprising:

identifying an inode to path (I2P) data structure from the file system, the I2P data structure comprising a mapping of an inode to the file; and

defining the front-end data path based at least in part upon the mapping.

9. The method of claim 1, the namespace volume comprising one or more entries associated with a plurality of data constituent volumes configured as a single front-end object volume.

10. The method of claim 1, the identifying file data comprising:

utilizing data module to network module communication without performing a complete traversal of a source stack.

11. The method of claim 1, comprising:

migrating a plurality of files, comprised within a snapshot of the file system, from the file storage system into one or more data constituent volumes of the object storage system.

12. The method of claim 1, comprising:

receiving a plurality of snapshots through one or more data streams;

assigning a priority to respective data streams; and

queuing the one or more data streams for processing based upon priorities assigned to respective data streams.

13. The method of claim 1, the storing the object, comprising:

associating at least one of user defined metadata or system defined metadata with the object as attached metadata.

14. The method of claim 13, comprising:

querying the object based upon a metadata query corresponding to at least some of the attached metadata.

15. A system for migrating a dataset from a file storage system to an object storage system by converting a file to an object, comprising:

a migration component configured to;

identify file data associated with a file of a file system of a file storage system;

convert the file into an object using the file data;

responsive to determining that a prior object version of the object does not exist within at least one data constituent volume of an object storage system;

store the object within a first data constituent volume of the object storage system;

identify an inode to path (I2P) data structure from the file system, the I2P data structure comprising a mapping of an inode to the file;

define a front-end data path based at least in part upon the mapping; and

create an entry within a namespace volume associated with the object storage system, the entry comprising a redirector mapping the front-end data path for the object to a back-end data path, the back-end data path specifying a path to the object within the first data constituent volume; and

responsive to determining that the prior object version of the object does exist within a second data constituent volume of the object storage system;

receive a data difference between the prior object version and the object as the file data;

create a new object within the second data constituent volume based upon the prior object version and the data difference, comprising;

create a new redirector, within the namespace volume, for the new object.

16. The system of claim 15, the migration component configured to:

identify a service level objective for the file; and

select the first data constituent volume for storage of the object based upon the first data constituent volume comprising one or more characteristics that correspond to the service level objective.

17. The system of claim 15, the migration component configured to:

invoke an object management component to update the redirector to map to a new location of the object within a third data constituent volume responsive to the object being moved from the first data constituent volume to the third data constituent volume.

18. The system of claim 15, the migration component configured to:

receive a plurality of snapshots through one or more data streams;

assign priorities to the one or more data streams; and

queue the one or more data streams for processing based upon the assigned priorities.

19. The system of claim 15, the migration component configured to:

clone the prior object version to create a cloned object; and

apply the data difference to the cloned object to create the new object.

20. A non-transitory computer-readable storage medium comprising processor-executable instructions that when executed perform a method for migrating a dataset from a file storage system to an object storage system by converting a file to an object, comprising:

receiving a plurality of snapshots through one or more data streams;

assigning a priority to respective data streams;

queuing the one or more data streams for processing based upon priorities assigned to respective data streams;

identifying file data, within a snapshot of the plurality of snapshots, associated with a file of a file system of a file storage system;

converting the file into an object using the file data;

responsive to determining that a prior object version of the object does not exist within at least one data constituent volume of an object storage system;

storing the object within a first data constituent volume of the object storage system;

creating an entry within a namespace volume associated with the object storage system, the entry comprising a redirector mapping a front-end data path for the object to a back-end data path, the back-end data path specifying a path to the object within the first data constituent volume; and

responsive to determining that the prior object version of the object does exist within a second data constituent volume of the object storage system;

receiving a data difference between the prior object version and the object;

creating a new object within the second data constituent volume based upon the prior object version and the data difference; and

creating a new redirector, within the namespace volume, for the new object.