EFFICIENT MIGRATION TO DISTRIBUTED STORAGE

• US 20180165034A1
• Filed: 06/13/2017
• Published: 06/14/2018
• Est. Priority Date: 12/13/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Danilov, Mikhail, Skripko, Gregory, Gutsalov, Nikita, Tchoub, Ivan, Fedorov, Alexander, Koyushev, Sergey, Gavrilova, Maria
• Granted Patent: US 10,564,883 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A computer program product, system, and method for determining a list of objects, within source storage, to migrate; generating a chunk layout for the objects to migrate; and for each unencoded chunk within the chunk layout: retrieving objects from source storage specified by the unencoded chunk within the chunk layout; generating data and coded fragments for the unencoded chunk using the retrieved objects; and storing the data and coded fragments to primary storage.

Claims

1. A method comprising:

determining a list of objects within source storage to migrate;

generating a chunk layout for the objects to migrate; and

for each unencoded chunk within the chunk layout;

retrieving objects from source storage specified by the unencoded chunk within the chunk layout;

generating data and coded fragments for the unencoded chunk using the retrieved objects; and

storing the data and coded fragments to primary storage.

2. The method of claim 1 wherein determining the list of objects within source storage to migrate includes querying the source storage.

3. The method of claim 2 wherein querying the source storage includes retrieving a list of object sizes from source storage.

4. The method of claim 1 wherein generating the chunk layout includes placing related objects within contiguous chunk segments and/or within contiguous chunks.

5. The method of claim 1 wherein generating data and coded fragments for each unencoded chunk within the chunk layout includes generating data and coded fragments in parallel across multiple nodes of distributed storage system.

6. A system comprising:

a processor;

a volatile memory; and

a non-volatile memory storing computer program code that when executed on the processor causes the processor to execute a process operable to perform the operations of;

determining a list of objects within source storage to migrate;

generating a chunk layout for the objects to migrate; and

for each unencoded chunk within the chunk layout;

retrieving objects from source storage specified by the unencoded chunk within the chunk layout;

generating data and coded fragments for the unencoded chunk using the retrieved objects; and

storing the data and coded fragments to primary storage.

7. The system of claim 6 wherein determining the list of objects within source storage to migrate includes querying the source storage.

8. The system of claim 7 wherein querying the source storage includes retrieving a list of object sizes from source storage.

9. The system of claim 6 wherein generating the chunk layout includes placing related objects within contiguous chunk segments and/or within contiguous chunks.

10. The system of claim 6 wherein generating data and coded fragments for each unencoded chunk within the chunk layout includes generating data and coded fragments in parallel across multiple nodes of distributed storage system.

11. A computer program product tangibly embodied in a non-transitory computer-readable medium, the computer-readable medium storing program instructions that are executable to:

determine a list of objects within source storage to migrate;

generate a chunk layout for the objects to migrate; and

for each unencoded chunk within the chunk layout;

retrieve objects from source storage specified by the unencoded chunk within the chunk layout;

generate data and coded fragments for the unencoded chunk using the retrieved objects; and

store the data and coded fragments to primary storage.

12. The computer program product of claim 11 wherein determining the list of objects within source storage to migrate includes querying the source storage.

13. The computer program product of claim 12 wherein querying the source storage includes retrieving a list of object sizes from source storage.

14. The computer program product of claim 11 wherein generating the chunk layout includes placing related objects within contiguous chunk segments and/or within contiguous chunks.

15. The computer program product of claim 11 wherein generating data and coded fragments for each unencoded chunk within the chunk layout includes generating data and coded fragments in parallel across multiple nodes of distributed storage system.