Implementing storage volumes over multiple tiers

• US 10,599,622 B2
• Filed: 07/31/2018
• Issued: 03/24/2020
• Est. Priority Date: 07/31/2018
• Status: Active Grant

• Current Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Original Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Inventors: Mukku, Jagadish Kumar, Alluboyina, Ravi Kumar, Venkatesan, Dhanashankar
• Primary Examiner(s): Lo, Ann J
• Assistant Examiner(s): Wei, Jane
• Application Number:
  US16/050,801
• Publication Number:
  US 20200042618A1
• Time in Patent Office: 602 Days
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

A distributed storage system stores a storage volume as segments that are allocated as needed and assigned VSIDs according to a monotonically increasing counter. The storage volume may be provisioned by an orchestration layer that manages the storage volumes as well as containers executing executable components of the storage volume. A storage volume may be stored across multiple tiers with a storage node managing on which tier a segment is stored based on a volume tier policy and usage of the segments. The tiering policy may be used to relocate the entire storage volume to a tier, such as a cloud computing platform, after which an application may be restored on the tier. Segments on a tier with built in redundancy may be used as both primary and backup copies.

Claims

1. A method comprising:

defining, by a computer system in a distributed storage system including a plurality of devices including the computer system, a storage volume, the distributed storage system defining a plurality of tiers each tier including a portion of the plurality of devices in the distributed storage system;

receiving, by the computer system, a volume tiering policy for the storage volume; and

moving, by the computer system, segments of the storage volume among tiers of the plurality of tiers according to usage of the segments and the volume tiering policy;

wherein, the method further comprising;

storing all of the segments of the storage volume on one or more local storage devices mounted to the computer system while a number of the segments of the storage volume is below a minimum tiering level, the minimum tiering level specifying a minimum number of the segments of the storage volume that must be stored on the one or more local storage devices before any of the segments may be redistributed to a different tier of the plurality of tiers; and

redistributing a portion of the segments of the storage volume to a remote storage device that is not local to the computer system only when a number of segments exceeds the minimum tiering level.

2. The method of claim 1, wherein a first tier of the plurality of tiers is a solid state drive mounted to the computer system.

3. The method of claim 2, wherein a second tier of the plurality of tiers is a hard disk drive mounted to the computer system.

4. The method of claim 3, wherein a third tier of the plurality of tiers is a cloud storage system remote from the computer system.

6. The method of claim 1, further comprising:

determining, by the computer system, that a tier of the plurality of tiers is full;

in response to determining that the tier of the plurality of tiers is full, adding, by the computer system, a new tier to the plurality of tiers and modifying the volume tiering policy to reference the new tier; and

redistributing, by the computer system, a portion of the segments of the storage volume to the new tier according to the volume tiering policy.

7. The method of claim 1, further comprising:

receiving, by the computer system, notification of addition of a new tier to the plurality of tiers;

adding, by the computer system, reference to the new tier to the volume tiering policy;

modifying, by the computer system, the volume tiering policy such that a maximum number of segments for a first tier of the plurality of tiers is zero, the first tier being other than the new tier; and

migrating, by the computer system, segments stored in the first tier to the new tier according to the volume tiering policy.

8. The method of claim 7, further comprising:

copying, by the computer system, application metadata to the new tier, the application metadata being for the application performing IOP (input/output) operations on the storage volume;

restarting the application using the application metadata in the new tier once all segments of the storage volume are migrated to the new tier.

9. The method of claim 8, wherein the new tier is a cloud storage and computing tier.

10. The method of claim 1, wherein the storage volume is a first storage volume, the method further comprising:

an orchestration layer executing in the distributed storage system;

instantiating, by the orchestration layer, an application in the distributed storage system;

instructing, by the orchestration layer, a storage manager, to provision a plurality of storage volumes including the first storage volume for use by the application;

specifying, by the orchestration layer to the storage manager for each storage volume, usage percentages for each tier of the plurality of tiers; and

specifying, by the storage manager, a volume tiering policies for each storage volume of the plurality of storage volumes corresponding to the usage percentages.

11. The method of claim 1, wherein the plurality of tiers include a first tier and a second tier, the first tier including one or more storage devices mounted locally to the computer system and the second tier including a cloud storage system remote from the computer system, the method further comprising:

creating, by the computer system, a backup copy of the storage volume in the second tier such that copies of all of the segments of the storage volume are stored in the second tier;

(a) determining, by the computer system, that a first segment of the segments of the storage volume are to be transferred to the second tier according to the volume tiering policy;

in response to (a), deleting the first segment in the first tier and incrementing a reference count corresponding to a copy of the first segment in the backup copy of the storage volume on the second tier.

12. The method of claim 1, wherein the plurality of tiers include a first tier and a second tier, the first tier including one or more storage devices mounted locally to the computer system and the second tier including a cloud storage system remote from the computer system, the method further comprising:

redistributing, by the computer system, the segments of the storage volume to the second tier such that the segments are stored as objects in an object store;

provisioning, by the second tier, a virtual disk;

copying, by the computer system, metadata for an application accessing the storage volume to the first tier;

mounting, by the first tier, the virtual disk to the application;

setting, by the first tier, the virtual disk as a tier of the plurality of tiers;

restarting, by the first tier, the application with the metadata in the first tier;

migrating, by the first tier, at least a portion of the segments from the object store to the virtual disk according to usage of the segments by the application according to the volume tiering policy.

13. The method of claim 1, wherein the plurality of tiers include a first tier and a second tier, the first tier including one or more storage devices mounted locally to the computer system and the second tier including a cloud storage system remote from the computer system, the method further comprising:

creating, by the computer system, by allocating the segments of the storage volume as the segments are required to store data from write operations from an application such that each segment of the segments is assigned an identifier according to a monotonically increasing index upon allocation;

creating, by the computer system, a backup copy of the storage volume in the second tier such that copies of all of the segments of the storage volume at the time of creating the backup copy are stored in the second tier;

restoring, by the computer system, the storage volume from the backup copy by;

determining a highest identifier of the copies of the segments of the backup copy;

determining a highest identifier of the segments of the storage volume at the time of restoring;

when the highest identifier of the copies of the segments of the backup copy are higher than the highest identifier of the segments of the storage volume at the time of restoring, restoring to the storage volume those copies of the segments in the backup copy having identifiers thereof above the highest identifier of the segments of the storage volume at the time of restoring.

5. A method comprising:

defining, by a computer system in a distributed storage system including a plurality of devices including the computer system, a storage volume, the distributed storage system defining a plurality of tiers each tier including a portion of the plurality of devices in the distributed storage system;

receiving, by the computer system, a volume tiering policy for the storage volume; and

moving, by the computer system, segments of the storage volume among tiers of the plurality of tiers according to usage of the segments and the volume tiering policy;

wherein the storage volume is a first storage volume and the volume tiering policy is a first volume tiering policy, the method further comprising defining a second storage volume having a second volume tiering policy, the method further comprising;

receiving, by the computer system, a system tiering policy including a minimum tiering level;

storing, by the computer system, all of the segments of the first storage volume and all segments of the second storage volume on one or more local storage devices mounted to the computer system while a number of the segments of the storage volume is below the minimum tiering level, the minimum tiering level specifying a minimum total number of the segments of the first storage volume and the second storage volume that must be stored on the one or more local storage devices before any of the segments of the first storage volume and the second storage volume may be redistributed to a different tier of the plurality of tiers; and

redistributing a portion of the segments of the first storage volume according to the first volume tiering policy and a portion of the segments of the second storage volume according to the second volume tiering policy to one or more remote storage devices that are not local to the computer system only when a total number of the segments of the first storage volume and the segments of the second storage volume exceeds the minimum tiering level.

14. A system comprising:

a distributed storage system including a plurality of devices including a computer system;

wherein the computer system is programmed to;

define a storage volume, the distributed storage system defining a plurality of tiers each tier including a portion of the plurality of devices in the distributed storage system;

receive a volume tiering policy for the storage volume; and

store segments of the storage volume among tiers of the plurality of tiers according to usage of the segments of the storage volume and the volume tiering policy;

wherein the computer system is further programmed to;

store all of the segments of the storage volume on one or more local storage devices mounted to the computer system while a number of the segments of the storage volume is below a minimum tiering level, the minimum tiering level specifying a minimum number of the segments of the storage volume that must be stored on the one or more local storage devices before any of the segments of the storage volume may be redistributed to other tiers of the plurality of tiers; and

redistribute a portion of the segments of the storage volume to a remote storage device that is not local to the computer system only when a number of the segments of the storage volume exceeds the minimum tiering level.

15. The system of claim 14, wherein:

a first tier of the plurality of tiers is a solid state drive mounted to the computer system;

a second tier of the plurality of tiers is a hard disk drive mounted to the computer system; and

a third tier of the plurality of tiers is a cloud storage system remote from the computer system.

16. The system of claim 14, wherein the storage volume is a first storage volume and the volume tiering policy is a first volume tiering policy;

wherein the computer system is further programmed to;

define a second storage volume having a second volume tiering policy;

receive a system tiering policy including the minimum tiering level;

store all of the segments of the first storage volume and all segments of the second storage volume on one or more local storage devices mounted to the computer system while a total number of the segments of the first storage volume and the second storage volume is below the minimum tiering level; and

redistribute a portion of the segments of the first storage volume according to the first volume tiering policy and a portion of the segments of the second storage volume according to the second volume tiering policy to one or more remote storage devices that are not local to the computer system only when the total number of the segments of the first storage volume and the segments of the second storage volume exceeds the minimum tiering level.

17. The system of claim 14, wherein the computer system is further programmed to:

determine, that a tier of the plurality of tiers is full;

in response to determining that the tier of the plurality of tiers is full, add a new tier to the plurality of tiers;

modify the volume tiering policy to reference the new tier; and

redistribute a portion of the segments of the storage volume to the new tier according to the volume tiering policy.

18. The system of claim 14, wherein the computer system is further programmed to:

receive a notification of addition of a new tier to the plurality of tiers;

add reference to the new tier to the volume tiering policy;

modify the volume tiering policy such that a maximum number of segments for a first tier of the plurality of tiers is zero, the first tier being other than the new tier; and

migrate segments stored in the first tier to the new tier according to the volume tiering policy.