Implementing a hybrid storage node in a distributed storage system

• US 10,430,110 B2
• Filed: 12/19/2017
• Issued: 10/01/2019
• Est. Priority Date: 12/19/2017
• Status: Active Grant

• Current Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Original Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Inventors: Mukku, Jagadish K., Venkatesan, Dhanashankar, Patel, Ripulkumar Hemantbhai
• Primary Examiner(s): Mackall, Larry T
• Application Number:
  US15/847,652
• Publication Number:
  US 20190187907A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 12/0253 Garbage collection, i.e. reclamation of unreferenced memory
• CPC Class Codes: G06F 12/1072 Decentralised address translation, e.g. in distributed shared memory systems
• CPC Class Codes: G06F 2212/7205 Cleaning, compaction, garbage collection, erase control
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0664 at device level, e.g. emulation of a storage device or system
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/068 Hybrid storage device
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/545 where tasks reside in different layers, e.g. user- and kernel-space
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services

Abstract

A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. The snapshots may be represented by a storage manager in a hierarchy. Deleted snapshots may be flagged as such in the hierarchy and deletion may be implemented only in memory on a storage node, which is then restored from the hierarchy in the event of a crash. A hybrid storage node may perform both computing and storage services. An IO module determines whether IOPs reference a locally-mounted storage device or a remote storage device. The IO module issues a library call for local IOPs using a shared memory in kernel space.

Claims

1. A method for processing input/output operations (IOPs) in a distributed storage system, the method comprising:

receiving, by an input/output (TO) module executing on a storage node comprising one or more processing devices and one or more storage devices, a first IOP from a first application;

determining, by the IO module, that (a) the first application is executing on the storage node; and

in response to determining (a), issuing, by the IO module, a first library function call to disk virtualization manager (DVM) executing on the storage node to execute the first IOP;

in response to the first library function call, performing, by the DVM;

executing the first IOP, the first IOP being a read IOP, to obtain payload data;

writing the payload data to a shared memory in kernel space of an operating system executing on the storage node, the DVM and IO module executing in a same process in user space of the operating system and both having access to the shared memory; and

issuing a second library function call to the IO module that does not include the payload data; and

in response to the second library function call, reading, by the IO module, the payload data from the shared memory and returning the payload data to the first application.

2. The method of claim 1, further comprising:

returning, by the DVM, a result of the first IOP to the IO module; and

returning, by the TO module, the result to the first application.

3. The method of claim 2, wherein the first IOP is a write IOP, the method further comprising:

writing, by the IO module, payload data from the first IOP to a shared memory in kernel space of an operating system executing on the storage node, the DVM and IO module executing in a user space of the operating system and both having access to the shared memory;

issuing, by the IO module, the first library function call to the DVM for the first IOP without including the payload data in the first library function call; and

in response to the first library function call, reading, by the DVM, the payload data from the shared memory and writing the payload data to a location referenced by the first IOP.

4. The method of claim 1, further comprising:

receiving, by the IO module, a second IOP from a second application;

determining, by the IO module, that (b) the second application is not executing on the storage node; and

in response to determining (a), issuing a remote procedure call (RPC), by the IO module, to a DVM executing on a remote node corresponding to the second IOP.

5. The method of claim 1, wherein the IO module and DVM execute within a same process on the storage node.

6. The method of claim 5, wherein the same process is a daemon.

7. The method of claim 6, wherein the daemon executes in a same user application space in an operation system of the storage node as the first application.

8. A system comprising one or more processing devices, one or more memory devices, and one or more storage devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:

receive, by an input/output (TO) module, a first input/output operation (TOP) from a first application;

determine, using the IO module, that (a) the first application is executing on the system; and

in response to determining (a), issue, by the IO module, a first library function call to disk virtualization manager (DVM) executing on the system to execute the first IOP;

in response to the first library function call;

execute, by the DVM, the first TOP, the first TOP being a read TOP, to obtain payload data;

write, by the DVM, the payload data to a shared memory in kernel space of an operating system executing on the storage node, the DVM and IO module executing in a user space of the operating system and both having access to the shared memory; and

issue, by the DVM, a second library function call to the IO module that does not include the payload data; and

in response to the second library function call, read, by the IO module, the payload data from the shared memory and return the payload data to the first application.

9. The system of claim 8, wherein the executable code is further effective to cause the one or more processing devices to:

return, by the DVM, a result of the first TOP to the IO module; and

return, by the IO module, the result to the first application.

10. The system of claim 9, wherein the first TOP is a write TOP, the executable code is further effective to cause the one or more processing devices to:

write, by the TO module, payload data from the first TOP to a shared memory in kernel space of an operating system executing on the storage node, the DVM and TO module executing in a user space of the operating system and both having access to the shared memory;

issue, by the TO module, the first library function call to the DVM for the first TOP without including the payload data in the first library function call; and

read, by the DVM, the payload data from the shared memory and write the payload data to a location referenced by the first TOP.

11. The system of claim 8, the executable code is further effective to cause the one or more processing devices to:

receive, by the TO module, a second TOP from a second application;

determine, by the TO module, that (b) the second application is not executing on the storage node; and

in response to determining (a), issue, by the TO module, a remote procedure call (RPC) to a DVM executing on a remote node corresponding to the second TOP.

12. The system of claim 8, wherein the TO module and DVM execute within a same process on the storage node.

13. The system of claim 12, wherein the same process is a daemon.

14. The system of claim 13, wherein the daemon executes in a same user application space in an operation system of the storage node as the first application.