Unified datapath architecture

• US 9,286,007 B1
• Filed: 03/14/2013
• Issued: 03/15/2016
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Bono, Jean-Pierre
• Primary Examiner(s): Chace, Christian P
• Assistant Examiner(s): Warren, Tracy A
• Application Number:
  US13/828,322
• Time in Patent Office: 1,097 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0662 Virtualisation aspects
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

An improved technique for a data storage apparatus that combines both block-based and file-based functionality in a unified data path architecture. The improved technique brings together IO processing of block-based storage systems and file-based storage systems by expressing both block-based objects (e.g., LUNs) and file-based objects (e.g., file systems) in the form of files. These files are parts of an underlying, internal set of file systems, which are stored on a set of storage units served by a storage pool. Because block and file-based objects are all expressed as files of this set of file systems, a common set of services can be applied across block-based and file-based objects. Also, storage units released by any file or files of the underlying, internal set of file systems can be reused by any other file or files, regardless of whether the files represent block-based objects or file-based objects. Inefficiencies of stranded storage are thus greatly reduced or eliminated altogether.

Claims

1. A method of managing host data of a set of hosts in a data storage apparatus, the method comprising:

allocating a set of slices of a storage pool to a set of file systems of the data storage apparatus;

storing a first file in the set of slices, the first file provided as a file in the set of file systems and providing a file representation of a LUN (logical unit number) storing block-based host data, wherein the block-based host data stored in the first file is accessible to the set of hosts using a block-based protocol;

storing a second file in the set of slices, the second file provided as a file in the set of file systems and providing a file representation of a host file system storing file-based host data, wherein the file-based host data stored in the second file is accessible to the set of hosts using a file-based protocol;

deallocating a slice of the set of slices from the set of file systems in response to the slice storing block-based host data of the first file and then becoming empty;

reallocating the slice to the set of file systems in response to the host file system stored in the second file requiring additional storage space; and

while the first file is providing the LUN, storing portions of the second file on the reallocated slice, thereby allowing the slice previously used for storing portions of the LUN to be reused for storing portions of the host file system,

wherein the method further comprises;

deallocating a second slice of the set of slices from the set of file systems in response to the slice storing file-based host data of the second file and then becoming empty;

reallocating the second slice to the set of file systems; and

while the second file is providing the host file system, storing portions of the first file on the reallocated second slice, thereby allowing the slice previously used for storing portions of the host file system to be reused for storing portions of the LUN;

wherein the method further comprises;

performing a data protection operation on the LUN by copying or snapping the first file to a first location; and

performing a data protection operation on the host file system by copying or snapping the second file to a second location,

wherein the first location and the second location are each one of (i) a location within the data storage apparatus and (ii) a location geographically remote from the data storage apparatus

and wherein the method further comprises;

storing in a configuration database of the data storage apparatus, a set of records defining at least one virtualized storage processor, the set of records identifying multiple file systems of each virtualized storage processor;

storing the file systems of each virtualized storage processor as respective files in the set of slices, each of the respective files provided as a file in the set of file systems and providing a file representation of a file system of the respective virtualized storage processor;

instantiating a set of host interfaces for each of the virtualized storage processors;

receiving, via the set of host interfaces for each of the virtualized storage processors, a respective set of read and write requests from the set of hosts; and

for each virtualized storage processor, processing the respective set of read and write requests received via the respective set of host interfaces, to read data from and write data to at least one of the file systems of the virtualized storage processor.

2. The method of claim 1, further comprising:

storing a third file in the set of slices, the third file provided as a file in the set of file systems and providing a file representation of a virtual volume associated with a host virtual machine;

deallocating a third slice of the set of slices from the set of file systems in response to the third slice storing file-based host data of the second file and then becoming empty;

reallocating the third slice to the set of file systems in response to the virtual volume stored in the third file requiring additional storage space; and

while second first file is providing the host file system, storing portions of the third file on the reallocated third slice, thereby allowing the slice previously used for storing portions of the host file system to be reused for storing portions of the virtual volume.

3. The method of claim 1, wherein the data storage apparatus includes a second storage pool, and wherein the method further comprises:

allocating a second set of slices of the second storage pool to a second set of file systems of the data storage apparatus;

storing multiple file systems of a second virtualized storage processor as respective files in the second set of slices allocated from the second storage pool, each of the respective files provided as a file in the second set of file systems and providing a file representation of a file system of the second virtualized storage processor;

instantiating a set of host interfaces for the second virtualized storage processor;

receiving, via the set of host interfaces for the second virtualized storage processor, a set of read and write requests from the set of hosts; and

processing the set of read and write requests received via the set of host interfaces by the second virtualized storage processor to read data from and write data to at least one of the file systems of the second virtualized storage processor.

4. The method of claim 1, further comprising processing IO requests received from the set of hosts via an IO stack, the IO stack having a front end and a back end, the front end including multiple protocol end points, for communicating with hosts using block and file protocols, a storage pool manager, and a cache manager, the back end including a RAID Manager for managing RAID groups used to provide slices from the storage pool.

5. The method of claim 4, further comprising creating a container on a storage processor of the data storage apparatus, the container providing an isolated userspace execution environment on the storage processor, wherein processing IO requests received from the set of hosts using the IO stack includes running both the front end and the back end of the IO stack in the container on the storage processor.

6. The method of claim 4, further comprising:

creating a first container on a first storage processor of the data storage apparatus, the first container providing a first isolated userspace execution environment on the first storage processor; and

creating a second container on a second storage processor of the data storage apparatus, the second container providing a second isolated execution environment on the second storage processor,

wherein processing IO requests received from the set of hosts using the IO stack includes—

running the front end of the IO stack in the first container of the first storage processor to perform front-end processing of the IO requests,

conveying the IO requests processed by the front end in the first container of the first storage processor to the back end in the second container of the second storage processor using an interconnection, and

running the back end of the IO stack in the second container of the second storage processor to perform back-end processing of the IO requests.

7. The method of claim 6, further comprising:

creating a third container on a third storage processor of the data storage apparatus, the third container providing a third isolated userspace execution environment on the third storage processor; and

processing second IO requests received from the set of hosts using the IO stack, wherein processing the second IO requests includes—

running a second front end of the IO stack in the third container of the third storage processor to perform front-end processing of the second IO requests,

conveying the second IO requests processed by the front end in the third container of the third storage processor to the back end in the second container of the second storage processor using an interconnection, and

running the back end of the IO stack in the second container of the second storage processor to perform back-end processing of the second IO requests.

8. The method of claim 4, further comprising:

providing the front end within a storage processor of the data storage system;

providing additional front ends within additional respective storage processors of the data storage system;

providing the back end within a block-based storage array of the data storage system; and

deriving the set of slices of the storage pool from storage multiple drives within the block-based storage array.

9. The method of claim 4, further comprising:

receiving, by a first storage processor of the data storage apparatus, an IO request from a host designating a set of data to be written;

executing a set of instructions in the front end to store the set of data in a cache of the first storage processor;

directing the set of data to be mirrored to a cache in a second storage processor via a connection between the first storage processor and the second storage processor;

acknowledging to the host that the set of data was successfully written when both the first storage processor and the second storage processor acknowledge that the set of data was successfully written to the respective caches; and

destaging the set of data from the cache of on the first storage processor to the set of slices after acknowledging that the set of data was successfully written.

10. The method of claim 9, wherein destaging the set of data from the cache to the set of slices includes mapping the set of data to a particular location in a file stored in the set of file systems, and wherein executing the set of instructions in the front end to store the set of data in the cache of the first storage processor is initiated prior to mapping the set of data to the particular location in the file stored in the set of file systems.

11. The method of claim 1, wherein the storage pool is a first storage pool for providing storage for a first tenant of the data storage apparatus, and wherein the method further comprises storing a second set of file systems of a second virtualized storage processor on slices provisioned from a second storage pool distinct from the first storage pool, the second storage pool providing storage for a second tenant of the data storage apparatus.

12. The method of claim 1, wherein the set of records stored in the configuration database further includes at least one record indicating a host interface that defines a host IO protocol that the respective virtualized storage processor is equipped to handle.

13. The method of claim 1, wherein the set of records defines a plurality of virtualized storage processors that operate together on one physical storage processor of the data storage system, the configuration database storing, for each of the plurality of virtualized storage processors, a set of file systems associated with that virtualized storage processor and a set of host interfaces associated with that virtualized storage processor, and wherein IO requests directed to file systems of different ones of the plurality of virtualized storage processors are processed by a single IO stack operating in the physical storage processor.

14. A data storage apparatus, comprising:

a storage processor including control circuitry, memory accessible by the control circuitry, and multiple hardware interfaces arranged to connect to a set of hosts to enable the set of hosts to access host data stored on the data storage apparatus,

wherein the memory stores instructions which, when carried out by the control circuitry, cause the control circuitry to;

allocate a set of slices of a storage pool to a set of file systems of the data storage apparatus;

store a first file in the set of slices, the first file provided as a file in the set of file systems and providing a file representation of a LUN (logical unit number) storing block-based host data, wherein the block-based host data stored in the first file is accessible to the set of hosts using a block-based protocol;

store a second file in the set of slices, the second file provided as a file in the set of file systems and providing a file representation of a host file system storing file-based host data, wherein the file-based host data stored in the second file is accessible to the set of hosts using a file-based protocol;

deallocate a slice of the set of slices from the set of file systems in response to the slice storing block-based host data of the first file and then becoming empty;

reallocate the slice to the set of file systems in response to the host file system stored in the second file requiring additional storage space; and

while the first file is stored to provide the LUN, store portions of the second file on the reallocated slice, to thereby allow the slice previously used for storing portions of the LUN to be reused for storing portions of the host file system,

wherein the instructions, when carried out by the control circuitry, further cause the control circuitry to process IO requests received from the set of hosts via an IO stack, the IO stack having a front end and a back end, the front end including multiple protocol end points, for communicating with hosts using block and file protocols, a storage pool manager, and a cache manager, the back end including a RAID Manager for managing RAID groups used to provide slices from the storage pool.

15. The data storage apparatus of claim 14, wherein the instructions, when carried out by the control circuitry, further cause the control circuitry to:

perform a data protection operation on the LUN by copying or snapping the first file to a first location; and

perform a data protection operation on the host file system by copying or snapping the second file to a second location,

wherein the first location and the second location are each one of (i) a location within the data storage apparatus and (ii) a location geographically remote from the data storage apparatus.

16. The data storage apparatus of claim 14, wherein the instructions, when carried out by the control circuitry, further cause the control circuitry to:

create a first container on a first storage processor of the data storage apparatus, the first container providing a first isolated userspace execution environment on the first storage processor; and

create a second container on a second storage processor of the data storage apparatus, the second container providing a second isolated execution environment on the second storage processor,

wherein, when caused to process IO requests received from the set of hosts using the IO stack, the control circuitry is further caused to—

run the front end of the IO stack in the first container of the first storage processor to perform front-end processing of the IO requests,

convey the IO requests processed by the front end in the first container of the first storage processor to the back end in the second container of the second storage processor using a physical interconnect, and

run the back end of the IO stack in the second container of the second storage processor to perform back-end processing of the IO requests.

17. The data storage apparatus of claim 16, wherein the first container is a software process the runs on a kernel of an operating system of the first storage processor, and wherein the second container is a software process that runs on a kernel of an operating system of the second storage processor.

18. The data storage apparatus of claim 14, wherein the front end and the back end of the IO stack both reside within a single container, the container running as a process on a kernel of an operating system running on the storage processor and providing an isolated userspace execution environment on the storage processor.

19. A computer program product having a non-transitory computer readable medium including instructions which, when executed by control circuitry of a data storage apparatus, cause the control circuitry to perform a method of managing host data of a set of hosts in a data storage apparatus, the method comprising:

allocating a set of slices of a storage pool to a set of file systems of the data storage apparatus;

storing a first file in the set of slices, the first file provided as a file in the set of file systems and providing a file representation of a LUN (logical unit number) storing block-based host data, wherein the block-based host data stored in the first file is accessible to the set of hosts using a block-based protocol;

storing a second file in the set of slices, the second file provided as a file in the set of file systems and providing a file representation of a host file system storing file-based host data, wherein the file-based host data stored in the second file is accessible to the set of hosts using a file-based protocol;

deallocating a slice of the set of slices from the set of file systems in response to the slice storing block-based host data of the first file and then becoming empty;

reallocating the slice to the set of file systems in response to the host file system stored in the second file requiring additional storage space; and

while the first file is providing the LUN, storing portions of the second file on the reallocated slice, thereby allowing the slice previously used for storing portions of the LUN to be reused for storing portions of the host file system,

wherein the method further comprises;

storing in a configuration database of the data storage apparatus, a set of records defining at least one virtualized storage processor, the set of records identifying multiple file systems of each virtualized storage processor;

storing the file systems of each virtualized storage processor as respective files in the set of slices, each of the respective files provided as a file in the set of file systems and providing a file representation of a file system of the respective virtualized storage processor;

instantiating a set of host interfaces for each of the virtualized storage processors;

receiving, via the set of host interfaces for each of the virtualized storage processors, a respective set of read and write requests from the set of hosts; and

for each virtualized storage processor, processing the respective set of read and write requests received via the respective set of host interfaces, to read data from and write data to at least one of the file systems of the virtualized storage processor.

20. The computer program product of claim 19, wherein the method further comprises:

performing a data protection operation on the LUN by copying or snapping the first file to a first location; and

performing a data protection operation on the host file system by copying or snapping the second file to a second location,

wherein the first location and the second location are each one of (i) a location within the data storage apparatus and (ii) a location geographically remote from the data storage apparatus.