Scalable distributed storage system infrastructure

• US 9,858,013 B1
• Filed: 06/30/2014
• Issued: 01/02/2018
• Est. Priority Date: 05/05/2014
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Srivastav, Shashwat, Sankaran, Sriram, Shah, Vishrut, Zhang, Qi, Luo, Jun, Wang, Chen, Yuan, Huapeng, Navaneethakrishnan, Karthik, Song, Jie, Yin, Wei
• Primary Examiner(s): Chappell, Daniel C
• Application Number:
  US14/319,368
• Time in Patent Office: 1,282 Days
• Field of Search: 711114
• CPC Class Codes: G06F 12/0831 using a bus scheme, e.g. with bus monitoring or watching means
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0655 Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

A computer-executable method, computer program product, and system for managing metadata within a distributed data storage system, including a compute node in communication with a data storage array, the computer-executable method, computer program product, and system comprising receiving a data I/O from an application executing within the distributed data storage system, and creating a first storage system within the compute node, wherein the first storage system is enabled to manage metadata related to the data I/O, and processing the data I/O using the first storage system.

Claims

1. A computer-executable method for managing metadata within a distributed data storage system, including a compute node in communication with a data storage array, the computer-executable method comprising:

receiving a data I/O from an application executing within the distributed data storage system; and

creating a first storage system within the compute node, wherein the first storage system is enabled to manage metadata related to the data I/O, wherein the first storage system is enabled to create one or more storage systems within the first storage system to manage the metadata;

dynamically creating and managing a first level of sub storage systems to manage portions of the metadata associated with the data I/O, wherein each of the first level sub storage systems are enabled to create and manage a second level of sub storage systems wherein each of the second level of sub storage systems are enabled to manage portions of the portions of metadata managed by the first level sub storage system associated with each of the second level sub storage systems; and

processing the data I/O using the first storage system, wherein the processing comprises creating a first portion of metadata related to the data I/O;

analyzing the first portion of metadata to make a determination based on one or more characteristics of the first portion of metadata;

upon a positive determination, creating a second storage system within the compute node, wherein the second storage system is enabled to manage metadata created by the first storage system;

creating a second portion of metadata based on the first portion of metadata; and

storing the second portion of metadata on the data storage array.

2. The computer-executable method of claim 1, wherein a first characteristic of the one or more characteristic of the metadata is an amount of the metadata.

3. A system, comprising:

a distributed data storage system including a compute node in communication with a data storage array; and

computer-executable program logic encoded in memory of one or more computers in communication with the distributed data storage system to enable management of metadata within the distributed data storage system, wherein the computer-executable program logic is configured for the execution of;

receiving a data I/O from an application executing within the distributed data storage system; and

creating a first storage system within the compute node, wherein the first storage system is enabled to manage metadata related to the data I/O, wherein the first storage system is enabled to create one or more storage systems within the first storage system to manage the metadata;

dynamically creating and managing a first level of sub storage systems to manage portions of the metadata associated with the data I/O, wherein each of the first level sub storage systems are enabled to create and manage a second level of sub storage systems wherein each of the second level of sub storage systems are enabled to manage portions of the portions of metadata managed by the first level sub storage system associated with each of the second level sub storage systems; and

processing the data I/O using the first storage system 22, wherein the processing comprises creating a first portion of metadata related to the data I/O;

analyzing the first portion of metadata to make a determination based on one or more characteristics of the first portion of metadata;

upon a positive determination, creating a second storage system within the compute node, wherein the second storage system is enabled to manage metadata created by the first storage system;

creating a second portion of metadata based on the first portion of metadata; and

storing the second portion of metadata on the data storage array.

4. The system of claim 3, wherein a first characteristic of the one or more characteristic of the metadata is an amount of the metadata.

5. A computer program product for managing metadata within a distributed data storage system, including a compute node in communication with a data storage array, the computer program product comprising:

a non-transitory computer readable medium encoded with computer-executable program code for using read signatures in replication, the code configured to enable the execution of;

receiving a data I/O from an application executing within the distributed data storage system; and

creating a first storage system within the compute node, wherein the first storage system is enabled to manage metadata related to the data I/O, wherein the first storage system is enabled to create one or more storage systems within the first storage system to manage the metadata;

dynamically creating and managing a first level of sub storage systems to manage portions of the metadata associated with the data I/O, wherein each of the first level sub storage systems are enabled to create and manage a second level of sub storage systems wherein each of the second level of sub storage systems are enabled to manage portions of the portions of metadata managed by the first level sub storage system associated with each of the second level sub storage systems; and

processing the data I/O using the first storage system 21, wherein the processing comprises creating a first portion of metadata related to the data I/O;

analyzing the first portion of metadata to make a determination based on one or more characteristics of the first portion of metadata;

upon a positive determination, creating a second storage system within the compute node, wherein the second storage system is enabled to manage metadata created by the first storage system;

creating a second portion of metadata based on the first portion of metadata; and

storing the second portion of metadata on the data storage array.

6. The computer program product of claim 5, wherein a first characteristic of the one or more characteristic of the metadata is an amount of the metadata.