Object store architecture for distributed data processing system

• US 9,582,213 B2
• Filed: 11/12/2015
• Issued: 02/28/2017
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Makkar, Gaurav, Narayanamurthy, Srinivasan, Muthyala, Kartheek, Daniel, Stephen
• Primary Examiner(s): Peugh, Brian
• Assistant Examiner(s): Bone, Dustin
• Application Number:
  US14/939,760
• Publication Number:
  US 20160062694A1
• Time in Patent Office: 474 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

Embodiments described herein provide an object store that efficiently manages and services objects for use by clients of a distributed data processing system. Illustratively, the object store may be embodied as a quasi-shared storage system that interacts with nodes of the distributed data processing system to service the objects as blocks of data stored on a plurality of storage devices, such as disks, of the storage system. To that end, an architecture of the object store may include an on-disk layout, e.g., of the storage system, and an incore layout, e.g., of the nodes, that cooperate to illustratively convert the blocks to objects for access by the clients.

Claims

1. A method comprising:

mapping each compute node of a plurality of compute nodes to a different volume of a plurality of volumes that defines a logical arrangement of storage space of a storage system, wherein each compute node has read-write access to the volume to which the compute node is mapped and read-only access to those of the plurality of volumes not mapped to the compute node; and

in response to receiving an identifier for a data object at a first compute node of the plurality of compute nodes,

identifying an object descriptor associated with the identifier and a plurality of chunk descriptors referenced by the object descriptor, wherein the object descriptor and the plurality of chunk descriptors are arranged on the first compute node according to an incore layout;

utilizing the plurality of chunk descriptors to retrieve, via a network from the plurality of volumes, blocks of data stored according to an on-disk layout of the plurality of volumes, wherein the blocks of data include a plurality of chunks described by the plurality of chunk descriptors;

the first compute node assembling the plurality of chunks described by the plurality of chunk descriptors into the data object; and

returning the assembled data object.

2. The method of claim 1, wherein read-write access and read-only access assignments for the plurality of compute nodes are indicated in a configuration file;

wherein the configuration file loaded into memory of each of the plurality of compute nodes upon booting up.

3. The method of claim 1 further comprising:

in response to a failure of a first compute node of the plurality of compute nodes which has read-write access to a first unique set of volumes of the plurality of volumes,

routing read requests directed to the first unique set of volumes to compute nodes in the plurality of compute nodes other than the first compute node; and

routing write requests directed to the first compute node to a second compute node of the plurality of compute nodes, wherein the second compute node executes the write requests on a second unique set of volumes of the plurality of volumes.

4. The method of claim 1 further comprising:

in response to detecting a change to a first chunk of the plurality of chunks by the first compute node,

updating, by a second compute node of the plurality of compute nodes, a chunk descriptor in metadata in memory of the second compute node to reflect the change to the first chunk, wherein the chunk descriptor describes the first chunk; and

in response to detecting an error during the update of the chunk descriptor in the metadata, performing a reboot of the second compute node to reload the metadata in the memory of the second compute node.

5. The method of claim 1 further comprising:

routing write requests directed to a first unique set of volumes of the plurality of volumes to the first compute node, wherein the first compute node has been assigned read-write access to the first unique set of volumes; and

performing load balancing of read requests directed to the first unique set of volumes across each of the plurality of compute nodes.

6. The method of claim 1 further comprising:

storing block layout information on a master segment of a first volume the plurality of volumes, wherein the block layout information describes locations of the blocks of data and the plurality of chunks in the plurality of volumes;

loading the block layout information from the master segment in memory of at least the first compute node; and

utilizing the block layout information to link each of the plurality of chunk descriptors to the plurality of chunks stored in the blocks of data.

7. The method of claim 1, wherein a first chunk descriptor of the plurality of chunk descriptors comprises metadata that describes a location of a chunk with respect to a block of data on the plurality of volumes and a size of the chunk.

8. A computer readable medium having computer executable instructions stored therein, the instructions to:

map each compute node of a plurality of compute nodes to a different volume of a plurality of volumes that defines a logical arrangement of storage space of a storage system, wherein each compute node has read-write access to the volume to which the compute node is mapped and read-only access to those of the plurality of volumes not mapped to the compute node; and

in response to receipt of an identifier for a data object at a first compute node of the plurality of compute nodes,

identify an object descriptor associated with the identifier and a plurality of chunk descriptors referenced by the object descriptor, wherein the object descriptor and the plurality of chunk descriptors are arranged on the first compute node according to an incore layout;

utilize the plurality of chunk descriptors to retrieve blocks of data from the plurality of volumes, wherein the blocks of data are stored according to an on-disk layout of the plurality of volumes, wherein the blocks of data include a plurality of chunks described by the plurality of chunk descriptors;

assemble the plurality of chunks described by the plurality of chunk descriptors into the data object; and

return the assembled data object.

9. The computer readable medium of claim 8, wherein read-write access and read-only access assignments for the plurality of compute nodes are indicated in a configuration file;

wherein the configuration file loaded into memory of each of the plurality of compute nodes upon booting up.

10. The computer readable medium of claim 8 further comprising instructions to:

in response to a failure of a first compute node of the plurality of compute nodes which has read-write access to a first unique set of volumes of the plurality of volumes,

route read requests directed to the first unique set of volumes to compute nodes in the plurality of compute nodes other than the first compute node; and

route write requests directed to the first compute node to a second compute node of the plurality of compute nodes, wherein the second compute node executes the write requests on a second unique set of volumes of the plurality of volumes.

11. The computer readable medium of claim 8 further comprising instructions to:

in response to detection of a change to a first chunk of the plurality of chunks by the first compute node,

update, by a second compute node of the plurality of compute nodes, a chunk descriptor in metadata in memory of the second compute node to reflect the change to the first chunk, wherein the chunk descriptor describes the first chunk; and

in response to detection of an error during the update of the chunk descriptor in the metadata, perform a reboot of the second compute node to reload the metadata in the memory of the second compute node.

12. The computer readable medium of claim 8 further comprising instructions to:

route write requests directed to a first unique set of volumes of the plurality of volumes to the first compute node, wherein the first compute node has been assigned read-write access to the first unique set of volumes; and

perform load balancing of read requests directed to the first unique set of volumes across each of the plurality of compute nodes.

13. The computer readable medium of claim 8 further comprising instructions to:

store block layout information on a master segment of a first volume of the plurality of volumes, wherein the block layout information describes locations of the blocks of data and the plurality of chunks in the plurality of volumes;

load the block layout information from the master segment in memory of at least the first compute node; and

utilize the block layout information to link each of the plurality of chunk descriptors to the plurality of chunks stored in the blocks of data.

14. The computer readable medium of claim 8, wherein a first chunk descriptor of the plurality of chunk descriptors comprises metadata that describes a location of a chunk with respect to a block of data on the plurality of volumes and a size of the chunk.

15. An apparatus comprising:

a processor; and

a machine-readable medium having program code executable by the processor to cause the apparatus to,

map each compute node of a plurality of compute nodes to a different volume of a plurality of volumes that defines a logical arrangement of storage space of a storage system, wherein each compute node has read-write access to the volume to which the compute node is mapped and read-only access to those of the plurality of volumes not mapped to the compute node; and

in response to receipt of an identifier for a data object at a first compute node of the plurality of compute nodes,

identify an object descriptor associated with the identifier and a plurality of chunk descriptors referenced by the object descriptor, wherein the object descriptor and the plurality of chunk descriptors are arranged on the first compute node according to an incore layout;

utilize the plurality of chunk descriptors to retrieve, from the plurality of volumes via a network, blocks of data stored according to an on-disk layout of the plurality of volumes, wherein the blocks of data include a plurality of chunks described by the plurality of chunk descriptors;

the first compute node assemble the plurality of chunks described by the plurality of chunk descriptors into the data object; and

return the assembled data object.

16. The apparatus of claim 15, wherein read-write access and read-only access assignments for the plurality of compute nodes are indicated in a configuration file;

wherein the configuration file loaded into memory of each of the plurality of compute nodes upon booting up.

17. The apparatus of claim 15 further comprising program code executable by the processor to cause the apparatus to:

in response to a failure of a first compute node of the plurality of compute nodes which has read-write access to a first unique set of volumes of the plurality of volumes,

route read requests directed to the first unique set of volumes to compute nodes in the plurality of compute nodes other than the first compute node; and

route write requests directed to the first compute node to a second compute node of the plurality of compute nodes, wherein the second compute node executes the write requests on a second unique set of volumes of the plurality of volumes.

18. The apparatus of claim 15 further comprising program code executable by the processor to cause the apparatus to:

in response to detection of a change to a first chunk of the plurality of chunks by the first compute node,

update, by a second compute node of the plurality of compute nodes, a chunk descriptor in metadata in memory of the second compute node to reflect the change to the first chunk, wherein the chunk descriptor describes the first chunk; and

in response to detection of an error during the update of the chunk descriptor in the metadata, perform a reboot of the second compute node to reload the metadata in the memory of the second compute node.

19. The apparatus of claim 15 further comprising program code executable by the processor to cause the apparatus to:

route write requests directed to a first unique set of volumes of the plurality of volumes to the first compute node, wherein the first compute node has been assigned read-write access to the first unique set of volumes; and

perform load balancing of read requests directed to the first unique set of volumes across each of the plurality of compute nodes.

20. The apparatus of claim 15 further comprising program code executable by the processor to cause the apparatus to:

store block layout information on a master segment of a first volume of the plurality of volumes, wherein the block layout information describes locations of the blocks of data and the plurality of chunks in the plurality of volumes;

load the block layout information from the master segment in memory of at least the first compute node; and

utilize the block layout information to link each of the plurality of chunk descriptors to the plurality of chunks stored in the blocks of data.