Virtual file structure for data storage system

• US 10,769,177 B1
• Filed: 05/13/2019
• Issued: 09/08/2020
• Est. Priority Date: 09/02/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Bernbo, Stefan, Melander, Christian, Petersson, Gustav, Persson, Roger
• Primary Examiner(s): Truong, Lechi
• Application Number:
  US16/410,716
• Time in Patent Office: 484 Days
• Field of Search: 719310
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1834 implemented based on peer-to-peer networks, e.g. gnutella p2p communication protocols H04L67/104
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: H04L 41/0213 Standardised network management protocols, e.g. simple network management protocol [SNMP]

Abstract

An exemplary method includes identifying, within a first data item stored in a data storage system, a reference to a second data item stored in the data storage system, determining, based on the reference, that a storage location of the second data item is within a cluster of storage nodes included in the data storage system, sending, based on the determining that the storage location of the second data item is within the cluster of storage nodes, a request for the second data item to the storage nodes included in the cluster, and receiving, based on the request, the second data item from a particular storage node included in the cluster.

Claims

1. A method comprising:

identifying, within a collection object stored in a data storage system, a reference to a data item stored in the data storage system, the collection object corresponding to a directory of a distributed file system and the reference including an address of the data item in the storage system;

determining, based on the reference within the collection object, that a storage location of the data item is within a cluster of storage nodes included in the data storage system;

sending, based on the determining that the storage location of the data item is within the cluster of storage nodes, a request for the data item to the storage nodes included in the cluster;

receiving, in response to the request, one or more responses from one or more storage nodes, the one or more responses indicating that the one or more storage nodes each store the data item;

selecting, based on data included in the one or more responses, a particular storage node included in the one or more storage nodes;

sending an additional request for the data item to the particular storage node; and

receiving, in response to the additional request, the data item from the particular storage node;

wherein the identifying, the determining, the sending of the request, and the receiving of the data item are performed using an application programming interface (API).

2. The method of claim 1, wherein the selecting of the particular storage node comprises determining, based on the data included in the one or more responses, that the particular storage node has a lowest execution load out of all storage nodes included in the one or more storage nodes.

3. The method of claim 1, wherein the additional request comprises a unicast request.

4. The method of claim 1, wherein:

the collection object represents a directory in a virtual file structure; and

the data item is

a second collection object that represents a subdirectory of the directory in the virtual file structure, or

a data file that includes payload data.

5. The method of claim 4, wherein:

the collection object and the data item are stored in an unstructured storage layer of the storage system; and

the virtual file structure implements a structured layer for the unstructured storage layer.

6. The method of claim 1, wherein the API is executed by a device configured to communicate with the data storage system by way of a network.

7. The method of claim 1, wherein the API is executed by a storage node included in the data storage system.

8. The method of claim 1, wherein:

the address comprises a multicast address; and

the sending of the request comprises sending a multicast request to the multicast address.

9. The method of claim 1, further comprising:

Identifying, within the data item, a reference to an additional data item stored in the data storage system;

determining, based on the reference to the additional data item, that a storage location of the additional data item is within a second cluster of storage nodes included in the data storage system;

sending, based on the determining that the storage location of the additional data item is within the second cluster of storage nodes, a request for the additional data item to the storage nodes included in the second cluster; and

receiving, based on the request for the additional data item, the additional data item from a particular storage node included in the second cluster.

10. The method of claim 9, wherein the second cluster of storage nodes is the cluster of storage nodes.

11. The method of claim 9, wherein the second cluster of storage nodes is different than the cluster of storage nodes.

12. A device comprising a processor configured to:

identify, within a collection object stored in a data storage system, a reference to a data item stored in the data storage system, the collection object corresponding to a directory of a distributed file system and the reference including an address of the data item in the storage system;

determine, based on the reference within the collection object, that a storage location of the data item is within a cluster of storage nodes included in the data storage system;

send, based on the determining that the storage location of the data item is within the cluster of storage nodes, a request for the data item to the storage nodes included in the cluster;

receive, in response to the request, one or more responses from one or more storage nodes, the one or more responses indicating that the one or more storage nodes each store the data item;

select, based on data included in the one or more responses, a particular storage node included in the one or more storage nodes;

send an additional request for the data item to the particular storage node; and

receive, in response to the additional request, the data item from the particular storage node;

wherein the identifying, the determining, the sending of the request, and the receiving of the data item are performed using an application programming interface (API).

13. The device of claim 12, wherein the device is implemented by a server configured to communicate with the data storage system by way of a network.

14. The device of claim 12, wherein the device is implemented by a storage node within the data storage system.

15. The device of claim 12, wherein:

the second data item is

a second collection object that represents a subdirectory of the directory, or

a data file that includes payload data.

16. The device of claim 15, wherein:

the collection object and the data item are stored in an unstructured storage layer of the storage system; and

the virtual file structure implements a structured layer for the unstructured storage layer.

17. The device of claim 12, wherein the selecting of the particular storage node comprises determining, based on the data included in the one or more responses, that the particular storage node has a lowest execution load out of all storage nodes included in the one or more storage nodes.