DISCONNECTED INGEST IN A DISTRIBUTED STORAGE SYSTEM

• US 20150319242A1
• Filed: 05/05/2014
• Published: 11/05/2015
• Est. Priority Date: 05/05/2014
• Status: Active Grant

• Current Assignee: DataDirect Networks Incorporated
• Original Assignee: DataDirect Networks Incorporated
• Inventors: Olster, Dan, Fried-Gintis, Adam, Molaro, Donald J.
• Granted Patent: US 10,516,732 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A data storage system allowing for ingest of data when certain storage is unavailable is described herein. The storage system includes zones that are independent and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When data is to be stored in the data storage system according to a specified storage policy and the specified storage policy cannot be achieved, the data is stored according to a fallback storage policy. This allows a client to be able to continue executing without having to wait for a storage anomaly to be corrected or pass. After the data is stored according to a fallback storage policy, the data is at a later time stored according to the specified storage policy.

Claims

1. A data storage system comprising:

a plurality of storage zones coupled to a network, each storage zone comprising a plurality of nodes wherein each node comprises a plurality of storage devices, a server coupled to the network, the server including a processor, memory and local storage, the local storage having instructions stored thereon which when executed cause the server to perform actions including;

receiving a storage request from a client of the data storage system to store a data item, the storage request including a data item reference and a specified storage policy identifier;

evaluating whether storage zones implicated by the specified storage policy identifier are available;

when all the storage zones in the specified storage policy corresponding to the specified storage policy identifier are not available, creating a fallback storage policy for the data item specifying an available alternative storage zone;

storing the data item as an object on storage devices in storage zones according to the fallback storage policy;

creating an entry for the object in an object database including identifiers for both the specified storage policy and the fallback storage policy;

providing the object identifier for the object to the client.

2. The data storage system of claim 1 wherein the storage devices are hard disk drives and/or solid state storage devices.

3. The data storage system of claim 1 wherein the evaluating includes checking whether a storage zone is accessible and/or whether the data communications to the storage zone meets a specified throughput.

4. The data storage system of claim 1 wherein the storage policy identifier includes a replication and placement component specifying replication and placement requirements of the object.

5. The data storage system of claim 1 wherein the storing the data item includes storing the data item as part of an object group of other data items that have the same specified storage policy identifier.

6. The data storage system of claim 1 wherein the local storage of the server has further instructions stored thereon which when executed cause the server to perform further actions including:

receiving a get request including an object identifier of a requested object;

referring to the object database to ascertain the location of the requested object, the location being specified by the fallback policy identifier;

sending a read request to the zone where the requested object was stored according to the fallback identifier to obtain the requested object;

returning the requested object to the client.

7. The data storage system of claim 1 wherein the local storage of the server has further instructions stored thereon which when executed cause the server to perform further actions including:

reviewing the object database for objects with fallback policies and identifying those objects;

for an identified object, evaluating whether storage zones implicated by the specified storage policy are available;

when the storage zones implicated by the specified storage policy are not available, checking again later;

when at least one of the storage zones implicated by the specified storage policy is available,

updating the entry for the identified object in the object database including removing the fallback storage policy designation,

sending an object store request to a now available storage zone for the specified policy for the identified data object,

receiving confirmation that storage of the identified object was successful from the now available storage zone.

8. The data storage system of claim 7 wherein the local storage of the server has further instructions stored thereon which when executed cause the server to perform further actions including:

when at least one of the storage zones implicated by the specified storage policy is available, releasing a fallback version of the identified object.

9. A method for storing data comprising:

receiving from a client computing device a storage request to store a data item, the storage request including a data item reference and a specified storage policy identifier;

evaluating whether storage zones of a plurality of storage zones implicated by the specified storage policy identifier are available, wherein each storage zone includes a plurality of storage nodes and each storage node includes a plurality of storage devices;

when all the storage zones in the specified storage policy corresponding to the specified storage policy identifier are not available, creating a fallback storage policy for the data item specifying an available alternative storage zone;

storing the data item as an object on storage devices in storage zones according to the fallback storage policy;

creating an entry for the object in an object database including identifiers for both the specified storage policy and the fallback storage policy;

providing the object identifier for the object to the client computing device.

10. The method of claim 9 wherein the storage devices are hard disk drives and/or solid state storage devices.

11. The method of claim 9 wherein the evaluating includes checking whether a storage zone is accessible and/or whether the data communications to the storage zone meets a specified throughput.

12. The method of claim 9 wherein the specified storage policy identifier includes a replication and placement component specifying replication and placement requirements of the object.

13. The method of claim 9 wherein the storing the data item includes storing the data item as part of an object group of other data items that have the same specified storage policy identifier.

14. The method of claim 9 further comprising:

receiving a get request including an object identifier of a requested object;

referring to the object database to ascertain the location of the requested object, the location being specified by the fallback policy identifier;

sending a read request to the zone where the requested object was stored according to the fallback identifier to obtain the requested object;

returning the requested object to the client computing device.

15. The method of claim 9 further comprising:

reviewing the object database for objects with fallback policies and identifying those objects;

for an identified object, evaluating whether storage zones implicated by the specified storage policy are available;

when the storage zones implicated by the specified storage policy are not available, checking again later;

when the storage zones implicated by the specified storage policy are available,

updating the entry for the identified object in the object database including removing the fallback storage policy designation,

sending an object store request to a now available storage zone for the specified policy for the identified data object,

receiving confirmation that storage of the identified object was successful from the now available storage zone.

16. The method of claim 15 further comprising:

when at least one of the storage zones implicated by the specified storage policy is available, releasing a fallback version of the identified object.