Reconstructing data stored across archival data storage devices

• US 9,430,321 B2
• Filed: 05/13/2014
• Issued: 08/30/2016
• Est. Priority Date: 05/13/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Slik, David
• Primary Examiner(s): Bonzo, Bryce
• Assistant Examiner(s): NGUYEN, THIEN DANG
• Application Number:
  US14/276,676
• Publication Number:
  US 20150331744A1
• Time in Patent Office: 840 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: H03M 13/37 Decoding methods or techniques, not specific to the particular type of coding provided for in groups H03M13/03 - H03M13/35
• CPC Class Codes: H03M 13/373 with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes
• CPC Class Codes: H04L 65/40 Support for services or applications
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Techniques for operating a storage system are disclosed. A read request with an object identifier for a data object is received. A synchronous group of data storage devices across a plurality of enclosures is identified. The synchronous group is associated with the object identifier. A request is sent to the plurality of enclosures to synchronously activate the data storage devices in the synchronous group. After sending the request, data fragments associated with the object identifier are retrieved from the synchronous group of data storage devices. The data fragments are erasure decoded into a contiguous data range to reconstruct the data object.

Claims

1. A method of operating a storage system, comprising:

receiving a read request with an object identifier for a data object;

identifying a synchronous group of data storage devices across a plurality of enclosures, wherein the synchronous group of data storage devices is associated with the object identifier, wherein each of the plurality of enclosures comprises a plurality of data storage devices;

sending a request to the plurality of enclosures to synchronously activate the data storage devices in the synchronous group of data storage devices;

retrieving metadata based, at least in part, on the object identifier;

after sending the request to the plurality of enclosures, retrieving data fragments associated with the object identifier from the synchronous group of data storage devices;

determining, from the retrieved metadata, a first set of operations and an order of the first set of operations,

wherein the first set of operations were performed on the data object in the order,

wherein the first set of operations comprise at least two of compression, encryption, deduplication, or erasure coding; and

performing on the data fragments a second set of operations corresponding to the first set of operations in reverse of the order to reconstruct the data object.

2. The method of claim 1, wherein sending the request comprises sending the request to a superset of enclosures that includes the plurality of enclosures, wherein the request identifies at least the synchronous group of data storage devices and the data fragments.

3. The method of claim 1, further comprising:

determining identifiers of the synchronous group of data storage devices, wherein the request comprises the identifiers of the data storage devices.

4. The method of claim 1, further comprising:

determining identifiers of the plurality of enclosures, wherein the request comprises the identifiers of the plurality of enclosures.

5. The method of claim 1, wherein the synchronous group of data storage devices share a common attribute across the plurality of enclosures.

6. The method of claim 5, wherein the common attribute is a position of each of the synchronous group of data storage devices within a respective enclosure of the plurality of enclosures.

7. The method of claim 1, wherein a number of the data fragments retrieved from the synchronous group of data storage devices is sufficient to erasure decode and reconstruct the data object.

8. The method of claim 1, further comprising:

determining a reception efficiency stored in the metadata, wherein retrieving the data fragments is based, at least in part, on the reception efficiency.

9. A non-transitory computer-readable storage memory having executable instructions stored thereon, the executable instructions to:

receive a read request with an object identifier for a data object;

identify a first synchronous group of data storage devices across a first plurality of enclosures, wherein the first synchronous group of data storage devices is associated with the object identifier, wherein each of the first plurality of enclosures comprises a plurality of data storage devices;

send a request to the first plurality of enclosures to synchronously activate the data storage devices in the first synchronous group of data storage devices; and

retrieve metadata based, at least in part, on the object identifier;

after sending the request to the first plurality of enclosures, retrieve first data fragments associated with the object identifier from the first synchronous group of data storage devices;

determine, from the retrieved metadata, a first set of operations and an order of the first set of operations,

wherein the first set of operations were performed on the data object in the order,

wherein the first set of operations comprise at least two of compression, encryption, deduplication, or erasure coding; and

perform on the first data fragments a second set of operations corresponding to the first set of operations in reverse of the order to reconstruct the data object.

10. The non-transitory computer-readable storage memory of claim 9, wherein a number of the data fragments retrieved from the synchronous group of data storage devices are sufficient to erasure decode and reconstruct the data object.

11. The non-transitory computer-readable storage memory of claim 9, wherein the executable instructions further comprise executable instructions to:

determine a reception efficiency stored in the metadata, wherein the executable instructions to retrieve the data fragments is based, at least in part, on the reception efficiency.

12. The non-transitory computer-readable storage memory of claim 9, wherein the executable instructions further comprise executable instructions to:

receive a write request including payload data;

perform the first set of operations on the payload data to transform the payload data into second data fragments;

identify a second synchronous group of data storage devices across a second plurality of enclosures;

store the second data fragments across the data storage devices of the second synchronous group of data storage devices; and

associate the payload data with a group identifier of the second synchronous group of data storage devices.

13. The non-transitory computer-readable storage memory of claim 9, wherein the executable instructions to send the request comprises executable instructions to send the request to a superset of enclosures that includes the first plurality of enclosures, wherein the request identifies at least the synchronous group of data storage devices and the first data fragments.

14. The non-transitory computer-readable storage memory of claim 9, wherein the executable instructions further comprises executable instructions to determine identifiers of the synchronous group of data storage devices, wherein the request comprises the identifiers of the synchronous group of data storage devices.

15. An apparatus comprising:

a hardware processor; and

a computer-readable storage medium having instructions embodied therewith, the instructions comprising program code which, when executed by the processor, cause the apparatus to;

receive a read request with an object identifier for a data object;

identify a synchronous group of data storage devices across a plurality of enclosures, wherein the synchronous group of data storage devices is associated with the object identifier, wherein each of the plurality of enclosures comprises a plurality of data storage devices;

send a request to the plurality of enclosures to synchronously activate the data storage devices in the synchronous group of data storage devices;

retrieve metadata based, at least in part, on the object identifier;

after sending the request to the plurality of enclosures, retrieve data fragments associated with the object identifier from the synchronous group of data storage devices;

determine, from the retrieved metadata, a first set of operations and an order of the first set of operations,

wherein the first set of operations were performed on the data object in the order,

wherein the first set of operations comprise at least two of compression, encryption, deduplication, or erasure coding; and

perform on the data fragments a second set of operations corresponding to the first set of operations in reverse of the order to reconstruct the data object.

16. The apparatus of claim 15, wherein the instructions further comprise instructions which, when executed by the processor, cause the apparatus to:

determine a reception efficiency stored in the metadata, wherein instructions to retrieve the data fragments are based, at least in part, on the reception efficiency.

17. The apparatus of claim 15, wherein a number of the data fragments retrieved from the synchronous group of data storage devices are sufficient to erasure decode and reconstruct the data object.

18. The apparatus of claim 15, wherein the instructions to send the request comprise instructions which, when executed by the processor, cause the apparatus to send the request to a superset of enclosures that includes the plurality of enclosures, wherein the request identifies at least the synchronous group of data storage devices and the data fragments.

19. The apparatus of claim 15, wherein the instructions further comprise instructions which, when executed by the processor, cause the apparatus to determine identifiers of the synchronous group of data storage devices, wherein the request comprises the identifiers of the synchronous group of data storage devices.