Sequential resilvering of storage devices with reduced restarts

• US 10,671,500 B2
• Filed: 01/17/2017
• Issued: 06/02/2020
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Taylor, Eric Carl, Xiao, Zhu
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US15/407,979
• Publication Number:
  US 20170192864A1
• Time in Patent Office: 1,232 Days
• Field of Search: 714 381
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

A method of resilvering a plurality of failed devices in a storage pool may include detecting a failure of a first storage device in the storage pool, and determining a plurality of data blocks that are stored on the first storage device. The method may also include sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device, and detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device. The method may further include determining that one or more of the plurality of data blocks are also located on the second storage device, and reusing at least part of the one or more buckets to resilver the second storage device.

Claims

1. A method of resilvering a plurality of failed devices in a storage pool, the method comprising:

detecting a failure of a first storage device in the storage pool;

determining a plurality of data blocks that are stored on the first storage device;

coarsely sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device based on disk addresses, such that blocks within each of the one or more buckets are not required to be in disk address order;

detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device;

determining that one or more of the plurality of data blocks are also located on the second storage device; and

reusing at least part of the one or more buckets to resilver the second storage device.

2. The method of claim 1, wherein the storage pool comprises a Redundant Array of Independent Disks (RAID).

3. The method of claim 1, wherein determining the plurality of data blocks that are stored on the first storage device comprises traversing a file system to discover a physical location of each leaf node.

4. The method of claim 1 further comprising:

determining that each block in a bucket in the one or more buckets is also located on the second storage device.

5. The method of claim 1, wherein sorting the plurality of data blocks into one or more buckets comprises performing a second sorting operation that sorts the blocks within each of the one or more buckets according to disk address order.

6. The method of claim 5, wherein the blocks within each of the one or more buckets are sorted into Adelson-Velsky Landis (AVL) tree.

7. The method of claim 1, wherein buckets in the one or more buckets that are not used to resilver the second storage device are deleted prior to resilvering the second storage device.

8. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

detecting a failure of a first storage device in the storage pool;

determining a plurality of data blocks that are stored on the first storage device;

sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device;

detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device;

determining that one or more of the plurality of data blocks are also located on the second storage device;

determining that each data block in a first bucket in the one or more buckets is also located on the second storage device; and

reusing at least part of the one or more buckets to resilver the second storage device, including;

determining that each data block in the first bucket is in the same order on the first storage device as on the second storage device; and

reusing the first bucket to resilver the second storage device without requiring the first bucket to be resorted.

9. The non-transitory computer-readable medium according to claim 8, wherein sorting the plurality of data blocks into one or more buckets comprises performing a first sorting operation that coarsely sorts data blocks into each of the one or more buckets based on disk address, such that blocks within each of the one or more buckets are not required to be in disk address order.

10. The non-transitory computer-readable medium according to claim 8, wherein buckets in the one or more buckets that are not used to resilver the second storage device are deleted prior to resilvering the second storage device.

11. The non-transitory computer-readable medium according to claim 8, wherein the storage pool comprises a Redundant Array of Independent Disks (RAID).

12. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that data blocks in a second bucket in the one or more buckets are in a different order on the first storage device than on the second storage device;

using the second bucket to resilver the first storage device without requiring the second bucket to be resorted;

sorting, after resilvering the first storage device, the data blocks in the second bucket to be in order according to the second storage device; and

using the second bucket to resilver the second storage device.

13. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that at least one data block in a second bucket in the one or more buckets is also located on the second storage device;

determining that at least one data block in the second bucket is not located on the second storage device; and

removing, after resilvering the first storage device, the at least one data block in the second bucket that is not located on the second storage device.

14. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that a first data block in a second bucket in the one or more buckets is also located on the second storage device;

determining that a second data block that should be sorted into the second bucket from the second device is not in the second bucket; and

adding the second data block to the second bucket after resilvering the first storage device.

15. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that no data blocks in a second bucket in the one or more buckets are also located on the second storage device;

deleting the data blocks in the second bucket after resilvering the first storage device; and

sorting new data blocks from the second storage device into the second bucket.

16. A system comprising:

one or more hardware processors; and

one or more memory devices comprising instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising;

detecting a failure of a first storage device in the storage pool;

determining a plurality of data blocks that are stored on the first storage device;

sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device;

detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device;

determining that one or more of the plurality of data blocks are also located on the second storage device; and

reusing at least part of the one or more buckets to resilver the second storage device, wherein buckets in the one or more buckets that are not used to resilver the second storage device are deleted prior to resilvering the second storage device.

17. The system of claim 16 wherein:

the storage pool comprises a Redundant Array of Independent Disks (RAID); and

determining a plurality of data blocks that are stored on the first storage device comprises traversing a file system to discover a physical location of each leaf node.

18. The system of claim 16 wherein:

sorting the plurality of data blocks into one or more buckets comprises performing a first sorting operation that coarsely sorts data blocks into each of the one or more buckets based on disk address, such that blocks within each of the one or more buckets are not required to be in disk address order;

sorting the plurality of data blocks into one or more buckets comprises performing a second sorting operation that sorts the blocks within each of the one or more buckets according to disk address order; and

the blocks within each of the one or more buckets are sorted into Adelson-Velsky Landis (AVL) tree.

19. The system of claim 16, wherein the one or more memory devices further comprise additional instructions that cause the one or more processors to perform additional operations comprising:

determining that each block in a bucket in the one or more buckets is also located on the second storage device;

determining that each block in the bucket is in the same order on the first storage device as on the second storage device; and

reusing the bucket to resilver the second storage device without requiring the bucket to be resorted.

20. The system of claim 16, wherein the one or more memory devices further comprise additional instructions that cause the one or more processors to perform additional operations comprising:

determining that each block in a bucket in the one or more buckets is also located on the second storage device;

determining that blocks in the bucket are in a different order on the first storage device than on the second storage device;

using the bucket to resilver the first storage device without requiring the bucket to be resorted;

sorting, after resilvering the first storage device, the blocks in the bucket to be in order according to the second storage device; and

using the bucket to resilver the second storage device.