SEQUENTIAL RESILVERING OF STORAGE DEVICES WITH REDUCED RESTARTS

• US 20170192864A1
• Filed: 01/17/2017
• Published: 07/06/2017
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Taylor, Eric Carl, Xiao, Zhu
• Granted Patent: US 10,671,500 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

A method of resilvering a plurality of failed devices in a storage pool may include detecting a failure of a first storage device in the storage pool, and determining a plurality of data blocks that are stored on the first storage device. The method may also include sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device, and detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device. The method may further include determining that one or more of the plurality of data blocks are also located on the second storage device, and reusing at least part of the one or more buckets to resilver the second storage device.

Claims

1. A method of resilvering a plurality of failed devices in a storage pool, the method comprising:

detecting a failure of a first storage device in the storage pool;

determining a plurality of data blocks that are stored on the first storage device;

sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device;

detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device;

determining that one or more of the plurality of data blocks are also located on the second storage device; and

reusing at least part of the one or more buckets to resilver the second storage device.

2. The method of claim 1, wherein the storage pool comprises a Redundant Array of Independent Disks (RAID).

3. The method of claim 1, wherein determining the plurality of data blocks that are stored on the first storage device comprises traversing a file system to discover a physical location of each leaf node.

4. The method of claim 1, wherein sorting the plurality of data blocks into one or more buckets comprises performing a first sorting operation that coarsely sorts data blocks into each of the one or more buckets based on disk address, such that blocks within each of the one or more buckets are not required to be in disk address order.

5. The method of claim 4, wherein sorting the plurality of data blocks into one or more buckets comprises performing a second sorting operation that sorts the blocks within each of the one or more buckets according to disk address order.

6. The method of claim 5, wherein the blocks within each of the one or more buckets are sorted into Adelson-Velsky Landis (AVL) tree.

7. The method of claim 1, wherein buckets in the one or more buckets that are not used to resilver the second storage device are deleted prior to resilvering the second storage device.

8. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

detecting a failure of a first storage device in the storage pool;

determining a plurality of data blocks that are stored on the first storage device;

sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device;

detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device;

determining that one or more of the plurality of data blocks are also located on the second storage device; and

reusing at least part of the one or more buckets to resilver the second storage device.

9. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that each block in a bucket in the one or more buckets is also located on the second storage device.

10. The non-transitory computer-readable medium according to claim 9 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that each block in the bucket is in the same order on the first storage device as on the second storage device; and

reusing the bucket to resilver the second storage device without requiring the bucket to be resorted.

11. The non-transitory computer-readable medium according to claim 9 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that blocks in the bucket are in a different order on the first storage device than on the second storage device;

using the bucket to resilver the first storage device without requiring the bucket to be resorted;

sorting, after resilvering the first storage device, the blocks in the bucket to be in order according to the second storage device; and

using the bucket to resilver the second storage device.

12. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that at least one block in a bucket in the one or more buckets is also located on the second storage device;

determining that at least one block in the bucket is not located on the second storage device; and

removing, after resilvering the first storage device, the at least one block in the bucket that is not located on the second storage device.

13. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that a first block in a bucket in the one or more buckets is also located on the second storage device;

determining that a second block that should be sorted into the bucket from the second device is not in the bucket; and

adding the second block to the bucket after resilvering the first storage device.

14. The non-transitory computer-readable medium according to claim 8 comprising additional instruction that cause the one or more processors to perform additional operations comprising:

determining that no blocks in a bucket in the one or more buckets are also located on the second storage device;

deleting the blocks in the bucket after resilvering the first storage device; and

sorting new blocks from the second storage device into the bucket.

15. A system comprising:

one or more processors; and

one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

detecting a failure of a first storage device in the storage pool;

determining a plurality of data blocks that are stored on the first storage device;

sorting the plurality of data blocks into one or more buckets in an order in which the plurality of data blocks are located on the first storage device;

detecting a failure of a second storage device in the storage pool after detecting the failure of the first storage device;

determining that one or more of the plurality of data blocks are also located on the second storage device; and

reusing at least part of the one or more buckets to resilver the second storage device.

16. The system of claim 15 wherein:

the storage pool comprises a Redundant Array of Independent Disks (RAID); and

determining a plurality of data blocks that are stored on the first storage device comprises traversing a file system to discover a physical location of each leaf node.

17. The system of claim 15 wherein:

sorting the plurality of data blocks into one or more buckets comprises performing a first sorting operation that coarsely sorts data blocks into each of the one or more buckets based on disk address, such that blocks within each of the one or more buckets are not required to be in disk address order;

sorting the plurality of data blocks into one or more buckets comprises performing a second sorting operation that sorts the blocks within each of the one or more buckets according to disk address order; and

the blocks within each of the one or more buckets are sorted into Adelson-Velsky Landis (AVL) tree.

18. The system of claim 15 wherein buckets in the one or more buckets that are not used to resilver the second storage device are deleted prior to resilvering the second storage device.

19. The system of claim 15, wherein the one or more memory devices further comprise additional instructions that cause the one or more processors to perform additional operations comprising:

determining that each block in a bucket in the one or more buckets is also located on the second storage device;

determining that each block in the bucket is in the same order on the first storage device as on the second storage device; and

reusing the bucket to resilver the second storage device without requiring the bucket to be resorted.

20. The system of claim 15, wherein the one or more memory devices further comprise additional instructions that cause the one or more processors to perform additional operations comprising:

determining that each block in a bucket in the one or more buckets is also located on the second storage device;

determining that blocks in the bucket are in a different order on the first storage device than on the second storage device;

using the bucket to resilver the first storage device without requiring the bucket to be resorted;

sorting, after resilvering the first storage device, the blocks in the bucket to be in order according to the second storage device; and

using the bucket to resilver the second storage device.