ARCHIVING DATA IN CLOUD/OBJECT STORAGE USING LOCAL METADATA STAGING

• US 20200019531A1
• Filed: 07/13/2018
• Published: 01/16/2020
• Est. Priority Date: 07/13/2018
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Kashi Visvanathan, Satish Kumar, Sarda, Pooja, Kandambakkam, Arun, Zucca, Michael, Langouev, Ilya
• Granted Patent: US 10,853,312 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Techniques for archiving data in cloud/object storage using local metadata staging are provided. In one set of embodiments, a computer system residing at an on-premises site comprising on-premises storage can receive a snapshot of a dataset to be archived. The computer system can package data in the snapshot into one or more fixed-size data chunks and upload the one or more fixed-size data chunks to cloud/object storage. Further, concurrently with the packaging and the uploading, the computer system can stage metadata for the snapshot in the on-premises storage. Then, upon uploading all of the data of the snapshot, the computer system can upload the metadata staged in the on-premises storage to the cloud/object storage.

Claims

1. A method for archiving data in cloud/object storage using local metadata staging, the method comprising:

receiving, by a computer system residing at an on-premises site comprising on-premises storage, a snapshot of a dataset to be archived;

packaging, by the computer system, data in the snapshot into one or more fixed-size data chunks and uploading the one or more fixed-size data chunks to the cloud/object storage;

concurrently with the packaging and the uploading, staging, by the computer system, metadata for the snapshot in the on-premises storage; and

upon uploading all of the data of the snapshot, uploading, by the computer system, the metadata staged in the on-premises storage to the cloud/object storage.

2. The method of claim 1 wherein staging the metadata for the snapshot in the on-premises storage comprises:

creating and updating, in the on-premises storage, a B+ tree for the snapshot comprising internal nodes and leaf nodes, each leaf node representing a data block of the snapshot and including a pointer to a cloud physical block address (CPBA) in the cloud/object storage where the data block has been or will be uploaded.

3. The method of claim 1 wherein uploading the metadata comprises:

packaging the metadata into one or more fixed-size metadata chunks and uploading the one or more fixed-size metadata chunks to the cloud/object storage.

4. The method of claim 1 wherein a predefined range of local physical block addresses (LPBAs) is reserved for snapshot metadata for the dataset in the on-premises storage, and wherein an identical predefined range of CPBAs is reserved for snapshot metadata for the dataset in the cloud/object storage.

5. The method of claim 4 wherein staging the metadata for the snapshot in the on-premises storage comprises:

allocating blocks for the metadata sequentially from the predefined range of LPBAs.

6. The method of claim 5 wherein uploading the metadata staged in the on-premises storage to the cloud/object storage comprises:

packaging the metadata into one or more fixed-size metadata chunks having sequential chunk identifiers from the predefined range of CPBAs and uploading the one or more fixed-size metadata chunks with the sequential chunk identifiers to the cloud/object storage, such that the CPBA of each block of the metadata in the cloud/object storage maps directly to the LPBA of said each block of the metadata in the on-premises storage.

7. The method of claim 1 wherein staging the metadata for the snapshot in the on-premises storage comprises:

allocating blocks for the metadata randomly from a local physical block address space on the on-premises storage.

8. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system residing at an on-premises site comprising on-premises storage, the program code embodying a method for archiving data in cloud/object storage using local metadata staging, the method comprising:

receiving a snapshot of a dataset to be archived;

packaging data in the snapshot into one or more fixed-size data chunks and uploading the one or more fixed-size data chunks to the cloud/object storage;

concurrently with the packaging and the uploading, staging metadata for the snapshot in the on-premises storage; and

upon uploading all of the data of the snapshot, uploading the metadata staged in the on-premises storage to the cloud/object storage.

9. The non-transitory computer readable storage medium of claim 8 wherein staging the metadata for the snapshot in the on-premises storage comprises:

creating and updating, in the on-premises storage, a B+ tree for the snapshot comprising internal nodes and leaf nodes, each leaf node representing a data block of the snapshot and including a pointer to a cloud physical block address (CPBA) in the cloud/object storage where the data block has been or will be uploaded.

10. The non-transitory computer readable storage medium of claim 8 wherein uploading the metadata comprises:

packaging the metadata into one or more fixed-size metadata chunks and uploading the one or more fixed-size metadata chunks to the cloud/object storage.

11. The non-transitory computer readable storage medium of claim 8 wherein a predefined range of local physical block addresses (LPBAs) is reserved for snapshot metadata for the dataset in the on-premises storage, and wherein an identical predefined range of CPBAs is reserved for snapshot metadata for the dataset in the cloud/object storage.

12. The non-transitory computer readable storage medium of claim 11 wherein staging the metadata for the snapshot in the on-premises storage comprises:

allocating blocks for the metadata sequentially from the predefined range of LPBAs.

13. The non-transitory computer readable storage medium of claim 12 wherein uploading the metadata staged in the on-premises storage to the cloud/object storage comprises:

packaging the metadata into one or more fixed-size metadata chunks having sequential chunk identifiers from the predefined range of CPBAs and uploading the one or more fixed-size metadata chunks with the sequential chunk identifiers to the cloud/object storage, such that the CPBA of each block of the metadata in the cloud/object storage maps directly to the LPBA of said each block of the metadata in the on-premises storage.

14. The non-transitory computer readable storage medium of claim 8 wherein staging the metadata for the snapshot in the on-premises storage comprises:

allocating blocks for the metadata randomly from a local physical block address space on the on-premises storage.

15. A computer system residing at an on-premises site including on-premises storage, the computer system comprising:

a processor; and

a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to;

receive a snapshot of a dataset to be archived;

package data in the snapshot into one or more fixed-size data chunks and uploading the one or more fixed-size data chunks to cloud/object storage;

concurrently with the packaging and the uploading, stage metadata for the snapshot in the on-premises storage; and

upon uploading all of the data of the snapshot, upload the metadata staged in the on-premises storage to the cloud/object storage.

16. The computer system of claim 15 wherein staging the metadata for the snapshot in the on-premises storage comprises:

creating and updating, in the on-premises storage, a B+ tree for the snapshot comprising internal nodes and leaf nodes, each leaf node representing a data block of the snapshot and including a pointer to a cloud physical block address (CPBA) in the cloud/object storage where the data block has been or will be uploaded.

17. The computer system of claim 15 wherein uploading the metadata comprises:

packaging the metadata into one or more fixed-size metadata chunks and uploading the one or more fixed-size metadata chunks to the cloud/object storage.

18. The computer system of claim 15 wherein a predefined range of local physical block addresses (LPBAs) is reserved for snapshot metadata for the dataset in the on-premises storage, and wherein an identical predefined range of CPBAs is reserved for snapshot metadata for the dataset in the cloud/object storage.

19. The computer system of claim 18 wherein staging the metadata for the snapshot in the on-premises storage comprises:

allocating blocks for the metadata sequentially from the predefined range of LPBAs.

20. The computer system of claim 19 wherein uploading the metadata staged in the on-premises storage to the cloud/object storage comprises:

packaging the metadata into one or more fixed-size metadata chunks having sequential chunk identifiers from the predefined range of CPBAs and uploading the one or more fixed-size metadata chunks with the sequential chunk identifiers to the cloud/object storage, such that the CPBA of each block of the metadata in the cloud/object storage maps directly to the LPBA of said each block of the metadata in the on-premises storage.

21. The computer system of claim 15 wherein staging the metadata for the snapshot in the on-premises storage comprises:

allocating blocks for the metadata randomly from a local physical block address space on the on-premises storage.