Object format and upload process for archiving data in cloud/object storage

• US 10,503,444 B2
• Filed: 01/12/2018
• Issued: 12/10/2019
• Est. Priority Date: 01/12/2018
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Kashi Visvanathan, Satish Kumar, Sarda, Pooja, Langouev, Ilya, Kandambakkam, Arun
• Primary Examiner(s): Doan, Khoa D
• Application Number:
  US15/870,711
• Publication Number:
  US 20190220198A1
• Time in Patent Office: 697 Days
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Techniques for archiving a data set in cloud/object storage are provided. In one set of embodiments, a computer system can receive an incremental snapshot of the data set and package the incremental snapshot into one or more data objects and one or more metadata objects. The one or more data objects can include data blocks of the incremental snapshot, and the one or more metadata objects can include a metadata entry for each data block, the metadata entry comprising an identifier of the incremental snapshot, an identifier of the data block, and an identifier of a data object in which the data block is stored. The computer system can then write the one or more data objects and the one or more metadata objects to the cloud/object storage.

Claims

1. A method for archiving a data set in cloud storage, the method comprising:

receiving, by a computer system, an incremental snapshot of the data set;

packaging, by the computer system, the incremental snapshot into one or more data objects and one or more metadata objects, wherein the one or more data objects include data blocks of the incremental snapshot, and wherein the one or more metadata objects include a metadata entry for each data block, the metadata entry comprising an identifier of the incremental snapshot, an identifier of the data block, and an identifier of a data object in which the data block is stored; and

writing, by the computer system, the one or more data objects and the one or more metadata objects to the cloud storage,

wherein the packaging and writing comprises, as each data block of the incremental snapshot is received by the computer system;

adding the data block to a fixed-size data memory buffer;

adding a metadata entry for the data block to a metadata memory buffer;

determining whether the data memory buffer is full upon adding the data block; and

if the data memory buffer is determined to be full, writing contents of the data memory buffer to the cloud storage as a new data object, and

wherein once all data blocks of the incremental snapshot are processed via the data memory buffer, the computer system writes contents of the metadata memory buffer to the cloud storage as a new metadata object.

2. The method of claim 1 wherein the one or more data objects and the one or more metadata objects are new objects that do not exist on the cloud storage prior to the writing.

3. The method of claim 1 wherein if the data memory buffer is not determined to be full, the computer system processes additional data blocks in the incremental snapshot until the data memory buffer becomes full.

4. The method of claim 1 further comprising repeating the packaging and writing for additional incremental snapshots of the data set that are received by the computer system.

5. The method of claim 1 wherein each of the one or more metadata objects include metadata entries for data blocks corresponding to a fixed sub-range of an address space of the data set.

6. The method of claim 1 wherein the fixed size of the data memory buffer is based on a predefined size for the one or more data objects.

7. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code embodying a method for archiving a data set in cloud storage, the method comprising:

receiving an incremental snapshot of the data set;

packaging the incremental snapshot into one or more data objects and one or more metadata objects, wherein the one or more data objects include data blocks of the incremental snapshot, and wherein the one or more metadata objects include a metadata entry for each data block, the metadata entry comprising an identifier of the incremental snapshot, an identifier of the data block, and an identifier of a data object in which the data block is stored; and

writing the one or more data objects and the one or more metadata objects to the cloud storage,

wherein the packaging and writing comprises, as each data block of the incremental snapshot is received by the computer system;

adding the data block to a fixed-size data memory buffer;

adding a metadata entry for the data block to a metadata memory buffer;

determining whether the data memory buffer is full upon adding the data block; and

if the data memory buffer is determined to be full, writing contents of the data memory buffer to the cloud storage as a new data object, and

wherein once all data blocks of the incremental snapshot are processed via the data memory buffer, the computer system writes contents of the metadata memory buffer to the cloud storage as a new metadata object.

8. The non-transitory computer readable storage medium of claim 7 wherein the one or more data objects and the one or more metadata objects are new objects that do not exist on the cloud storage prior to the writing.

9. The non-transitory computer readable storage medium of claim 7 wherein if the data memory buffer is not determined to be full, the computer system processes additional data blocks in the incremental snapshot until the data memory buffer becomes full.

10. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises repeating the packaging and writing for additional incremental snapshots of the data set that are received by the computer system.

11. The non-transitory computer readable storage medium of claim 7 wherein each of the one or more metadata objects include metadata entries for data blocks corresponding to a fixed sub-range of an address space of the data set.

12. The non-transitory computer readable storage medium of claim 7 wherein the fixed size of the data memory buffer is based on a predefined size for the one or more data objects.

13. A computer system comprising:

a processor; and

a non-transitory computer readable medium having stored thereon program code for archiving a data set in cloud storage, the program code causing the processor to;

receive an incremental snapshot of the data set;

package the incremental snapshot into one or more data objects and one or more metadata objects, wherein the one or more data objects include data blocks of the incremental snapshot, and wherein the one or more metadata objects include a metadata entry for each data block, the metadata entry comprising an identifier of the incremental snapshot, an identifier of the data block, and an identifier of a data object in which the data block is stored; and

write the one or more data objects and the one or more metadata objects to the cloud storage,

wherein the program code that causes the processor to package incremental snapshot and write the one or more data objects comprises program code that causes the processor to, as each data block of the incremental snapshot is received;

add the data block to a fixed-size data memory buffer;

add a metadata entry for the data block to a metadata memory buffer;

determine whether the data memory buffer is full upon adding the data block; and

if the data memory buffer is determined to be full, write contents of the data memory buffer to the cloud storage as a new data object, and

wherein once all data blocks of the incremental snapshot are processed via the data memory buffer, the processor writes contents of the metadata memory buffer to the cloud storage as a new metadata object.

14. The computer system of claim 13 wherein the one or more data objects and the one or more metadata objects are new objects that do not exist on the cloud storage prior to the writing.

15. The computer system of claim 13 wherein if the data memory buffer is not determined to be full, the processor processes additional data blocks in the incremental snapshot until the data memory buffer becomes full.

16. The computer system of claim 13 wherein the program code further causes the processor to repeat the packaging and writing for additional incremental snapshots of the data set that are received.

17. The computer system of claim 13 wherein each of the one or more metadata objects include metadata entries for data blocks corresponding to a fixed sub-range of an address space of the data set.

18. The computer system of claim 13 wherein the fixed size of the data memory buffer is based on a predefined size for the one or more data objects.