Object storage in cloud with reference counting using versions

• US 10,652,330 B2
• Filed: 01/15/2017
• Issued: 05/12/2020
• Est. Priority Date: 01/15/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Frank, Shahar, Mosek, Amir
• Primary Examiner(s): Recek, Jason D
• Application Number:
  US15/406,724
• Publication Number:
  US 20180205791A1
• Time in Patent Office: 1,213 Days
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A data storage apparatus includes an interface and one or more processors. The interface is configured for communicating with a cloud-based object storage system having a built-in versioning mechanism that assigns version numbers to objects stored therein. The one or more processors are configured to receive data for storage from one or more workloads, to store the data as objects in the cloud-based object storage system, and to update and record reference counts for at least some of the objects, by forcing the built-in versioning mechanism of the cloud-based object storage system to update the version numbers so as to match the reference counts.

Claims

1. A data storage apparatus, comprising:

an interface, for communicating with a cloud-based object storage system having a built-in versioning mechanism that assigns version numbers to objects stored therein, each object stored in the cloud-based object storage system comprising a corresponding value and corresponding object metadata; and

one or more processors, configured to;

receive data for storage from one or more workloads;

to calculate a hash value over an object of the received data;

to identify whether an object corresponding to the calculated hash value is stored in the object storage system;

to send the object of the received data to the cloud-based object storage system for storage, in cases in which a corresponding object was not identified in the cloud-based object storage system; and

in cases in which a corresponding object was identified in the cloud-based object storage system, to send to the cloud-based object storage system an update metadata command related to the corresponding object, the update metadata command when received by the cloud-based object storage system forcing the built-in versioning mechanism of the cloud-based object storage system to;

update the corresponding object metadata of the corresponding object identified in the cloud-based object storage system; and

issue a new version number for the corresponding updated object metadata, wherein the new version number serves as a reference count for the corresponding object identified in the cloud-based storage system, the corresponding object identified in the cloud-based storage system having content identical to the object of the received data.

2. The apparatus according to claim 1, wherein the one or more processors are configured to store the objects in the cloud-based object storage system with hash values of the stored objects serving as keys.

3. The apparatus according to claim 1, wherein the update metadata command forces the built-in versioning mechanism to update the corresponding object metadata of the corresponding object but not the corresponding value of the corresponding object.

4. The apparatus according to claim 1, wherein the one or more processors are further configured to send an instruction to the cloud-based object storage system to update the corresponding value of the corresponding object.

5. The apparatus according to claim 1, wherein, for a given object, the one or more processors are configured to force the built-in versioning mechanism to update a version number of the given object by sending to the cloud-based object storage system an instruction to update a dummy object associated with the given object.

6. The apparatus according to claim 1, wherein the one or more processors do not store reference counts for the objects external to the cloud-based object storage system.

7. The apparatus according to claim 1, wherein the one or more processors are configured to force the built-in versioning mechanism to update a version number of an object by an instruction that does not update any metadata parameters.

8. The apparatus according to claim 1, wherein the one or more processors are configured, in response to an instruction to delete a file, to force the cloud-based object storage system to decrement the version number of an object corresponding to the file.

9. The apparatus according to claim 1, wherein the one or more processors are configured, in response to an instruction to delete a file, to check the version number of an object corresponding to the file and if the version number is 1, to delete the corresponding object.

10. The apparatus according to claim 1, wherein the one or more processors are configured to carry out a background “

garbage collection”

process that uses the version numbers in the cloud-based object storage system to identify unreferenced objects, and deletes the unreferenced objects.

11. The apparatus according to claim 1, wherein the one or more processors are configured to force the built-in versioning mechanism of the cloud-based object storage system to update the version numbers of some objects by providing an instruction to update an associated dummy object, and to update the version numbers of other objects by providing an instruction to update the object itself or the object metadata of the object.

12. A method for data storage, comprising:

receiving data for storage from one or more workloads;

calculating a hash value over an object of the received data;

identifying whether a cloud-based object storage system, having a built-in versioning mechanism that assigns version numbers to the objects stored therein, stores an object corresponding to the calculated hash value, each object stored in the cloud-based object storage system comprising a corresponding value and corresponding object metadata;

sending the object of the received data to the cloud-based object storage system for storage, when a corresponding object was not identified in the cloud-based object storage system; and

when a corresponding object was identified in the cloud-based object storage system, sending to the cloud-based object storage system an update metadata command related to the corresponding object, the update metadata command when received by the cloud-based object storage system forcing the built-in versioning mechanism of the cloud-based object storage system to;

update the corresponding object metadata of the corresponding object identified in the cloud-based object storage system; and

issue a new version number for the corresponding updated object metadata, wherein the new version number serves as a reference count for the corresponding object identified in the cloud-based storage system, the corresponding object identified in the cloud-based storage system having content identical to the object of the received data.

13. The method according to claim 12, wherein the cloud-based object storage system stores objects with hash values of the stored objects serving as keys.

14. The method according to claim 12, wherein the update metadata command forces the built-in versioning mechanism to update the corresponding object metadata of the corresponding object but not the corresponding value of the corresponding object.

15. The method according to claim 12, further comprising sending an instruction to the cloud-based object storage system to update the corresponding value of the corresponding object.

16. The method according to claim 12, further comprising, for a given object, sending the cloud-based object storage system an instruction to update a dummy object associated with the given object.

17. The method according to claim 16, wherein the dummy object associated with the given object has a size of zero.

18. The method according to claim 16, wherein the dummy object associated with the given object has a size which is a minimal size supported by the cloud-based object storage system.

19. The method according to claim 12, wherein identifying whether the cloud-based object storage system stores an object corresponding to the calculated hash value comprises instructing the cloud-based object storage system to update an object corresponding to the calculated hash value and determining whether the update succeeded or failed.

20. The method according to claim 19, wherein instructing the cloud-based object storage system to update an object corresponding to the calculated hash value comprises instructing the cloud-based object storage system to update a dummy object corresponding to the calculated hash value.

21. The method according to claim 19, wherein instructing the cloud-based object storage system to update an object corresponding to the calculated hash value comprises instructing the cloud-based object storage system to update the object metadata of an object corresponding to the calculated hash value.