Systems and methods for managing data in cloud storage using deduplication techniques

• US 8,650,159 B1
• Filed: 08/26/2010
• Issued: 02/11/2014
• Est. Priority Date: 08/26/2010
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Zhang, Xianbo, Wu, Weibao
• Primary Examiner(s): Fleurantin, Jean B
• Assistant Examiner(s): Ly, Anh
• Application Number:
  US12/868,869
• Time in Patent Office: 1,265 Days
• Field of Search: 707/664, 707/696, 707/674, 707/E17.007, 707/E17.005, 711/162, 711/163, 711/114, 711/161
• US Class Current: 707/664
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A computer-implemented method for managing data in cloud storage using deduplication techniques is described. At least one unique segment of data is received from at least one backup source. The at least one unique segment of data is written to at least one container file associated with the at least one back up source. An index file is created for the at least one container file. The index file includes information regarding the unique segments of data. A plurality of the unique segments of data is transmitted from the at least one container file as an object to a virtual container on cloud storage. A virtual container is composed of an index file and one or more container files, and the index file and each container file are stored as individual cloud storage objects. Spatial locality of the plurality of unique segments of data is maintained in cloud storage.

Claims

1. A computer-implemented method for managing data in cloud storage using deduplication techniques, comprising:

receiving, by a server from a client, a file divided, by the client, into a plurality of segments of data;

identifying unique segments of data and one or more groups of identical segments of data among the plurality of segments of data;

writing, by a processor, a portion of the plurality of segments of data to at least one container file, the at least one container file being associated with a single client, wherein the portion of the plurality of segments of data comprises each unique segment of data and one segment from each group of identical segments of data;

creating, by the processor, an index file for the at least one container file, wherein the index file comprises an association with client, and information regarding the portion of the plurality of segments of data written to the at least one container file; and

transmitting an object from the server to a virtual container on cloud storage, the object comprising the at least one container file, wherein a segment of the unique segments of data is maintained in the object in the same spatial locality as any other segments of the unique segments of data from the same backup source.

2. The method of claim 1, further comprising transmitting the index file as an object to the virtual container on cloud storage, wherein the index file object combined with the at least one container file comprises the virtual container.

3. The method of claim 1, wherein a spatial locality of the portion of the plurality of segments of data written to the at least one container file is maintained in the at least one container file.

4. The method of claim 1, further comprising identifying a virtual container storing at least one requested segment of data on cloud storage.

5. The method of claim 4, further comprising retrieving a first object from the identified virtual container that includes the at least one requested segment of data.

6. The method of claim 5, further comprising writing the contents of the first object to local memory.

7. The method of claim 6, further comprising retrieving at least one additional object from the identified virtual container on cloud storage, wherein the at least one additional object is spatially located near the first object.

8. The method of claim 4, further comprising retrieving a first index file associated with the identified virtual container on cloud storage.

9. The method of claim 8, further comprising writing the first index file to local memory.

10. The method of claim 9, further comprising modifying the first index file in local memory by marking at least one metadata records in the first index file that is associated with the at least one requested segment of data on cloud storage.

11. The method of claim 10, further comprising transmitting a removal command to an object in cloud storage that comprises the at least one requested segment of data once all segments contained in a cloud storage object are marked as deleted.

12. The method of claim 11, further comprising uploading the modified first index file to cloud storage to replace the first index file.

13. A computing device configured to manage data in cloud storage using deduplication techniques, comprising:

a processor;

memory in electronic communication with the processor;

a deduplication module configured to;

receive, by a server from a client, a file divided, by the client, into a plurality of segments of data;

write a portion of the plurality of segments of data to at least one container file, the at least one container file being associated with a single client, wherein the portion of the plurality of segments of data comprises each unique segment of data and one segment from each group of identical segments of data;

create an index file for the at least one container file, wherein the index file comprises an association with the client, and information regarding the portion of the plurality of segments of data written to the at least one container file; and

transmit an object from the server to a virtual container on cloud storage, the object comprising the at least one container file, wherein a segment of the unique segments of data is maintained in the object in the same spatial locality as any other segments of the unique segments of data from the same backup source.

14. The computing device of claim 13, wherein the deduplication module is further configured to transmit the index file as an object to the virtual container on cloud storage, wherein the index file object combined with the at least one container file comprises the virtual container.

15. The computing device of claim 13, wherein a spatial locality of the portion of the plurality of segments of data written to the at least one container file is maintained in the at least one container file.

16. The computing device of claim 13, wherein the deduplication module is further configured to identify a virtual container storing at least one requested segment of data on cloud storage.

17. The computing device of claim 16, wherein the deduplication module is further configured to retrieve a first object from the identified virtual container that includes the at least one requested segment of data.

18. The computing device of claim 17, wherein the deduplication module is further configured to write the contents of the first object to local memory.

19. The computing device of claim 18, wherein the deduplication module is further configured to retrieve at least one additional object from the identified virtual container on cloud storage, wherein the at least one additional object is spatially located near the first object.

20. A computer-program product for managing data in cloud storage using deduplication techniques, the computer-program product comprising a non-transitory computer-readable medium having instructions thereon, the instructions comprising:

code programmed to receive, by a server from a client, a file divided, by the client, into a plurality of segments of data;

code programmed to write a portion of the plurality of segments of data to at least one container file, the at least one container file being associated with a single client, wherein the portion of the plurality of segments of data comprises each unique segment of data and one segment from each group of identical segments of data;

code programmed to create an index file for the at least one container file, wherein the index file comprises an association with the client, and information regarding the portion of the plurality of segments of data written to the at least one container file; and

code programmed to transmit an object from the server to a virtual container on cloud storage, the object comprising the at least one container file, wherein a segment of the unique segments of data is maintained in the object in the same spatial locality as any other segments of the unique segments of data from the same backup source.