Method and system for handling object boundaries of a data stream to optimize deduplication

• US 9,087,086 B1
• Filed: 12/18/2012
• Issued: 07/21/2015
• Est. Priority Date: 12/18/2012
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Li, Junxu, Hsu, Windsor W.
• Primary Examiner(s): Nguyen, Phong
• Application Number:
  US13/718,827
• Time in Patent Office: 945 Days
• Field of Search: 707/664, 707/692, 707/693
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1752 based on file chunks

Abstract

Techniques for deduplicating a data stream based on boundary markers embedded therein are described. According to one embodiment, a data stream is received from a client having a sequence of a plurality of data objects, where to data stream represents a file or a directory of one or more files of a file system associated with the client. In response, the data stream is deduplicated into a plurality of deduplicated chunks in view of boundaries of the data objects.

Claims

1. A computer-implemented method for deduplicating data, comprising:

receiving at a storage system over a network from a client a data stream having a sequence of a plurality of data objects, the data stream representing a file or a directory of one or more files of a file system associated with the client, wherein the data stream includes a plurality of boundary markers inserted by the client prior to being received at the storage system;

scanning the data stream to recognize a plurality of boundary markers each being associated with each of the data objects, the boundary markers identifying boundaries of the data objects; and

deduplicating the data stream into a plurality of deduplicated chunks in view of boundaries of the data objects marked by the boundary markers, wherein deduplicating the data stream comprises

anchoring the data stream using a predetermined chunking algorithm to create a plurality of anchor points, each anchor point identifying a chunking boundary for deduplication;

relocating at least one of the anchor points to a location that is identified by at least one boundary markers; and

chunking the data stream into the deduplicated chunks based on the anchor points that include at least one relocated anchor point.

2. The method of claim 1, wherein the data stream is segmented and deduplicated in view of the boundaries of the data objects such that no deduplicated chunk includes content associated with multiple data objects.

3. The method of claim 1, wherein the data stream is segmented and deduplicated in view of the boundaries of the data objects such that a number of deduplicated chunks that include content associated with multiple data objects is reduced.

4. The method of claim 1, further comprising packing the deduplicated chunks that are associated with the same data object in a single compression region within a single storage container for storage.

5. A computer-implemented method for deduplicating data, comprising:

receiving from a client a data stream having a sequence of a plurality of data objects, the data stream representing a file or a directory of one or more files of a file system associated with the client;

scanning the data stream to recognize a plurality of boundary markers each being associated with each of the data objects, the boundary markers identifying boundaries of the data objects; and

deduplicating the data stream into a plurality of deduplicated chunks in view of boundaries of the data objects, wherein deduplicating the data stream comprises

anchoring the data stream using a predetermined chunking algorithm to create a plurality of anchor points, each anchor point identifying a chunking boundary for deduplication,

relocating at least one of the anchor points to a location that is identified by at least one boundary marker, including

determining a distance between a first anchor point and an adjacent data object boundary that is identified by a first boundary marker, and

relocating the first anchor point to the adjacent data object boundary within the data stream if the distance is below a predetermined threshold, and

chunking the data stream into the deduplicated chunks based on the anchor points that include at least one relocated anchor point.

6. The method of claim 5, further comprising adding a second anchor point at the adjacent data object boundary within the data stream if the distance is equal to or above the predetermined threshold.

7. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for deduplicating data, the operations comprising:

receiving at a storage system over a network from a client a data stream having a sequence of a plurality of data objects, the data stream representing a file or a directory of one or more files of a file system associated with the client, wherein the data stream includes a plurality of boundary markers inserted by the client prior to being received at the storage system;

scanning the data stream to recognize a plurality of boundary markers each being associated with each of the data objects, the boundary markers identifying boundaries of the data objects; and

deduplicating the data stream into a plurality of deduplicated chunks in view of boundaries of the data objects marked by the boundary markers, wherein deduplicating the data stream comprises

anchoring the data stream using a predetermined chunking algorithm to create a plurality of anchor points, each anchor point identifying a chunking boundary for deduplication;

relocating at least one of the anchor points to a location that is identified by at least one boundary markers; and

chunking the data stream into the deduplicated chunks based on the anchor points that include at least one relocated anchor point.

8. The non-transitory machine-readable medium of claim 7, wherein the data stream is segmented and deduplicated in view of the boundaries of the data objects such that no deduplicated chunk includes content associated with multiple data objects.

9. The non-transitory machine-readable medium of claim 7, wherein the data stream is segmented and deduplicated in view of the boundaries of the data objects such that a number of deduplicated chunks that include content associated with multiple data objects is reduced.

10. The non-transitory machine-readable medium of claim 7, wherein the operations further comprise packing the deduplicated chunks that are associated with the same data object in a single compression region within a single storage container for storage.

11. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for deduplicating data, the operations comprising:

receiving from a client a data stream having a sequence of a plurality of data objects, the data stream representing a file or a directory of one or more files of a file system associated with the client;

scanning the data stream to recognize a plurality of boundary markers each being associated with each of the data objects, the boundary markers identifying boundaries of the data objects; and

deduplicating the data stream into a plurality of deduplicated chunks in view of boundaries of the data objects, wherein deduplicating the data stream comprises

anchoring the data stream using a predetermined chunking algorithm to create a plurality of anchor points, each anchor point identifying a chunking boundary for deduplication,

relocating at least one of the anchor points to a location that is identified by at least one boundary marker, including

determining a distance between a first anchor point and an adjacent data object boundary that is identified by a first boundary marker, and

relocating the first anchor point to the adjacent data object boundary within the data stream if the distance is below a predetermined threshold, and

chunking the data stream into the deduplicated chunks based on the anchor points that include at least one relocated anchor point.

12. The non-transitory machine-readable medium of claim 11, wherein the operations further comprise adding a second anchor point at the adjacent data object boundary within the data stream if the distance is equal to or above the predetermined threshold.

13. A data processing system, comprising:

a processor; and

a memory storing instructions, which when executed from the memory, cause the processor to

receive over a network from a client a data stream having a sequence of a plurality of data objects, the data stream representing a file or a directory of one or more files of a file system associated with the client, wherein the data stream includes a plurality of boundary markers inserted by the client prior to being received at the data processing system,

scan the data stream to recognize a plurality of boundary markers each being associated with each of the data objects, the boundary markers identifying boundaries of the data objects, and

deduplicate the data stream into a plurality of deduplicated chunks in view of boundaries of the data objects marked by the boundary markers, wherein deduplicating the data stream comprises

anchoring the data stream using a predetermined chunking algorithm to create a plurality of anchor points, each anchor point identifying a chunking boundary for deduplication;

relocating at least one of the anchor points to a location that is identified by at least one boundary markers; and

chunking the data stream into the deduplicated chunks based on the anchor points that include at least one relocated anchor point.

14. The system of claim 13, wherein the data stream is segmented and deduplicated in view of the boundaries of the data objects such that no deduplicated chunk includes content associated with multiple data objects.

15. The system of claim 13, wherein the data stream is segmented and deduplicated in view of the boundaries of the data objects such that a number of deduplicated chunks that include content associated with multiple data objects is reduced.

16. A data processing system, comprising:

a processor; and

a memory storing instructions, which when executed from the memory, cause the processor to

receive from a client a data stream having a sequence of a plurality of data objects, the data stream representing a file or a directory of one or more files of a file system associated with the client;

scan the data stream to recognize a plurality of boundary markers each being associated with each of the data objects, the boundary markers identifying boundaries of the data objects; and

deduplicate the data stream into a plurality of deduplicated chunks in view of boundaries of the data objects, wherein deduplicating the data stream comprises

anchoring the data stream using a predetermined chunking algorithm to create a plurality of anchor points, each anchor point identifying a chunking boundary for deduplication,

relocating at least one of the anchor points to a location that is identified by at least one boundary marker, including

determining a distance between a first anchor point and an adjacent data object boundary that is identified by a first boundary marker, and

relocating the first anchor point to the adjacent data object boundary within the data stream if the distance is below a predetermined threshold, and

chunking the data stream into the deduplicated chunks based on the anchor points that include at least one relocated anchor point.

17. The system of claim 16, wherein a second anchor point is added at the adjacent data object boundary within the data stream if the distance is equal to or above the predetermined threshold.

18. The system of claim 17, wherein the deduplicated chunks that are associated with the same data object are packed in a single compression region within a single storage container for storage.