REPRESENTING DE-DUPLICATED FILE DATA

• US 20140279958A1
• Filed: 05/28/2014
• Published: 09/18/2014
• Est. Priority Date: 06/25/2010
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Huang, Mark
• Granted Patent: US 9,128,950 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641

Abstract

Providing a subset of de-duplicated as output is disclosed. In some embodiments, the output comprises a subset of data stored in de-duplicated form in a plurality of containers each comprising a plurality of data segments comprising the data. For each container that includes one or more data segments comprising the subset, a corresponding container data is included in the output. Each container may include one or more segments not included in the subset. For each container the corresponding container data of which is included in the output, a corresponding value in a data structure comprising for each container stored on the de-duplicated storage system a data value indicating whether or not the corresponding container data has been included in the output is updated.

Claims

1. A method comprising:

receiving an indication identifying a subset of data to be provided as output, wherein the subset of data includes a first data segment;

identifying a first container that includes the first data segment;

determining whether corresponding container data associated with the first container has already been included in an output stream based at least in part on a data structure comprising for each of at least a subset of containers an indication of whether or not the corresponding container data has been included in the output stream; and

in the event that it is determined, based at least in part on the data structure, that the first container comprises container data that has not already been included in the output stream;

retrieving the corresponding container data associated with the first container and including the corresponding container data in the output stream, wherein the corresponding container data includes the one or more other segments that are not is included in the subset of data to be provided as output; and

updating a corresponding value in the data structure indicating that the container data associated with the first container has been included in the output stream.

2. The method of claim 1, wherein the container data is included in the output stream in a form as it is stored in a de-duplicated storage system.

3. The method of claim 2, wherein container data is stored in the de-duplicated storage system in a compressed and de-duplicated form.

4. The method of claim 1, wherein the data structure comprises a bit mask.

5. The method of claim 4, wherein the bit mask comprises for each of said plurality of containers a corresponding bit.

6. The method of claim 1, further comprising writing the output stream to a tape or other removable storage media.

7. A data storage system, comprising:

a processor configured to;

receive an indication identifying a subset of data to be provided as output, wherein the subset of data includes a first data segment;

identify a first container that includes the first data segment;

determine whether corresponding container data associated with the first container has already been included in an output stream based at least in part on a data structure comprising for each of at least a subset of containers an indication of whether or not the corresponding container data has been included in the output stream; and

in the event that it is determined, based at least in part on the data structure, that the first container comprises container data that has not already been included in the output stream;

retrieve the corresponding container data associated with the first container and include the corresponding container data in the output stream, is wherein the corresponding container data includes the one or more other segments that are not included in the subset of data to be provided as output; and

update a corresponding value in the data structure indicating that the container data associated with the first container has been included in the output stream; and

a memory coupled to the processor and configured to store the data structure.

8. The system of claim 7, wherein the container data is included in the output stream in a form as it is stored in a de-duplicated storage system.

9. The system of claim 8, wherein container data is stored in the de-duplicated storage system in a compressed and de-duplicated form.

10. The system of claim 7, wherein the data structure comprises a bit mask.

11. The system of claim 10, wherein the bit mask comprises for each of said plurality of containers a corresponding bit.

12. The system of claim 7, wherein the processor is further configured to write the output stream to a tape or other removable storage media.

13. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving an indication identifying a subset of data to be provided as output, wherein the subset of data includes a first data segment;

identifying a first container that includes the first data segment;

determining whether corresponding container data associated with the first container has already been included in an output stream based at least in part on a data structure comprising for each of at least a subset of containers an indication of whether or not the corresponding container data has been included in the output stream; and

in the event that it is determined, based at least in part on the data structure, that the first container comprises container data that has not already been included in the output stream;

retrieving the corresponding container data associated with the first container and including the corresponding container data in the output stream, wherein the corresponding container data includes the one or more other segments that are not is included in the subset of data to be provided as output; and

updating a corresponding value in the data structure indicating that the container data associated with the first container has been included in the output stream.

14. The computer program product of claim 13, wherein the container data is included in the output stream in a form as it is stored in a de-duplicated storage system.

15. The computer program product of claim 13, wherein the data structure comprises a bit mask.

16. The computer program product of claim 15, wherein the bit mask comprises for each of said plurality of containers a corresponding bit.

17. The computer program product of claim 13, further comprising computer instructions for writing the output stream to a tape or other removable storage media.