Method and system for assuring integrity of deduplicated data

• US 7,814,074 B2
• Filed: 03/14/2008
• Issued: 10/12/2010
• Est. Priority Date: 03/14/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Anglin, Matthew J., Cannon, David M.
• Primary Examiner(s): Mofiz; Apu M
• Assistant Examiner(s): Le; Hung D
• Application Number:
  US12/048,850
• Publication Number:
  US 20090234892A1
• Time in Patent Office: 942 Days
• Field of Search: None
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1756 based on delta files

Abstract

The present invention provides for a system and method for assuring integrity of deduplicated data objects stored within a storage system. A data object is copied to secondary storage media, and a digital signature such as a checksum is generated of the data object. Then, deduplication is performed upon the data object and the data object is split into chunks. The chunks are combined when the data object is subsequently accessed, and a signature is generated for the reassembled data object. The reassembled data object is provided if the newly generated signature is identical to the originally generated signature, and otherwise a backup copy of the data object is provided from secondary storage media.

Claims

1. A method in a computer system for assuring integrity of deduplicated data, comprising:

copying a data object within a data system to a backup storage media;

generating an original object signature of the data object;

storing the original object signature of the data object in an index;

performing deduplication upon the data object, including dividing the data object into a set of one or more data chunks, and for each data chunk;

determining if a previously stored identical copy of the data chunk exists on a primary storage media;

storing the data chunk on the primary storage media in response to determining that a previously stored identical copy of the data chunk does not exist on the primary storage media; and

creating a pointer to the previously stored identical copy of the data chunk on the primary storage media in response to determining that a previously stored identical copy of data chunk exists on the primary storage media;

assembling the deduplicated data object into a reassembled state responsive to said data object being accessed by the computer system, wherein during a restore or a storage audit operation, the set of one more data chunks produced from the deduplicated data object are re-combined into a single data object;

generating a reassembled object signature for the reassembled data object;

comparing the reassembled object signature with the original object signature associated with the data object stored in the index;

providing the reassembled data object if the reassembled object signature matches the original object signature; and

providing the data object stored on the backup storage media if the reassembled object signature of the reassembled data object does not match the original object signature.

2. The method as in claim 1, wherein the original object signature and reassembled object signature are checksums using a hash function selected from the group consisting of cyclic redundancy check, MD5, SHA-1, SHA-256, SHA-512, and Whirlpool.

3. The method as in claim 1, wherein the step of determining if a previously stored data chunk is identical to the data chunk includes calculating a hash function value for the data chunk, and comparing said hash function value to an index of data chunk hash function values.

4. The method as in claim 1, wherein the data object is divided into data chunks using a Rabin fingerprinting technique.

5. The method as in claim 1, wherein the backup storage media is tape.

6. The method as in claim 1, wherein copying the data object to a backup storage media occurs synchronously during ingestion of the data object into the data system.

7. The method as in claim 1, wherein the generation of the original object signature is performed during ingestion of the data object into the data system.

8. The method as in claim 1, wherein copying the data object to a backup storage media occurs asynchronously during an operation subsequent to ingestion of the data object into the data system.

9. The method as in claim 1, wherein generation of the original object signature is performed subsequent to ingestion of the data object into the data system.

10. The method as in claim 1, wherein the data object is marked as damaged within the data system if the reassembled object signature of the reassembled data object does not match the original object signature stored in the index.

11. The method as in claim 1, wherein the data object is retrieved from the backup storage media and stored on the primary storage media if the reassembled object signature of the reassembled data object does not match the original object signature.

12. A computer program product comprising a computer useable medium having a computer readable program for assuring integrity of deduplicated data, wherein the computer readable program when executed on a computer causes the computer to:

copy a data object within a data system to a backup storage media;

generate an original object signature of the data object;

store the original object signature of the data object in an index;

perform deduplication upon the data object, including dividing the data object into a set of one or more data chunks, and for each data chunk;

determining if a previously stored identical copy of the data chunk exists on a primary storage media;

storing the data chunk on the primary storage media in response to determining that a previously stored identical copy of the data chunk does not exist on the primary storage media; and

creating a pointer to the previously stored identical copy of the data chunk on the primary storage media in response to determining that a previously stored identical copy of data chunk exists on the primary storage media;

assemble the deduplicated data object into a reassembled state responsive to said data object being accessed by the computer system, wherein during a restore or a storage audit operation, the set of one more data chunks produced from the deduplicated data object are re-combined into a single data object;

generate a reassembled object signature for the reassembled data object;

compare the reassembled object signature with the original object signature associated with the data object stored in the index;

provide the reassembled data object if the reassembled object signature matches the original object signature; and

provide the data object stored on the backup storage media if the reassembled object signature of the reassembled data object does not match the original object signature.

13. The computer program product as in claim 12, wherein copying the data object to a backup storage media occurs synchronously during ingestion of the data object into the data system.

14. The computer program product as in claim 12, wherein the generation of the original object signature is performed during ingestion of the data object into the data system.

15. The computer program product as in claim 12, wherein copying the data object to a backup storage media occurs asynchronously during an operation subsequent to ingestion of the data object into the data system.

16. The computer program product as in claim 12, wherein generation of the original object signature is performed subsequent to ingestion of the data object into the data system.

17. A system, comprising:

at least one processor; and

at least one memory storing instructions operable with the at least one processor for assuring integrity of deduplicated data, the instructions being executed for;

copying a data object within a data system to a backup storage media;

generating an original object signature of the data object, including dividing the data object into a set of one or more data chunks, and for each data chunk;

determining if a previously stored identical copy of the data chunk exists on a primary storage media;

storing the data chunk on the primary storage media in response to determining that a previously stored identical copy of the data chunk does not exist on the primary storage media; and

creating a pointer to the previously stored identical copy of the data chunk on the primary storage media in response to determining that a previously stored identical copy of data chunk exists on the primary storage media;

storing the original object signature of the data object in an index;

performing deduplication upon the data object;

assembling the deduplicated data object into a reassembled state responsive to said data object being accessed by the computer system, wherein during a restore or a storage audit operation, the set of one more data chunks produced from the deduplicated data object are re-combined into a single data object;

generating a reassembled object signature for the reassembled data object;

comparing the reassembled object signature with the original object signature associated with the data object stored in the index;

providing the reassembled data object if the reassembled object signature matches the original object signature; and

providing the data object stored on the backup storage media if the reassembled object signature of the reassembled data object does not match the original object signature.

18. The system as in claim 17, wherein copying the data object to a backup storage media occurs synchronously during ingestion of the data object into the data system.

19. The system as in claim 17, wherein the generation of the original object signature is performed during ingestion of the data object into the data system.

20. The system as in claim 17, wherein copying the data object to a backup storage media occurs asynchronously during an operation subsequent to ingestion of the data object into the data system.

21. The system as in claim 17, wherein generation of the original object signature is performed subsequent to ingestion of the data object into the data system.

22. A method in a computer system for assuring integrity of deduplicated data, comprising:

copying a data object within a data system to a backup storage media;

generating an original object signature of the data object;

storing the original object signature of the data object in an index;

performing deduplication upon the data object, including dividing the data object into a set of one or more data chunks, and for each data chunk;

determining if a previously stored identical copy of the data chunk exists on a primary storage media;

storing the data chunk on the primary storage media in response to determining that a previously stored identical copy of the data chunk does not exist on the primary storage media; and

creating a pointer to the previously stored identical copy of the data chunk on the primary storage media in response to determining that a previously stored identical copy of the data chunk exists on the primary storage media;

assembling the deduplicated data object into a reassembled state responsive to said data object being accessed by the computer system, wherein during a restore or a storage audit operation, the set of one or more data chunks produced from the deduplicated data object are re-combined into a single data object;

generating a reassembled object signature for the reassembled data object;

comparing the reassembled object signature with the original object signature associated with the data object stored in the index; and

providing the reassembled data object if the reassembled object signature matches the original object signature.

23. The method as in claimed 22, further comprising providing notification if the reassembled object signature dose not match the original object signature.