METHOD TO CHECK FILE DATA INTEGRITY AND REPORT INCONSISTENCIES WITH BULK DATA MOVEMENT

• US 20200133720A1
• Filed: 10/24/2018
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Chinthekindi, Ramprasad, Duggal, Abhinav, Srinivasan, Srikanth, Bai, Lan
• Granted Patent: US 10,649,807 B1
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1441 Resetting or repowering
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/152 using file content signatures, e.g. hash values
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2201/83 the solution involving signatures
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence
• CPC Class Codes: G06F 9/4875 with migration policy, e.g. auction, contract negotiation
• CPC Class Codes: H04L 9/0643 Hash functions, e.g. MD5, SHA, HMAC or f9 MAC

Abstract

In an embodiment, a method for validating data integrity of a seeding process is described. The seeding process for migrating data from a source tier to a target tier persists a perfect hash vector (PHV) to a disk when the seeding process is suspended for various reasons. The PHV includes bits for fingerprints for data segments corresponding to the data, and can be reloaded into memory upon resumption of the seeding process. One or more bits corresponding to fingerprints for copied data segments are reset prior to starting the copy phase in the resumed run. A checksum of the PHV is calculated after the seeding process completes copying data segments in the containers. A non-zero checksum of the PHV indicates that one or more data segments are missing on the source tier or the data segments are not successfully copied to the target tier. The missing data segments and/or one or more related files are reported to a user via a user interface.

Claims

1. A computer-implemented method for detecting and reporting data inconsistency in a storage system, the method comprising:

resuming, by a data seeding module, a suspended process for migrating data from a source tier to a target tier, wherein the data corresponds to a plurality of data segments in a plurality of containers;

loading, by the data seeding module, a vector in memory, wherein the vector represents data segments in the plurality of containers;

resetting, in the vector, a bit corresponding to a fingerprint for each data segment that has been copied to the target tier;

determining that a checksum of the vector is non-zero after the data seeding module completes copying data segments in plurality of containers; and

repeating the loading, the copying and the determining operations in response to determining that the checksum is non-zero.

2. The method of claim 1, wherein the vector is a perfect hash vector (PHV), which includes a fingerprint for each segment associated with the data to be migrated.

3. The method of claim 2, wherein the non-zero checksum of the PHV indicates that a data segment that is missing on the source tier is populated in the vector, or that the data segment that is present on the source tier and populated in the vector is not copied to the target tier.

4. The method of claim 3, further comprising:

reporting the data segment that is missing either on the source tier or the target tier to a user via a user interface.

5. The method of claim 4, wherein a file associated with the missing segment is identified and reported to the user.

6. The method of claim 1, wherein the source tier is an active tier and the target tier is a cloud tier.

7. The method of claim 1, wherein the vector is persisted into a disk when the process for migrating the data is suspended, either independently or as part of a resumption context.

8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations comprising:

resuming, by a data seeding module, a suspended process for migrating data from a source tier to a target tier, wherein the data corresponds to a plurality of data segments in a plurality of containers;

loading, by the data seeding module, a vector in memory, wherein the vector represents data segments in the plurality of containers;

resetting, in the vector, a bit corresponding to a fingerprint for each data segment that has been copied to the target tier;

determining that a checksum of the vector is non-zero after the data seeding module completes copying data segments in plurality of containers; and

repeating the loading, the copying and the determining operations in response to determining that the checksum is non-zero.

9. The non-transitory machine-readable medium of claim 8, wherein the vector is a perfect hash vector (PHV), which includes a fingerprint for each segment associated with the data to be migrated.

10. The non-transitory machine-readable medium of claim 9, wherein the non-zero checksum of the PHV indicates that a data segment that is missing on the source tier is populated in the vector, or that the data segment that is present on the source tier and populated in the vector is not copied to the target tier.

11. The non-transitory machine-readable medium of claim 10, wherein the operations further comprise:

reporting the data segment that is missing either on the source tier or the target tier to a user via a user interface.

12. The non-transitory machine-readable medium of claim 11, wherein a file associated with the missing segment is identified and reported to the user.

13. The non-transitory machine-readable medium of claim 8, wherein the source tier is an active tier and the target tier is a cloud tier.

14. The non-transitory machine-readable medium of claim 8, wherein the vector is persisted into a disk when the process for migrating the data is suspended, either independently or as part of a resumption context.

15. A data processing system, comprising:

a processor;

a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations comprising;

resuming, by a data seeding module, a suspended process for migrating data from a source tier to a target tier, wherein the data corresponds to a plurality of data segments in a plurality of containers;

loading, by the data seeding module, a vector in memory, wherein the vector represents data segments in the plurality of containers;

resetting, in the vector, a bit corresponding to a fingerprint for each data segment that has been copied to the target tier;

determining that a checksum of the vector is non-zero after the data seeding module completes copying data segments in plurality of containers; and

repeating the loading, the copying and the determining operations in response to determining that the checksum is non-zero.

16. The system of claim 15, wherein the vector is a perfect hash vector (PHV), which includes a fingerprint for each segment associated with the data to be migrated.

17. The system of claim 16, wherein the non-zero checksum of the PHV indicates that a data segment that is missing on the source tier is populated in the vector, or that the data segment that is present on the source tier and populated in the vector is not copied to the target tier.

18. The system of claim 17, wherein the operations further comprise:

reporting the data segment that is missing either on the source tier or the target tier to a user via a user interface.

19. The system of claim 18, wherein a file associated with the missing segment is identified and reported to the user.

20. The system of claim 15, wherein the vector is persisted into a disk when the process for migrating the data is suspended, either independently or as part of a resumption context.