Mutable data objects content verification tool

• US 10,146,600 B2
• Filed: 06/27/2016
• Issued: 12/04/2018
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Kurilov, Andrey, Danilov, Mikhail, Rakulenko, Alexander, Gusakov, Kirill, Fomin, Andrey
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US15/193,144
• Publication Number:
  US 20170177249A1
• Time in Patent Office: 890 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers

Abstract

Data object content verification systems and processes provide perfect reliability and low storage overhead. Object data is generated in a reproducible manner based upon object locally stored object metadata. The object data is stored to an object storage system. The stored object data is subsequently verified by retrieving the object metadata, reproducing the original object data, and comparing the stored and original object data. The data object content verification systems and processes support both mutable and immutable data object content verification.

Claims

1. A method for use with a distributed object storage system, the method comprising:

generating, in a processor, a plurality of object IDs and corresponding original object metadata;

for each object ID, generating corresponding original object data in a reproducible manner based upon the corresponding original object metadata, the original object data generated based upon a first object size;

storing the corresponding original object metadata to a metadata storage device and writing the corresponding original object data to the distributed object storage system;

selecting a plurality of the object IDs of the original object data to be mutated; and

for the object IDs to be mutated;

generating corresponding mutated object data in a reproducible manner based upon corresponding mutated object metadata, the mutated object data generated based upon a second object size that is greater than the first object size;

storing the corresponding mutated object metadata to the metadata storage device and writing the mutated object data to the distributed object storage system;

retrieving the object ID and mutated object metadata from the metadata storage device;

reproducing the mutated object data using the mutated object metadata; and

comparing the reproduced mutated object data to the retrieved mutated object metadata to identify corruption in the distributed object storage system.

2. The method of claim 1 wherein mutated object metadata comprises an updated object ID and a bitmask.

3. The method of claim 2 wherein reproducing the mutated object data using the mutated object metadata comprises:

reproducing original object data using the original object metadata;

generating updated object data using the updated object ID; and

combining the reproduced original data and the updated object data according to the bitmask to reproduce the mutated object data.

4. The method of claim 2 wherein mutated object metadata comprises an ordered list of updated object ID and bitmask tuples.

5. The method of claim 4 wherein reproducing the mutated object data using the mutated object metadata comprises:

reproducing original object data using the original object metadata;

for each tuple in the ordered list of updated object ID and bitmask tuples, iteratively generating updated object data using the updated object ID and combining the reproduced original data and the updated object data according to the bitmask to generate the reproduce the mutated object data.

6. A system for use with a distributed object storage system, the system comprising:

a content generator configured to generate object data in a reproducible manner based upon corresponding object metadata, the original object data generated based upon a first object size;

an object creator configured to;

generate a plurality of object IDs and corresponding original object metadata;

for each object ID, generate corresponding original object data using the content generator and the corresponding original object metadata;

store the corresponding original object metadata to a metadata storage device; and

write the corresponding original object data to the distributed object storage system;

an object mutator configured to;

select a plurality of the object IDs of the original object data to be mutated; and

for the object IDs to be mutated;

generate corresponding mutated object data in a reproducible manner using the content generator and the mutated object metadata, the mutated object data generated based upon a second object size that is greater than the first object size;

store the corresponding mutated object metadata to the metadata storage device; and

write the mutated object data to the distributed object storage system; and

a content verifier configure to;

for the object IDs to be mutated;

retrieve the object ID and mutated object metadata from the metadata storage device;

reproduce the mutated object data using the mutated object metadata; and

compare the reproduced mutated object data to the retrieved mutated object metadata to identify corruption in the distributed object storage system.

7. The system of claim 6 wherein mutated object metadata comprises an updated object ID and a bitmask.

8. The system of claim 7 wherein the content verifier is configured to:

reproduce original object data using the original object metadata;

generate updated object data using the updated object ID; and

combine the reproduced original data and the updated object data according to the bitmask to reproduce the mutated object data.

9. The system of claim 7 wherein mutated object metadata comprises an ordered list of updated object ID and bitmask tuples.

10. The system of claim 9 wherein the content verifier is configured to:

reproduce original object data using the original object metadata;

for each tuple in the ordered list of updated object ID and bitmask tuples, iteratively generate updated object data using the updated object ID and combine the reproduced original data and the updated object data according to the bitmask to generate the reproduce the mutated object data.