Secure online distributed data storage services

• US 9,483,657 B2
• Filed: 01/10/2014
• Issued: 11/01/2016
• Est. Priority Date: 01/14/2013
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Paul, Sanjoy, Sengupta, Shubhashis, Mohamedrasheed, Annervaz Karukapadath, Saxena, Amitabh, Kaulgud, Vikrant
• Primary Examiner(s): Zand, Kambiz
• Assistant Examiner(s): Zoubair, Noura
• Application Number:
  US14/152,452
• Publication Number:
  US 20140201541A1
• Time in Patent Office: 1,026 Days
• Field of Search: 713/168, 713/189, 713/193, 713/194, 714/15, 714/758, 714/821
• US Class Current: 1/1
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes

Abstract

The data vaporizer provides secure online distributed data storage services that securely store and retrieve data in a public distributed storage substrate such as public cloud. The data vaporizer vaporizes (e.g., fragmented into tiny chunks of configurable sizes) data and distributes the fragments to multiple storage nodes so that the data is not vulnerable to local disk failures, secures data so that even if some of the storage nodes are compromised, the data is undecipherable to the attacker, stores data across multiple cloud storage providers and/or parties using keys (e.g., tokens) provided by multiple parties (including the owners of the data) and maintains data confidentiality and integrity even where one or more data storage provider is compromised. The data vaporizer is configurable for different domain requirements including data privacy and anonymization requirements, encryption mechanisms, regulatory compliance of storage locations, and backup and recovery constraints.

Claims

1. A system for the distributed storage of electronic data, the system comprising:

a communications interface and a non-transitory memory coupled with a processor;

the communications interface configured to receive input data blocks and user storage constraints from a user;

the memory having stored thereon;

one or more input data blocks comprising data fields;

the user storage constraints; and

processor executable instructions that when executed by the processor comprise;

a shuffler configured to;

anonymize the data fields of each of the one or more input data blocks;

an encryptor configured to;

generate a cipher key and encrypt, using the cipher key, the anonymized data fields based on the user storage constraints to obtain a ciphertext file comprising ciphertext data blocks;

an erasure coder configured to;

generate coded chunks from the ciphertext data blocks and erasure coding details;

a distributor configured to;

distribute the coded chunks to a first set of two or more storage providers based on the user storage constraints; and

generate distribution details for the distributed coded chunks;

a metadata generator configured to;

generate secret shares of metadata for each of the coded chunks, wherein the secret shares of metadata includes decoding information for decoding each respective coded chunk including at least the cipher key;

a secret sharer configured to;

generate secret share keys for decoding the secret shares of metadata;

generate a secret sharing ratio X/Y that indicates a number of X storage providers out of Y storage providers used to reconstruct the secret shares of metadata; and

distribute fragments of the secret shares of metadata across a second set of two or more storage providers, wherein the fragments of the secret shares of metadata are distributed to the second set of two or more storage providers separately from corresponding coded chunks; and

a secret sharer distributor configured to;

distribute the secret share keys across a third set of two or more storage providers, wherein the secret share keys are distributed to the third set of two or more storage providers separately from corresponding coded chunks.

2. The system of claim 1, wherein the encryptor is further configured to:

generate an encryption key for the ciphertext file, wherein the encryption key comprises;

a shuffle key, the cipher key, the erasure coding details, and the distribution details;

wherein the communications interface communicates the encryption key to the user.

3. The system of claim 2, wherein the distributed coded chunks and the secret shares of metadata stored together at each of the storage providers are identified as storage shares.

4. The system of claim 3, wherein the user storage constraints comprise:

a customer type that identifies an industry of the user and a type of data from which the input data blocks are comprised;

a domain compliance for the industry of the user that identifies a type of data compliance, the type of data compliance comprising HIPAA, financial, or multi-media;

a fault tolerance threshold;

a security type, or a security threshold, or both;

a retrievability threshold that identifies a probability of retrieving stored data blocks;

a repairability threshold that identifies a probability of repairing stored data blocks;

a simultaneous provider attacks threshold that identifies a subset of the number of the storage providers required to aggregate the storage shares to re-generate the encryption key communicated to the user;

or

budget for storage and retrieval of the input data blocks;

or any combination thereof.

5. The system of claim 3, wherein the executable instructions further comprise:

a protector;

a retriever; and

a decoder.

6. The system of claim 5, wherein the protector is configured to:

interrogate the storage shares by executing a corrupt or modified block check to test the retrievability of the distributed coded chunks and secret shares of metadata of the storage shares;

mark the distributed coded chunks and secret shares of metadata that fail the corrupt or modified block check; and

communicate the distributed coded chunks and secret shares of metadata that fail the corrupt or modified block check to the retriever.

7. The system of claim 6, wherein the retriever is configured to:

retrieve a number of uncorrupted data shares from the storage providers;

re-generate the encryption key from the secret shares of metadata of the uncorrupted data shares;

retrieve, using the re-generated encryption key, uncorrupted distributed coded chunks from the storage providers;

communicate the retrieved uncorrupted distributed coded chunks to the decoder.

8. The system of claim 7, wherein the decoder is configured to:

decode the retrieved uncorrupted distributed coded chunks to obtain the data fields of the input data blocks for the corrupted distributed coded chunks;

re-code the data fields of the input data blocks for the corrupted distributed coded chunks;

communicate re-coded chunks to the distributor to distribute the re-coded chunks to the storage providers based on the user storage constraints; and

generate distribution details for the distributed re-coded chunks.

9. A method for the distributed storage of electronic data comprising:

receiving into a non-transitory memory coupled with a processor, using a communications interface;

one or more input data blocks comprising data fields;

user storage constraints;

anonymizing the data fields of each of the one or more input data blocks;

generating a cipher key;

encrypting, using the cipher key, the anonymized data fields based on the user storage constraints to obtain a ciphertext file comprising ciphertext data blocks;

generating coded chunks from the ciphertext data blocks and erasure coding details;

distributing the coded chunks to a first set of two or more storage providers based on the user storage constraints;

generating distribution details for the distributed coded chunks;

generating, by a metadata generator, secret shares of metadata for each of the coded chunks, wherein the secret shares of metadata includes decoding information for decoding each respective coded chunk including at least the cipher key;

generating, by a secret sharer, secret share keys for the secret shares of metadata;

generating, by the secret sharer, a secret sharing ratio X/Y that indicates a number of X storage providers out of Y storage providers used to reconstruct the secret shares of metadata;

distributing, by the secret sharer, fragments of the secret shares of metadata across a second set of two or more storage providers, wherein the fragments of the secret shares of metadata are distributed to the second set of two or more storage providers separately from corresponding coded chunks; and

distributing the secret share keys across the two or more storage providers, wherein the secret share keys are distributed to the third set of two or more storage providers separately from corresponding coded chunks.

10. The method of claim 9, further comprising:

generating an encryption key for the ciphertext file from a shuffle key, the cipher key, the erasure coding details, and the distribution details;

communicating, the communications interface, the encryption key to the user.

11. The method of claim 10, further comprising:

distributing the secret shares of metadata to the storage providers, where the distributed coded chunks and the secret shares of metadata stored together at each of the storage providers are identified as storage shares.

12. The method of claim 11, wherein the user storage constraints comprise:

a customer type that identifies an industry of the user and a type of data from which the input data blocks are comprised;

a domain compliance for the industry of the user that identifies a type of data compliance, the type of data compliance comprising HIPAA, financial, or multi-media;

a fault tolerance threshold;

a security type, or a security threshold, or both;

a retrievability threshold that identifies a probability of retrieving stored data blocks;

a repairability threshold that identifies a probability of repairing stored data blocks;

simultaneous provider attacks threshold that identifies a subset of the number of the providers required to aggregate the storage shares to re-generate the encryption key communicated to the user; and

budget for storage and retrieval of the input data blocks.

13. The method of claim 11, further comprising:

interrogating the storage shares by executing a corrupt or modified block check to test the retrievability of the distributed coded chunks and secret shares of the storage shares;

marking the distributed coded chunks and secret shares that fail the corrupt or modified block check.

14. The method of claim 13, further comprising:

retrieving a number of uncorrupted data shares from the storage providers;

re-generating the encryption key from the secret shares of the uncorrupted data shares;

retrieving, using the re-generated encryption key, uncorrupted distributed coded chunks from the storage providers.

15. The method of claim 14, further comprising:

decoding the retrieved uncorrupted distributed coded chunks to obtain the data fields of the input data blocks for the corrupted distributed coded chunks;

re-coding the data fields of the input data blocks for the corrupted distributed coded chunks;

distributing re-coded chunks to the storage providers based on the user storage constraints; and

generating distribution details for the distributed re-coded chunks;

generating a new encryption key for the ciphertext file from the shuffle key, the cipher, the erasure coding details, and the distribution details for the distributed re-coded chunks;

generating a new set of secret shares and new secret share keys from the new encryption key;

distributing the secret shares to the storage providers; and

communicating the new encryption key to the user.

16. A product comprising:

a non-transitory computer readable medium with processor executable instructions stored thereon, wherein the instructions when executed by the processor cause the processor to;

receive, using a communications interface;

one or more input data blocks comprising data fields;

user storage constraints;

anonymize the data fields of each the one or more input data blocks;

generate a cipher key and encrypt, using the cipher key, the anonymized data fields based on the user storage constraints to obtain a ciphertext file comprising ciphertext data blocks;

generate coded chunks from the ciphertext data blocks and erasure coding details;

distribute the coded chunks to a first set of two or more storage providers based on the user storage constraints; and

generate distribution details for the distributed coded chunks;

generate secret shares of metadata for each of the coded chunks, wherein the secret shares of metadata includes decoding information for decoding each respective coded chunk including at least the cipher key;

generate, by a secret sharer, secret share keys for decoding the secret shares of metadata;

generate a secret sharing ratio X/Y that indicates a number of X storage providers out of Y storage providers used to reconstruct the secret shares of metadata;

distribute fragments of the secret shares of metadata across a second set of two or more storage providers, wherein the fragments of the secret shares of metadata are distributed to the second set of two or more storage providers separately from corresponding coded chunks; and

distribute the secret share keys across the two or more storage providers, wherein the secret share keys are distributed to the third set of two or more storage providers separately from corresponding coded chunks.

17. The product of claim 16, wherein the instructions further cause the processor to:

generate an encryption key for the ciphertext file from a shuffle key, the cipher key, the erasure coding details, and the distribution details;

wherein the communications interface communicates the encryption key to the user.

18. The product of claim 17, wherein the distributed coded chunks and the secret shares of metadata stored together at each of the storage providers are identified as storage shares.

19. The product of claim 18, wherein the user storage constraints comprise:

a customer type that identities an industry of the user and a type of data from which the input data blocks are comprised;

a domain compliance for the industry of the user that identifies a type of data compliance, the type of data compliance comprising HIPAA, financial, or multi-media;

a fault tolerance threshold;

a security type, or a security threshold, or both;

a retrievability threshold that identifies a probability of retrieving stored data blocks;

a repairability threshold that identifies a probability of repairing stored data blocks;

simultaneous provider attacks threshold that identifies a subset of the number of the providers required to aggregate the storage shares to re-generate the encryption key communicated to the user; and

budget for storage and retrieval of the input data blocks.

20. The product of claim 18, wherein the instructions further cause the processor to:

interrogate the storage shares by executing a corrupt or modified block check to test the retrievability of the distributed coded chunks and secret shares of metadata of the storage shares;

mark the distributed coded chunks and secret shares of metadata that fail the corrupt or modified block check.

21. The product of claim 20, wherein the instructions further cause the processor to:

retrieve a number of the uncorrupted data shares from the storage providers;

re-generate the encryption key from the secret shares of metadata of the uncorrupted data shares;

retrieve, using the re-generated encryption key, the uncorrupted data shares from the storage providers;

communicate the retrieved uncorrupted distributed coded chunks to the decoder;

decode the retrieved uncorrupted distributed coded chunks to obtain the data fields of the input data blocks for the corrupted distributed coded chunks;

re-code the data fields of the input data blocks for the corrupted distributed coded chunks;

communicate re-coded chunks to the distributor to distribute the re-coded chunks to the storage providers based on the user storage constraints; and

generate distribution details for the distributed re-coded chunks.