Distributed data set encryption and decryption

• US 10,185,722 B2
• Filed: 05/30/2018
• Issued: 01/22/2019
• Est. Priority Date: 07/27/2015
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bowman, Brian Payton, Gass, III, Mark Kuebler
• Primary Examiner(s): Woolwine, Shane
• Application Number:
  US15/992,670
• Publication Number:
  US 20180276231A1
• Time in Patent Office: 237 Days
• Field of Search: 711173, 711135
• CPC Class Codes: G06F 12/0292 using tables or multilevel address translation means G06F12/023 takes precedence; address translation in virtual memory systems G06F12/10
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 2212/1016 Performance improvement
• CPC Class Codes: G06F 2212/1056 Simplification
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/262 configured as RAID
• CPC Class Codes: G06F 2212/263 Network storage, e.g. SAN or NAS
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: H05K 999/99 dummy group

Abstract

An apparatus may include a processor component caused to: generate map entries in map data descriptive of encrypted data blocks within a data file; use first map block encryption data to encrypt a first map extension of the map data; transmit the encrypted first map extension for storage within the data file; store the first map block encryption data within the second map extension; use second map block encryption data to encrypt a second map extension of the map data after storage of the first map block encryption data therein; transmit encrypted second map extension for storage within the data file; store the second map block encryption data within the map base; use third map block encryption data to encrypt a map base of the map data after storage of the second map block encryption data therein; and transmit the encrypted map base for storage within the data file.

Claims

1. An apparatus comprising a processor component and a storage to store instructions that, when executed by the processor component, cause the processor component to perform operations comprising:

use, by the processor component, first map block encryption data to encrypt a first map extension of multiple map extensions of a map data to generate a first encrypted map extension of multiple encrypted map extensions, wherein;

each map extension of the multiple map extensions comprises multiple map entries; and

each map entry of the multiple map entries within each map extension comprises data block encryption data used to encrypt a portion of a data set to generate a corresponding encrypted data block of multiple encrypted data blocks, and an indication of a data block size of the corresponding encrypted data block;

store the first encrypted map extension at a first location within a data file maintained by one or more storage devices to store the data set and the map data in encrypted form;

use, by the processor component, second map block encryption data to encrypt a second map extension of the multiple map extensions to generate a second encrypted map extension of the multiple encrypted map extensions, wherein;

the second map block encryption data differs from the first map block encryption data; and

the second map extension comprises the first map block encryption data;

store the second encrypted map extension at a second location within the data file;

use, by the processor component, third map block encryption data to encrypt a map base of the map data to generate an encrypted map base, wherein;

the third map block encryption data differs from the first map block encryption data and the second map block encryption data;

the map base comprises the second map block encryption data; and

the map base comprises multiple extension pointers that each point to a corresponding location within the data file at which a corresponding encrypted map extension of the multiple encrypted map extensions is stored; and

store the encrypted map base at a third location within the data file.

2. The apparatus of claim 1, wherein the processor component is caused to perform operations comprising:

compare a size of the map data to a threshold map data size;

determine, by the processor component, whether to divide the map data into the map base and the multiple map extensions based on the comparison; and

in response to a determination that the size of the map data exceeds the threshold map data size, perform operations comprising;

determine, by the processor component, a quantity of the multiple map extensions based on a size of the map data; and

generate the map extensions of the multiple map extensions to have exponentially increasing sizes wherein the first map extension is twice the size of the second map extension.

3. The apparatus of claim 1, wherein:

the third location within the data file at which the encrypted map base is to be stored comprises a predetermined location within the data file relative to a starting end of the data file; and

the processor component is caused to perform operations comprising;

derive the second location within the data file at which the second map extension is to be stored based on the third location and a size of the encrypted map base, wherein the second location follows the encrypted map base and precedes the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension; and

derive the first location within the data file at which the first map extension is to be stored based on the second location, a size of the second encrypted map extension and a total size of the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension, wherein the second location follows the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension and precedes the multiple encrypted data blocks corresponding to the multiple map entries of the first map extension.

4. The apparatus of claim 1, wherein the processor component is caused to perform operations comprising:

randomly generate a first salt value as part of the first map block encryption data;

use, by the processor component, the first map block encryption data and a pass phrase to generate a first encryption cipher;

use, by the processor component, the first encryption cipher and a size of the first map extension to encrypt the first map extension to generate the first encrypted map extension;

randomly generate a second salt value as part of the second map block encryption data;

use, by the processor component, the second map block encryption data and the pass phrase to generate a second encryption cipher;

use, by the processor component, the second encryption cipher and a size of the second map extension to encrypt the second map extension to generate the second encrypted map extension;

randomly generate a third salt value as part of the third map block encryption data;

use, by the processor component, the third map block encryption data and the pass phrase to generate a third encryption cipher; and

use, by the processor component, the third encryption cipher and a size of the map base to encrypt the map base to generate the encrypted map base, wherein the pass phrase is not to be transmitted to the one or more storage devices.

5. The apparatus of claim 4, wherein the processor component is caused to perform operations comprising:

randomly generate a fourth salt value as part of a metadata block encryption data;

use, by the processor component, the metadata block encryption data and the pass phrase to generate a fourth encryption cipher;

use, by the processor component, the fourth encryption cipher and a size of metadata descriptive of an organization of data of the data set to encrypt the metadata to generate encrypted metadata; and

store the encrypted metadata at a fourth location within the data file, wherein the map base comprises the metadata block encryption data.

6. The apparatus of claim 1, wherein the processor component is caused to perform operations comprising:

perform an XOR operation with the third map block encryption data; and

following the performance of the XOR operation, transmit the third map block encryption data to the one or more storage devices to be stored at a predetermined location within the data file.

7. The apparatus of claim 1, wherein the processor component is caused to perform operations comprising:

receive, at a control device that comprises the processor component, the data block encryption data and the data block size for each encrypted data block of the multiple encrypted data blocks from multiple node devices;

receive, at the control device and from each node device of the multiple node devices, a request for a pointer to at location within the data file at which the node device is to store an encrypted data block of the multiple encrypted data blocks; and

in response to each request for a pointer received from a node device of the multiple node devices, derive the location within the data file at which the node device is to store the encrypted data block, and transmit, to node device, a pointer to the location within the data file.

8. The apparatus of claim 1, wherein the processor component is caused to perform operations comprising:

receive, at a control device that comprises the processor component, the data block encryption data and the data block size for each encrypted data block of the multiple encrypted data blocks from multiple node devices;

receive, at the control device and from each node device of the multiple node devices, at least one encrypted data block of the multiple encrypted data blocks to store an encrypted data block of the multiple encrypted data blocks; and

in response to each receipt of at least one encrypted data block from a node device of the multiple node devices, derive the location within the data file at which the control device is to store the at least one encrypted data block, and transmit the at least one encrypted data block to the one or more storage devices to store within the data file at the location.

9. The apparatus of claim 1, wherein the processor component is caused, at a time following storage of the data set as the multiple encrypted data blocks within the data file, and at a time following storage of the map data as the encrypted map base and the multiple encrypted map extensions within the data file, to perform operations comprising:

retrieve the third map block encryption data from a predetermined location within the data file;

retrieve the encrypted map base from the third location within the data file;

use, by the processor component, the third map block encryption data and a pass phrase to generate a third decryption cipher;

use, by the processor component, the third decryption cipher and a size of the encrypted map base to decrypt the encrypted map base to regenerate the map base;

retrieve the second map block encryption data from the map base;

retrieve the second encrypted map extension from the second location within the data file;

use, by the processor component, the second map block encryption data and the pass phrase to generate a second decryption cipher;

use, by the processor component, the second decryption cipher and a size of the second encrypted map extension to decrypt the second encrypted map extension to regenerate the second map extension;

retrieve the first map block encryption data from the second map extension;

retrieve the first encrypted map extension from the first location within the data file;

use, by the processor component, the first map block encryption data and the pass phrase to generate a first decryption cipher; and

use, by the processor component, the first decryption cipher and a size of the first encrypted map extension to decrypt the first encrypted map extension to regenerate the first map extension.

10. The apparatus of claim 9, wherein the processor component is caused to perform operations comprising:

derive, by the processor component, a distribution of the multiple encrypted data blocks among multiple node devices; and

for each encrypted data block of the multiple encrypted data blocks, transmit at least the corresponding data block encryption data to a node device of the multiple node devices to which the encrypted data block is assigned in the distribution to enable the node device to decrypt the encrypted data block.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor component to perform operations comprising:

use, by the processor component, first map block encryption data to encrypt a first map extension of multiple map extensions of a map data to generate a first encrypted map extension of multiple encrypted map extensions, wherein;

each map extension of the multiple map extensions comprises multiple map entries; and

each map entry of the multiple map entries within each map extension comprises data block encryption data used to encrypt a portion of a data set to generate a corresponding encrypted data block of multiple encrypted data blocks, and an indication of a data block size of the corresponding encrypted data block;

store the first encrypted map extension at a first location within a data file maintained by one or more storage devices to store the data set and the map data in encrypted form;

use, by the processor component, second map block encryption data to encrypt a second map extension of the multiple map extensions to generate a second encrypted map extension of the multiple encrypted map extensions, wherein;

the second map block encryption data differs from the first map block encryption data; and

the second map extension comprises the first map block encryption data;

store the second encrypted map extension at a second location within the data file;

use, by the processor component, third map block encryption data to encrypt a map base of the map data to generate an encrypted map base, wherein;

the third map block encryption data differs from the first map block encryption data and the second map block encryption data;

the map base comprises the second map block encryption data; and

the map base comprises multiple extension pointers that each point to a corresponding location within the data file at which a corresponding encrypted map extension of the multiple encrypted map extensions is stored; and

store the encrypted map base at a third location within the data file.

12. The computer-program product of claim 11, wherein the processor component is caused to perform operations comprising:

compare a size of the map data to a threshold map data size;

determine, by the processor component, whether to divide the map data into the map base and the multiple map extensions based on the comparison; and

in response to a determination that the size of the map data exceeds the threshold map data size, perform operations comprising;

determine, by the processor component, a quantity of the multiple map extensions based on a size of the map data; and

generate the map extensions of the multiple map extensions to have exponentially increasing sizes.

13. The computer-program product of claim 11, wherein:

the third location within the data file at which the encrypted map base is to be stored comprises a predetermined location within the data file relative to a starting end of the data file; and

the processor component is caused to perform operations comprising;

derive the second location within the data file at which the second map extension is to be stored based on the third location and a size of the encrypted map base, wherein the second location follows the encrypted map base and precedes the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension; and

derive the first location within the data file at which the first map extension is to be stored based on the second location, a size of the second encrypted map extension and a total size of the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension, wherein the second location follows the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension and precedes the multiple encrypted data blocks corresponding to the multiple map entries of the first map extension.

14. The computer-program product of claim 11, wherein the processor component is caused to perform operations comprising:

randomly generate a first salt value as part of the first map block encryption data;

use, by the processor component, the first map block encryption data and a pass phrase to generate a first encryption cipher;

use, by the processor component, the first encryption cipher and a size of the first map extension to encrypt the first map extension to generate the first encrypted map extension;

randomly generate a second salt value as part of the second map block encryption data;

use, by the processor component, the second map block encryption data and the pass phrase to generate a second encryption cipher;

use, by the processor component, the second encryption cipher and a size of the second map extension to encrypt the second map extension to generate the second encrypted map extension;

randomly generate a third salt value as part of the third map block encryption data;

use, by the processor component, the third map block encryption data and the pass phrase to generate a third encryption cipher; and

use, by the processor component, the third encryption cipher and a size of the map base to encrypt the map base to generate the encrypted map base, wherein the pass phrase is not to be transmitted to the one or more storage devices.

15. The computer-program product of claim 14, wherein the processor component is caused to perform operations comprising:

randomly generate a fourth salt value as part of a metadata block encryption data;

use, by the processor component, the metadata block encryption data and the pass phrase to generate a fourth encryption cipher;

use, by the processor component, the fourth encryption cipher and a size of metadata descriptive of an organization of data of the data set to encrypt the metadata to generate encrypted metadata; and

store the encrypted metadata at a fourth location within the data file, wherein the map base comprises the metadata block encryption data.

16. The computer-program product of claim 11, wherein the processor component is caused to perform operations comprising:

perform an XOR operation with the third map block encryption data; and

following the performance of the XOR operation, transmit the third map block encryption data to the one or more storage devices to be stored at a predetermined location within the data file.

17. The computer-program product of claim 11, wherein the processor component is caused to perform operations comprising:

receive, at a control device that comprises the processor component, the data block encryption data and the data block size for each encrypted data block of the multiple encrypted data blocks from multiple node devices;

receive, at the control device and from each node device of the multiple node devices, a request for a pointer to at location within the data file at which the node device is to store an encrypted data block of the multiple encrypted data blocks; and

in response to each request for a pointer received from a node device of the multiple node devices, derive the location within the data file at which the node device is to store the encrypted data block, and transmit, to node device, a pointer to the location within the data file.

18. The computer-program product of claim 11, wherein the processor component is caused to perform operations comprising:

receive, at a control device that comprises the processor component, the data block encryption data and the data block size for each encrypted data block of the multiple encrypted data blocks from multiple node devices;

receive, at the control device and from each node device of the multiple node devices, at least one encrypted data block of the multiple encrypted data blocks to store an encrypted data block of the multiple encrypted data blocks; and

in response to each receipt of at least one encrypted data block from a node device of the multiple node devices, derive the location within the data file at which the control device is to store the at least one encrypted data block, and transmit the at least one encrypted data block to the one or more storage devices to store within the data file at the location.

19. The computer-program product of claim 11, wherein the processor component is caused, at a time following storage of the data set as the multiple encrypted data blocks within the data file, and at a time following storage of the map data as the encrypted map base and the multiple encrypted map extensions within the data file, to perform operations comprising:

retrieve the third map block encryption data from a predetermined location within the data file;

retrieve the encrypted map base from the third location within the data file;

use, by the processor component, the third map block encryption data and a pass phrase to generate a third decryption cipher;

use, by the processor component, the third decryption cipher and a size of the encrypted map base to decrypt the encrypted map base to regenerate the map base;

retrieve the second map block encryption data from the map base;

retrieve the second encrypted map extension from the second location within the data file;

use, by the processor component, the second map block encryption data and the pass phrase to generate a second decryption cipher;

use, by the processor component, the second decryption cipher and a size of the second encrypted map extension to decrypt the second encrypted map extension to regenerate the second map extension;

retrieve the first map block encryption data from the second map extension;

retrieve the first encrypted map extension from the first location within the data file;

use, by the processor component, the first map block encryption data and the pass phrase to generate a first decryption cipher; and

use, by the processor component, the first decryption cipher and a size of the first encrypted map extension to decrypt the first encrypted map extension to regenerate the first map extension.

20. The computer-program product of claim 19, wherein the processor component is caused to perform operations comprising:

derive, by the processor component, a distribution of the multiple encrypted data blocks among multiple node devices; and

for each encrypted data block of the multiple encrypted data blocks, transmit at least the corresponding data block encryption data to a node device of the multiple node devices to which the encrypted data block is assigned in the distribution to enable the node device to decrypt the encrypted data block.

21. A computer-implemented method comprising:

using, by a processor component, first map block encryption data to encrypt a first map extension of multiple map extensions of a map data to generate a first encrypted map extension of multiple encrypted map extensions, wherein;

each map extension of the multiple map extensions comprises multiple map entries; and

each map entry of the multiple map entries within each map extension comprises data block encryption data used to encrypt a portion of a data set to generate a corresponding encrypted data block of multiple encrypted data blocks, and an indication of a data block size of the corresponding encrypted data block;

storing the first encrypted map extension at a first location within a data file maintained by one or more storage devices to store the data set and the map data in encrypted form;

using, by the processor component, second map block encryption data to encrypt a second map extension of the multiple map extensions to generate a second encrypted map extension of the multiple encrypted map extensions, wherein;

the second map block encryption data differs from the first map block encryption data; and

the second map extension comprises the first map block encryption data;

storing the second encrypted map extension at a second location within the data file;

using, by the processor component, third map block encryption data to encrypt a map base of the map data to generate an encrypted map base, wherein;

the third map block encryption data differs from the first map block encryption data and the second map block encryption data;

the map base comprises the second map block encryption data; and

the map base comprises multiple extension pointers that each point to a corresponding location within the data file at which a corresponding encrypted map extension of the multiple encrypted map extensions is stored; and

storing the encrypted map base at a third location within the data file.

22. The computer-implemented method of claim 21, comprising:

comparing a size of the map data to a threshold map data size;

determining, by the processor component, whether to divide the map data into the map base and the multiple map extensions based on the comparison; and

in response to a determination that the size of the map data exceeds the threshold map data size, performing operations comprising;

determining, by the processor component, a quantity of the multiple map extensions based on a size of the map data; and

generating the map extensions of the multiple map extensions to have exponentially increasing sizes.

23. The computer-implemented method of claim 21, wherein:

the third location within the data file at which the encrypted map base is to be stored comprises a predetermined location within the data file relative to a starting end of the data file; and

the method comprises;

deriving the second location within the data file at which the second map extension is to be stored based on the third location and a size of the encrypted map base, wherein the second location follows the encrypted map base and precedes the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension; and

deriving the first location within the data file at which the first map extension is to be stored based on the second location, a size of the second encrypted map extension and a total size of the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension, wherein the second location follows the multiple encrypted data blocks corresponding to the multiple map entries of the second map extension and precedes the multiple encrypted data blocks corresponding to the multiple map entries of the first map extension.

24. The computer-implemented method of claim 21, comprising:

randomly generating a first salt value as part of the first map block encryption data;

using, by the processor component, the first map block encryption data and a pass phrase to generate a first encryption cipher;

using, by the processor component, the first encryption cipher and a size of the first map extension to encrypt the first map extension to generate the first encrypted map extension;

randomly generating a second salt value as part of the second map block encryption data;

using, by the processor component, the second map block encryption data and the pass phrase to generate a second encryption cipher;

using, by the processor component, the second encryption cipher and a size of the second map extension to encrypt the second map extension to generate the second encrypted map extension;

randomly generating a third salt value as part of the third map block encryption data;

using, by the processor component, the third map block encryption data and the pass phrase to generate a third encryption cipher; and

using, by the processor component, the third encryption cipher and a size of the map base to encrypt the map base to generate the encrypted map base, wherein the pass phrase is not to be transmitted to the one or more storage devices.

25. The computer-implemented method of claim 24, comprising:

randomly generating a fourth salt value as part of a metadata block encryption data;

using, by the processor component, the metadata block encryption data and the pass phrase to generate a fourth encryption cipher;

using, by the processor component, the fourth encryption cipher and a size of metadata descriptive of an organization of data of the data set to encrypt the metadata to generate encrypted metadata; and

storing the encrypted metadata at a fourth location within the data file, wherein the map base comprises the metadata block encryption data.

26. The computer-implemented method of claim 21, comprising:

performing an XOR operation with the third map block encryption data; and

following the performance of the XOR operation, transmitting the third map block encryption data to the one or more storage devices to be stored at a predetermined location within the data file.

27. The computer-implemented method of claim 21, comprising:

receiving, at a control device that comprises the processor component, the data block encryption data and the data block size for each encrypted data block of the multiple encrypted data blocks from multiple node devices;

receiving, at the control device and from each node device of the multiple node devices, a request for a pointer to at location within the data file at which the node device is to store an encrypted data block of the multiple encrypted data blocks; and

in response to each request for a pointer received from a node device of the multiple node devices, deriving the location within the data file at which the node device is to store the encrypted data block, and transmit, to node device, a pointer to the location within the data file.

28. The computer-implemented method of claim 21, comprising:

receiving, at a control device that comprises the processor component, the data block encryption data and the data block size for each encrypted data block of the multiple encrypted data blocks from multiple node devices;

receiving, at the control device and from each node device of the multiple node devices, at least one encrypted data block of the multiple encrypted data blocks to store an encrypted data block of the multiple encrypted data blocks; and

in response to each receipt of at least one encrypted data block from a node device of the multiple node devices, deriving the location within the data file at which the control device is to store the at least one encrypted data block, and transmitting the at least one encrypted data block to the one or more storage devices to store within the data file at the location.

29. The computer-implemented method of claim 21, comprising, at a time following storage of the data set as the multiple encrypted data blocks within the data file, and at a time following storage of the map data as the encrypted map base and the multiple encrypted map extensions within the data file, performing operations comprising:

retrieving the third map block encryption data from a predetermined location within the data file;

retrieving the encrypted map base from the third location within the data file;

using, by the processor component, the third map block encryption data and a pass phrase to generate a third decryption cipher;

using, by the processor component, the third decryption cipher and a size of the encrypted map base to decrypt the encrypted map base to regenerate the map base;

retrieving the second map block encryption data from the map base;

retrieving the second encrypted map extension from the second location within the data file;

using, by the processor component, the second map block encryption data and the pass phrase to generate a second decryption cipher;

using, by the processor component, the second decryption cipher and a size of the second encrypted map extension to decrypt the second encrypted map extension to regenerate the second map extension;

retrieving the first map block encryption data from the second map extension;

retrieving the first encrypted map extension from the first location within the data file;

using, by the processor component, the first map block encryption data and the pass phrase to generate a first decryption cipher; and

using, by the processor component, the first decryption cipher and a size of the first encrypted map extension to decrypt the first encrypted map extension to regenerate the first map extension.

30. The computer-implemented method of claim 29, comprising:

deriving, by the processor component, a distribution of the multiple encrypted data blocks among multiple node devices; and

for each encrypted data block of the multiple encrypted data blocks, transmitting at least the corresponding data block encryption data to a node device of the multiple node devices to which the encrypted data block is assigned in the distribution to enable the node device to decrypt the encrypted data block.