Distributed data set encryption and decryption

• US 9,946,719 B2
• Filed: 09/01/2017
• Issued: 04/17/2018
• Est. Priority Date: 07/27/2015
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bowman, Brian Payton, Gass, III, Mark Kuebler
• Primary Examiner(s): Shah, Sanjiv
• Assistant Examiner(s): WOOLWINE, SHANE D
• Application Number:
  US15/694,674
• Publication Number:
  US 20180011867A1
• Time in Patent Office: 228 Days
• Field of Search: 711173, 711135
• CPC Class Codes: G06F 12/0292 using tables or multilevel address translation means G06F12/023 takes precedence; address translation in virtual memory systems G06F12/10
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 2212/1016 Performance improvement
• CPC Class Codes: G06F 2212/1056 Simplification
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/262 configured as RAID
• CPC Class Codes: G06F 2212/263 Network storage, e.g. SAN or NAS
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455

Abstract

An apparatus includes a processor component of a first node device caused to receive data block encryption data and an indication of size of an encrypted data block distributed to the first node device for decryption, and in response to the data set being of encrypted data: receive an indication of the quantity of sub-blocks within the encrypted data block, and a hashed identifier for each data sub-block; use the data block encryption data to decrypt the encrypted data block to regenerate data set portions from the data sub-blocks; analyze the hashed identifier of each data sub-block to determine whether all data set portions are distributed to the first node device for processing; and in response to a determination that at least one data set portion is to be distributed to a second node device for processing, transmit the at least one data set portion to the second node device.

Claims

1. An apparatus comprising a processor component of a first node device and a storage to store instructions that, when executed by the processor component, cause the processor component to perform operations comprising:

receive, at the first node device of multiple node devices, an indication of a processing task to perform with a data set and metadata indicative of an organization of data within the data set, wherein;

the data set is stored within a data file as multiple encrypted data blocks; and

each encrypted data block is generated by encryption of at least one data set portion of the data set using corresponding data block encryption data separately generated for each encrypted data block;

receive, at the first node device, data block encryption data and an indication of a size of an encrypted data block of the multiple encrypted data blocks that is distributed to the first node device for decryption;

analyze the metadata to determine whether the data of the data set is partitioned data;

in response to an indication in the metadata that the data of the data set is partitioned data, wherein the data within the data set is organized into multiple partitions that are each distributable to a single node device, the processor component is caused to perform operations comprising;

receive, at the first node device, an indication of a quantity of one or more data sub-blocks within the encrypted data block, and for each data sub-block of the encrypted data block, a sub-block size and a hashed identifier of the data sub-block, wherein;

each data sub-block of the encrypted data block corresponds to a data set portion of the data set; and

each data set portion comprises data of a partition of the multiple partitions that is identified by the corresponding hashed identifier;

use the data block encryption data to decrypt the encrypted data block to regenerate one or more data set portions from the one or more data sub-blocks of the encrypted data block;

analyze the hashed identifier of each data sub-block of the encrypted data block to determine whether all of the one or more data set portions are distributed to the first node device for processing to perform the processing task; and

in response to a determination that at least one data set portion of the one or more data set portions is to be distributed to a second node device of the multiple node devices for processing, the processor component is caused to perform operations comprising;

transmit, from the first node device, the at least one data set portion to the second node device; and

perform the processing task with any data set portion of the one or more data set portions that are distributed to the first node device for processing.

2. The apparatus of claim 1, wherein in response to a determination that all of the one or more data set portions are to be distributed to the first node device for processing, the processor component is caused to perform the processing task with all of the one or more data set portions.

3. The apparatus of claim 1, wherein in response to a lack of indication in the metadata that the data of the data set is partitioned data, wherein the encrypted data block comprises a single encrypted data set portion of the data set, the processor is caused to perform operations comprising:

use the data block encryption data to decrypt the encrypted data block to regenerate the single data set portion; and

perform the processing task with the single data set portion.

4. The apparatus of claim 1, wherein:

the multiple node devices are able to exchange encrypted data blocks with one or more storage devices that store the data file; and

the processor component is caused to perform operations comprising;

receive, at the first node device, a pointer to a location within the data file at which the encrypted data block is stored;

transmit an instruction to the one or more storage devices to provide the encrypted data block from the location specified by the pointer; and

receive, at the first node device, the encrypted data block from the one or more storage devices.

5. The apparatus of claim 1, wherein:

the multiple node devices are unable to exchange encrypted data blocks with one or more storage devices that store the data file; and

the processor component is caused to receive the encrypted data block from a control device along with the data block encryption data and the indication of the size of the encrypted data block.

6. The apparatus of claim 1, wherein:

the data block encryption data comprises a salt value previously randomly generated and used to encrypt one or more data set portions of the data set as one or more corresponding data sub-blocks to generate the encrypted data block; and

the processor component is caused to perform operations comprising;

receive, at the first node device, a pass phrase, wherein the pass phrase is to be used by the multiple node devices to decrypt the multiple encrypted data blocks, and is not to be stored with the data file;

use, by the processor component, the salt value and the pass phrase to generate a decryption cipher; and

use, by the processor component, the decryption cipher and the size of the encrypted data block to decrypt the encrypted data block.

7. The apparatus of claim 1, wherein:

the processor component comprises multiple processor cores; and

in response to an indication in the metadata that the data of the data set is partitioned data, the processor component is caused to perform the processing task with each data set portion of the one or more data set portions using a separate one of the multiple processor cores at least partially in parallel.

8. The apparatus of claim 1, wherein:

the processor component comprises multiple processor cores; and

the processor component is caused to decrypt each encrypted data block of a subset of the multiple encrypted data blocks using a separate one of the multiple processor cores at least partially in parallel.

9. The apparatus of claim 1, wherein:

a third node device of the multiple node devices decrypts another encrypted data block comprising a data sub-block that corresponds to another data set portion of the data set that is assigned to be processed by the first node device; and

the processor component is caused to receive, at the first node device, the other data set portion from the third node device.

10. The apparatus of claim 1, wherein:

the first node device comprises a controller to coordinate performances of decryption and the processing task among the multiple node devices; and

the controller comprises a controller processor component and a controller storage to store controller instructions that, when executed by the controller processor component, cause the controller processor component to perform operations comprising;

analyze the metadata to determine whether the data of the data set is partitioned data; and

in response to a determination that the data of the data set is partitioned data, the controller processor is caused to perform operations comprising;

derive, by the controller processor component, a distribution of the data sub-blocks within the multiple encrypted data blocks among the multiple node devices for purposes of processing to perform the processing task at least partially in parallel;

derive, by the controller processor component, a distribution of the multiple encrypted data blocks among the multiple node devices for purposes of decryption of the multiple encrypted data blocks at least partially in parallel based at least partially on the distribution of the data sub-blocks; and

for each encrypted data block of the multiple encrypted data blocks, transmit the corresponding data block encryption data to the node device to which the encrypted data block is to be distributed for decryption.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor component of a first node device to perform operations comprising:

receive, at the first node device of multiple node devices, an indication of a processing task to perform with a data set and metadata indicative of an organization of data within the data set, wherein;

the data set is stored within a data file as multiple encrypted data blocks; and

each encrypted data block is generated by encryption of at least one data set portion of the data set using corresponding data block encryption data separately generated for each encrypted data block;

receive, at the first node device, data block encryption data and an indication of a size of an encrypted data block of the multiple encrypted data blocks that is distributed to the first node device for decryption;

analyze the metadata to determine whether the data of the data set is partitioned data;

in response to an indication in the metadata that the data of the data set is partitioned data, wherein the data within the data set is organized into multiple partitions that are each distributable to a single node device, the processor component is caused to perform operations comprising;

receive, at the first node device, an indication of a quantity of one or more data sub-blocks within the encrypted data block, and for each data sub-block of the encrypted data block, a sub-block size and a hashed identifier of the data sub-block, wherein;

each data sub-block of the encrypted data block corresponds to a data set portion of the data set; and

each data set portion comprises data of a partition of the multiple partitions that is identified by the corresponding hashed identifier;

use the data block encryption data to decrypt the encrypted data block to regenerate one or more data set portions from the one or more data sub-blocks of the encrypted data block;

analyze the hashed identifier of each data sub-block of the encrypted data block to determine whether all of the one or more data set portions are distributed to the first node device for processing to perform the processing task; and

in response to a determination that at least one data set portion of the one or more data set portions is to be distributed to a second node device of the multiple node devices for processing, the processor component is caused to perform operations comprising;

transmit, from the first node device, the at least one data set portion to the second node device; and

perform the processing task with any data set portion of the one or more data set portions that are distributed to the first node device for processing.

12. The computer-program product of claim 11, wherein in response to a determination that all of the one or more data set portions are to be distributed to the first node device for processing, the processor component is caused to perform the processing task with all of the one or more data set portions.

13. The computer-program product of claim 11, wherein in response to a lack of indication in the metadata that the data of the data set is partitioned data, wherein the encrypted data block comprises a single encrypted data set portion of the data set, the processor is caused to perform operations comprising:

use the data block encryption data to decrypt the encrypted data block to regenerate the single data set portion; and

perform the processing task with the single data set portion.

14. The computer-program product of claim 11, wherein:

the multiple node devices are able to exchange encrypted data blocks with one or more storage devices that store the data file; and

the processor component is caused to perform operations comprising;

receive, at the first node device, a pointer to a location within the data file at which the encrypted data block is stored;

transmit an instruction to the one or more storage devices to provide the encrypted data block from the location specified by the pointer; and

receive, at the first node device, the encrypted data block from the one or more storage devices.

15. The computer-program product of claim 11, wherein:

the multiple node devices are unable to exchange encrypted data blocks with one or more storage devices that store the data file; and

the processor component is caused to receive the encrypted data block from a control device along with the data block encryption data and the indication of the size of the encrypted data block.

16. The computer-program product of claim 11, wherein:

the data block encryption data comprises a salt value previously randomly generated and used to encrypt one or more data set portions of the data set as one or more corresponding data sub-blocks to generate the encrypted data block; and

the processor component is caused to perform operations comprising;

receive, at the first node device, a pass phrase, wherein the pass phrase is to be used by the multiple node devices to decrypt the multiple encrypted data blocks, and is not to be stored with the data file;

use, by the processor component, the salt value and the pass phrase to generate a decryption cipher; and

use, by the processor component, the decryption cipher and the size of the encrypted data block to decrypt the encrypted data block.

17. The computer-program product of claim 11, wherein:

the processor component comprises multiple processor cores; and

in response to an indication in the metadata that the data of the data set is partitioned data, the processor component is caused to perform the processing task with each data set portion of the one or more data set portions using a separate one of the multiple processor cores at least partially in parallel.

18. The computer-program product of claim 11, wherein:

the processor component comprises multiple processor cores; and

the processor component is caused to decrypt each encrypted data block of a subset of the multiple encrypted data blocks using a separate one of the multiple processor cores at least partially in parallel.

19. The computer-program product of claim 11, wherein:

a third node device of the multiple node devices decrypts another encrypted data block comprising a data sub-block that corresponds to another data set portion of the data set that is assigned to be processed by the first node device; and

the processor component is caused to receive, at the first node device, the other data set portion from the third node device.

20. The computer-program product of claim 11, wherein:

the first node device comprises a controller to coordinate performances of decryption and the processing task among the multiple node devices; and

the controller comprises a controller processor component and a controller storage to store controller instructions that, when executed by the controller processor component, cause the controller processor component to perform operations comprising;

analyze the metadata to determine whether the data of the data set is partitioned data; and

in response to a determination that the data of the data set is partitioned data, the controller processor is caused to perform operations comprising;

derive, by the controller processor component, a distribution of the data sub-blocks within the multiple encrypted data blocks among the multiple node devices for purposes of processing to perform the processing task at least partially in parallel;

derive, by the controller processor component, a distribution of the multiple encrypted data blocks among the multiple node devices for purposes of decryption of the multiple encrypted data blocks at least partially in parallel based at least partially on the distribution of the data sub-blocks; and

for each encrypted data block of the multiple encrypted data blocks, transmit the corresponding data block encryption data to the node device to which the encrypted data block is to be distributed for decryption.

21. A computer-implemented method comprising:

receiving, at a first node device of multiple node devices, an indication of a processing task to perform with a data set and metadata indicative of an organization of data within the data set, wherein;

the data set is stored within a data file as multiple encrypted data blocks; and

each encrypted data block is generated by encryption of at least one data set portion of the data set using corresponding data block encryption data separately generated for each encrypted data block;

receiving, at the first node device, data block encryption data and an indication of a size of an encrypted data block of the multiple encrypted data blocks that is distributed to the first node device for decryption;

analyzing, by a processor component of the first node device, the metadata to determine whether the data of the data set is partitioned data;

in response to an indication in the metadata that the data of the data set is partitioned data, wherein the data within the data set is organized into multiple partitions that are each distributable to a single node device, performing operations comprising;

receiving, at the first node device, an indication of a quantity of one or more data sub-blocks within the encrypted data block, and for each data sub-block of the encrypted data block, a sub-block size and a hashed identifier of the data sub-block, wherein;

each data sub-block of the encrypted data block corresponds to a data set portion of the data set; and

each data set portion comprises data of a partition of the multiple partitions that is identified by the corresponding hashed identifier;

using, by the processor component, the data block encryption data to decrypt the encrypted data block to regenerate one or more data set portions from the one or more data sub-blocks of the encrypted data block;

analyzing, by the processor component, the hashed identifier of each data sub-block of the encrypted data block to determine whether all of the one or more data set portions are distributed to the first node device for processing to perform the processing task; and

in response to a determination that at least one data set portion of the one or more data set portions is to be distributed to a second node device of the multiple node devices for processing, performing operations comprising;

transmitting, from the first node device, the at least one data set portion to the second node device; and

performing, by the processor component, the processing task with any data set portion of the one or more data set portions that are distributed to the first node device for processing.

22. The computer-implemented method of claim 21, wherein in response to a determination that all of the one or more data set portions are to be distributed to the first node device for processing, performing, by the processor component, the processing task with all of the one or more data set portions.

23. The computer-implemented method of claim 21, wherein in response to a lack of indication in the metadata that the data of the data set is partitioned data, wherein the encrypted data block comprises a single encrypted data set portion of the data set, the performing operations comprising:

using, by the processor component, the data block encryption data to decrypt the encrypted data block to regenerate the single data set portion; and

performing, by the processor component, the processing task with the single data set portion.

24. The computer-implemented method of claim 21, wherein:

the multiple node devices are able to exchange encrypted data blocks with one or more storage devices that store the data file; and

the method comprises;

receiving, at the first node device, a pointer to a location within the data file at which the encrypted data block is stored;

transmitting, from the first node device, an instruction to the one or more storage devices to provide the encrypted data block from the location specified by the pointer; and

receiving, at the first node device, the encrypted data block from the one or more storage devices.

25. The computer-implemented method of claim 21, wherein:

the multiple node devices are unable to exchange encrypted data blocks with one or more storage devices that store the data file; and

the method comprises receiving, at the first node device, the encrypted data block from a control device along with the data block encryption data and the indication of the size of the encrypted data block.

26. The computer-implemented method of claim 21, wherein:

the data block encryption data comprises a salt value previously randomly generated and used to encrypt one or more data set portions of the data set as one or more corresponding data sub-blocks to generate the encrypted data block; and

the method comprises;

receiving, at the first node device, a pass phrase, wherein the pass phrase is to be used by the multiple node devices to decrypt the multiple encrypted data blocks, and is not to be stored with the data file;

using, by the processor component, the salt value and the pass phrase to generate a decryption cipher; and

using, by the processor component, the decryption cipher and the size of the encrypted data block to decrypt the encrypted data block.

27. The computer-implemented method of claim 21, wherein:

the processor component comprises multiple processor cores; and

the method comprises, in response to an indication in the metadata that the data of the data set is partitioned data, performing the processing task with each data set portion of the one or more data set portions using a separate one of the multiple processor cores at least partially in parallel.

28. The computer-implemented method of claim 21, wherein:

the processor component comprises multiple processor cores; and

the method comprises decrypting each encrypted data block of a subset of the multiple encrypted data blocks using a separate one of the multiple processor cores at least partially in parallel.

29. The computer-implemented method of claim 21, wherein:

a third node device of the multiple node devices decrypts another encrypted data block comprising a data sub-block that corresponds to another data set portion of the data set that is assigned to be processed by the first node device; and

the method comprises receiving, at the first node device, the other data set portion from the third node device.

30. The computer-implemented method of claim 21, wherein:

the first node device comprises a controller to coordinate performances of decryption and the processing task among the multiple node devices; and

the method comprises;

analyzing, by a controller processor component of the controller, the metadata to determine whether the data of the data set is partitioned data; and

in response to a determination that the data of the data set is partitioned data, performing operations comprising;

deriving, by the controller processor component, a distribution of the data sub-blocks within the multiple encrypted data blocks among the multiple node devices for purposes of processing to perform the processing task at least partially in parallel;

deriving, by the controller processor component, a distribution of the multiple encrypted data blocks among the multiple node devices for purposes of decryption of the multiple encrypted data blocks at least partially in parallel based at least partially on the distribution of the data sub-blocks; and

for each encrypted data block of the multiple encrypted data blocks, transmitting the corresponding data block encryption data to the node device to which the encrypted data block is to be distributed for decryption.