ENABLING SECURE BIG DATA ANALYTICS IN THE CLOUD

• US 20180173887A1
• Filed: 02/13/2018
• Published: 06/21/2018
• Est. Priority Date: 09/11/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Paulovicks, Brent, Sheinin, Vadim, Zerfos, Petros
• Granted Patent: US 10,410,011 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: H04L 63/00 Network architectures or network communication protocols for network security cryptographic mechanisms or cryptographic arrangements for secret or secure communication H04L9/00; network architectures or network communication protocols for wireless network security H04W12/00; security arrangements for protecting computers or computer systems against unauthorised activity G06F21/00
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Methods are provided for enabling secure big data analytics in the cloud. A method includes reading a secure file, by a Secure Distributed File System in a public cloud service provider. The reading step includes computing a hash of a name of the secure file to obtain a hashed file name, finding metadata for the secure file using the hashed file name, retrieving a sharing policy identifier from the metadata, and obtaining authorization from an external entity to decrypt the secure file. The reading step further includes extracting a security key and encrypted data file names from the metadata using the sharing policy identifier, requesting one or more encrypted data files that form the secure file from a node of the public cloud service provider, and at least one of decrypting and reconstructing plaintext data for the secure file from the one or more encrypted data files.

Claims

1. A method, comprising:

reading a secure file, by a Secure Distributed File System in a public cloud service provider having a processor and a memory device, the reading comprising;

computing a hash of a name of the secure file to obtain a hashed file name;

finding metadata for the secure file using the hashed file name, the metadata including a sharing policy identifier configured for obtaining a security key;

retrieving the sharing policy identifier from the metadata, and extracting the security key and encrypted data file names from the metadata using the sharing policy identifier;

at least one of decrypting and reconstructing plaintext data for the secure file from one or more requested encrypted data files.

2. The method of claim 1, wherein each of the one or more encrypted data files comprise a metadata portion and one or more data files, the one or more data files including encrypted data.

3. The method of claim 2, wherein the metadata portion comprises a plain text metadata portion that includes a share-policy-name, and an encrypted metadata portion that describes an encryption method used to encrypt the encrypted data and a listing of names of the one or more data files including the encrypted data.

4. The method of claim 2, wherein the metadata portion is stored as any of (a) a separate file, (b) as parameters of a key-value pair that are stored in a key-value store and that correspond to the one or more data files including the encrypted data, and (c) appended at an end of the one or more data files including the encrypted data.

5. The method of claim 2, wherein the metadata portion is stored as the separate file in another distributed file system included in the public cloud service provider.

6. The method of claim 1, wherein said reading step is performed responsive to a request for the secure file by a MapReduce job.

7. The method of claim 1, wherein the external entity is a key caching server.

8. The method of claim 1, wherein the secure file is synchronized with an enterprise storage system.

9. A non-transitory article of manufacture tangibly embodying a computer readable program which when executed causes a computer to perform the steps of claim 1.

10. A method, comprising:

creating, by a MapReduce application, a secure file configured for storage in a Secure Distributed File System (SDFS) in a public cloud service provider having a processor and a memory device, the creating further comprising;

determining a sharing policy identifier for the secure file, the sharing policy identifier being configured for obtaining a security key;

obtaining a security key and security parameters for the secure file from an external entity;

writing one or more encrypted data files for the secure file, and computing a hash of a name of the secure file;

storing the encrypted data files and metadata for the secure file, the encrypted data being configured for decryption by the SDFS.

11. The method of claim 10, wherein each of the one or more encrypted data files comprise a metadata portion and one or more data files, the one or more data files including encrypted data.

12. The method of claim 11, wherein the metadata portion comprises a plain text metadata portion that includes a share-policy-name, and an encrypted metadata portion that describes an encryption method used to encrypt the encrypted data and a listing of names of the one or more data files including the encrypted data.

13. The method of claim 11, wherein the metadata portion is stored as any of (a) a separate file, (b) as parameters of a key-value pair that are stored in a key-value store and that correspond to the one or more data files including the encrypted data, and (c) appended at an end of the one or more data files including the encrypted data.

14. The method of claim 10, wherein the secure file is synchronized with an enterprise storage system.

15. A non-transitory article of manufacture tangibly embodying a computer readable program which when executed causes a computer to perform the steps of claim 10.

16. A method, comprising:

determining, by a Secure Distributed File System in a public cloud service provider having a processor and a memory device, a security key for a file with an obfuscated file name, the determining further comprising;

determining a storage method used to storing metadata for the file with the obfuscated file name;

obtaining the metadata responsive to the storage method, and extracting a sharing policy identifier from the metadata;

forwarding the sharing policy identifier to an external entity to request a security key for the file with the obfuscated file name; and

receiving the security key from the external entity.

17. The method of claim 16, wherein the storage method comprises storing the metadata as any of (a) a separate file, (b) as parameters of a key-value pair that are stored in a key-value store and that correspond to the one or more data files including the encrypted data, and (c) appended at an end of one or more data files that form the file with the obfuscated file name, the one or more data files including encrypted data.

18. The method of claim 16, wherein the separate file is stored in another Distributed File System included in the public cloud service provider.

19. The method of claim 16, wherein the file is synchronized with an enterprise storage system.

20. A non-transitory article of manufacture tangibly embodying a computer readable program which when executed causes a computer to perform the steps of claim 16.