Compression and secure, end-to-end encrypted, ZFS cloud storage

• US 10,540,384 B2
• Filed: 05/31/2017
• Issued: 01/21/2020
• Est. Priority Date: 01/06/2017
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Maybee, Mark, Kremer, James, Gibson, Gavin
• Primary Examiner(s): Davis, Chenea
• Application Number:
  US15/610,380
• Publication Number:
  US 20180198765A1
• Time in Patent Office: 965 Days
• CPC Class Codes: G06F 1/28 Supervision thereof, e.g. detecting power-supply failure by out of limits supervision
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3037 where the computing system component is a memory, e.g. virtual memory, cache accessing, addressing or allocating within memory systems or architectures G06F12/00; checking stores for correct operation G11C29/00
• CPC Class Codes: G06F 11/328 Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/0897 with two or more cache hierarchy levels with multilevel cache hierarchies G06F12/0811
• CPC Class Codes: G06F 12/123 with age lists, e.g. queue, most recently used [MRU] list or least recently used [LRU] list
• CPC Class Codes: G06F 12/128 adapted to multidimensional cache systems, e.g. set-associative, multicache, multiset or multilevel
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/1748 : De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/182 : Distributed file systems
• CPC Class Codes: G06F 16/184 : implemented as replicated file system
• CPC Class Codes: G06F 16/1844 : Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/185 : Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 16/2365 : Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/434 : using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/9027 : Trees
• CPC Class Codes: G06F 2009/4557 : Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45583 : Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45595 : Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 21/602 : Providing cryptographic facilities or services
• CPC Class Codes: G06F 2201/80 : Database-specific techniques
• CPC Class Codes: G06F 2201/81 : Threshold
• CPC Class Codes: G06F 2201/84 : Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2212/154 : Networked environment
• CPC Class Codes: G06F 2212/225 : Hybrid cache memory, e.g. having both volatile and non-volatile portions
• CPC Class Codes: G06F 2212/262 : configured as RAID
• CPC Class Codes: G06F 2212/284 : being distributed
• CPC Class Codes: G06F 2212/313 : In storage device
• CPC Class Codes: G06F 2212/401 : Compressed data
• CPC Class Codes: G06F 2212/502 : using adaptive policy
• CPC Class Codes: G06F 3/0605 : by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0607 : by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0608 : Saving storage space on storage systems
• CPC Class Codes: G06F 3/061 : Improving I/O performance
• CPC Class Codes: G06F 3/0617 : in relation to availability
• CPC Class Codes: G06F 3/0623 : in relation to content
• CPC Class Codes: G06F 3/064 : Management of blocks
• CPC Class Codes: G06F 3/0641 : De-duplication techniques
• CPC Class Codes: G06F 3/0643 : Management of files
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0667 : at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5077 : Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: H03M 7/30 : Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
• CPC Class Codes: H03M 7/3086 : employing a sliding window, e.g. LZ77
• CPC Class Codes: H03M 7/40 : Conversion to or from variable length codes, e.g. Shannon-Fano code, Huffman code, Morse code
• CPC Class Codes: H04L 63/0428 : wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/0637 : Modes of operation, e.g. cipher block chaining [CBC], electronic codebook [ECB] or Galois/counter mode [GCM]
• CPC Class Codes: H04L 9/0822 : using key encryption key
• CPC Class Codes: H04L 9/14 : using a plurality of keys or algorithms

Abstract

Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.

Claims

1. A method of transforming blocks on a cloud object store that is remote from a block storage system, the method comprising:

receiving, at an application layer of the block storage system and through a system call interface of an interface layer of the block storage system, a first request to store or modify a file, the first request including file data;

generating, at a transactional object layer of the block storage system, a plurality of data blocks, each data block of the plurality of data blocks corresponding to at least a portion of the file data;

receiving, at the application layer of the block storage system and through the system call interface of the interface layer of the block storage system, one or more communications that identify one or more transformation actions to perform on the plurality of data blocks, the one or more transformation actions including one or more of;

a compression type to compress each data block of the plurality of data blocks by; and

an encryption type to encrypt each data block of the plurality of data blocks, wherein an encryption key associated with the encryption type is retrieved locally from the block storage system;

transforming each data block of the plurality of data blocks to form a plurality of transformed blocks by performing the one or more transformation actions on each data block of the plurality of data blocks;

generating, at the transactional object layer of the block storage system, a plurality of metadata blocks corresponding to the plurality of transformed blocks, the plurality of metadata blocks being configured to hierarchically point to lower-level blocks associated with the file and thereby correspond to at least part of a tree hierarchy for the file, wherein;

each metadata block of the plurality of metadata blocks includes one or more address pointers, each address pointer of the one or more address pointers being pointed to a transformed block of the plurality of transformed blocks or to a metadata block of the plurality of metadata blocks;

each transformed block of the plurality of transformed blocks being pointed to by at least one metadata block of the plurality of metadata blocks;

the plurality of metadata blocks includes a root block that is positioned at a top of the tree hierarchy for the file and one or more non-root metadata blocks; and

each non-root metadata block of the plurality of metadata blocks being pointed to by at least one metadata block of the plurality of metadata blocks of the tree hierarchy of the file;

causing a set of cloud storage objects to be stored in the cloud object store by transmitting the plurality of transformed blocks and the plurality of metadata blocks to a hybrid cloud storage system, the hybrid cloud storage system managing data storage in the cloud object store;

transmitting, to the hybrid cloud storage system, one or more second requests for a set of addresses, each address of the set of addresses corresponding to a transformed block of the plurality of transformed blocks or a metadata block of the plurality of metadata blocks; and

receiving, from the hybrid cloud storage system, one or more responses to the one or more second requests, each response of the one or more responses identifying an address corresponding to a transformed block of the plurality of transformed blocks or a metadata block of the plurality of metadata blocks, the address identifying a storage location in the cloud object store.

2. The method of claim 1, wherein the compression type includes one of:

LZ4, LZJB, GZIP, and ZLE.

3. The method of claim 1, wherein the encryption type includes advanced encryption standard (AES) with key lengths including 128, 192, and 256.

4. The method of claim 1, wherein the encryption key is randomly generated.

5. The method of claim 1, wherein a wrapping key is used to decrypt the encryption key.

6. The method of claim 5, wherein the wrapping key is a stored passphrase or an AES key.

7. The method of claim 1, wherein a compression ratio is 2 to 1.

8. One or more non-transitory tangible computer-readable storage media storing computer-executable instructions for performing a computer process for transforming blocks on a cloud object store that is remote from a block storage system on a computing system, the computer process comprising:

receiving, at an application layer of the block storage system and through a system call interface of an interface layer of the block storage system, a first request to store or modify a file, the first request including file data;

generating, at a transactional object layer of the block storage system, a plurality of data blocks, each data block of the plurality of data blocks corresponding to at least a portion of the file data;

receiving, at the application layer of the block storage system and through the system call interface of the interface layer of the block storage system, one or more communications that identify one or more transformation actions to perform on the plurality of data blocks, the one or more transformation actions including one or more of;

a compression type to compress each data block of the plurality of data blocks by; and

an encryption type to encrypt each data block of the plurality of data blocks, wherein an encryption key associated with the encryption type is retrieved locally from the block storage system; and

transforming each data block of the plurality of data blocks to form a plurality of transformed blocks by performing the one or more transformation actions on each data block of the plurality of data blocks; and

generating, at the transactional object layer of the block storage system, a plurality of metadata blocks corresponding to the plurality of transformed blocks, the plurality of metadata blocks being configured to hierarchically point to lower-level blocks associated with the file and thereby correspond to at least part of a tree hierarchy for the file, wherein;

each metadata block of the plurality of metadata blocks includes one or more address pointers, each address pointer of the one or more address pointers being pointed to a transformed block of the plurality of transformed blocks or to a metadata block of the plurality of metadata blocks;

each transformed block of the plurality of transformed blocks being pointed to by at least one metadata block of the plurality of metadata blocks;

the plurality of metadata blocks includes a root block that is positioned at a top of the tree hierarchy for the file and one or more non-root metadata blocks; and

each non-root metadata block of the plurality of metadata blocks being pointed to by at least one metadata block of the of the plurality of metadata blocks of the tree hierarchy of the file;

causing a set of cloud storage objects to be stored in the cloud object store by transmitting the plurality of transformed blocks and the plurality of metadata blocks to a hybrid cloud storage system, the hybrid cloud storage system managing data storage in the cloud object store;

transmitting, to the hybrid cloud storage system, one or more second requests for a set of addresses, each address of the set of addresses corresponding to a transformed block of the plurality of transformed blocks or a metadata block of the plurality of metadata blocks; and

receiving, from the hybrid cloud storage system, one or more responses to the one or more second requests, each response of the one or more responses identifying an address corresponding to a transformed block of the plurality of transformed blocks or a metadata block of the plurality of metadata blocks, the address identifying a storage location in the cloud object store.

9. The one or more non-transitory tangible computer-readable storage media of claim 8, wherein the compression type includes one of:

LZ4, LZJB, GZIP, and ZLE.

10. The one or more non-transitory tangible computer-readable storage media of claim 8, wherein the encryption type includes advanced encryption standard (AES) with key lengths including 128, 192, and 256.

11. The one or more non-transitory tangible computer-readable storage media of claim 8, wherein the encryption key is randomly generated.

12. The one or more non-transitory tangible computer-readable storage media of claim 8, wherein a wrapping key is used to decrypt the encryption key.

13. The one or more non-transitory tangible computer-readable storage media of claim 12, wherein the wrapping key is a stored passphrase or an AES key.

14. The one or more non-transitory tangible computer-readable storage media of claim 8, wherein a compression ratio is 2 to 1.

15. A system for transforming blocks on a cloud object store that is remote from a block storage system comprising:

one or more processors;

a non-transitory computer-readable medium coupled to the one or more processors and storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors, to perform operations including;

receiving, at an application layer of the block storage system and through a system call interface of an interface layer of the block storage system, a first request to store or modify a file, the first request including file data;

generating, at a transactional object layer of the block storage system, a plurality of data blocks, each data block of the plurality of data blocks corresponding to at least a portion of the file data;

receiving, at the application layer of the block storage system and through the system call interface of the interface layer of the block storage system, one or more communications that identify one or more transformation actions to perform on the plurality of data blocks, the one or more transformation actions including one or more of;

a compression type to compress each data block of the plurality of data blocks by; and

an encryption type to encrypt each data block of the plurality of data blocks, wherein an encryption key associated with the encryption type is retrieved locally from the block storage system; and

transforming each data block of the plurality of data blocks to form a plurality of transformed blocks by performing the one or more transformation actions on each data block of the plurality of data blocks;

generating, at the transactional object layer of the block storage system, a plurality of metadata blocks corresponding to the plurality of transformed blocks, the plurality of metadata blocks being configured to hierarchically point to lower-level blocks associated with the file and thereby correspond to at least part of a tree hierarchy for the file, wherein;

each metadata block of the plurality of metadata blocks includes one or more address pointers, each address pointer of the one or more address pointers being pointed to a transformed block of the plurality of transformed blocks or to a metadata block of the plurality of metadata blocks;

each transformed block of the plurality of transformed blocks being pointed to by at least one metadata block of the plurality of metadata blocks;

the plurality of metadata blocks includes a root block that is positioned at a top of the tree hierarchy for the file and one or more non-root metadata blocks; and

each non-root metadata block of the plurality of metadata blocks being pointed to by at least one metadata block of the plurality of metadata blocks of the tree hierarchy of the file;

causing a set of cloud storage objects to be stored in the cloud object store by transmitting the plurality of transformed blocks and the plurality of metadata blocks to a hybrid cloud storage system, the hybrid cloud storage system managing data storage in the cloud object store;

transmitting, to the hybrid cloud storage system, one or more second requests for a set of addresses, each address of the set of addresses corresponding to a transformed block of the plurality of transformed blocks or a metadata block of the plurality of metadata blocks; and

receiving, from the hybrid cloud storage system, one or more responses to the one or more second requests, each response of the one or more responses identifying an address corresponding to a transformed block of the plurality of transformed blocks or a metadata block of the plurality of metadata blocks, the address identifying a storage location in the cloud object store.

16. The system of claim 15, wherein the compression type includes one of:

LZ4, LZJB, GZIP, and ZLE.

17. The system of claim 15, wherein the encryption type includes advanced encryption standard (AES) with key lengths including 128, 192, and 256.

18. The system of claim 15, wherein the encryption key is randomly generated.

19. The system of claim 15, wherein a wrapping key is used to decrypt the encryption key.

20. The system of claim 19, wherein the wrapping key is a stored passphrase or an AES key.