Gateway for cloud-based secure storage

• US 10,031,679 B2
• Filed: 11/23/2015
• Issued: 07/24/2018
• Est. Priority Date: 11/21/2014
• Status: Active Grant

• Current Assignee: Security First Innovations, LLC
• Original Assignee: Security First Corp
• Inventors: O'Hare, Mark S., Orsini, Rick L.
• Primary Examiner(s): Nguyen, Trong
• Application Number:
  US14/949,370
• Publication Number:
  US 20160150047A1
• Time in Patent Office: 974 Days
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2005 using redundant communication controllers
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2212/1052 Security improvement
• CPC Class Codes: G06F 2212/224 Disk storage
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/061 : for key exchange, e.g. in peer-to-peer networks cryptographic mechanisms or cryptographic arrangements for key agreement H04L9/0838
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 9/3297 : involving time stamps, e.g. generation of time stamps

Abstract

The systems and methods disclosed herein transparently provide an improved scalable cloud-based dynamically adjustable or configurable storage volume. In one aspect, a gateway provides a dynamically or configurably adjustable storage volume, including a local cache. The storage volume may be transparently adjusted for the amount of data that needs to be stored using available local or cloud-based storage. The gateway may use caching techniques and block clustering to provide gains in access latency compared to existing gateway systems, while providing scalable off-premises storage.

Claims

1. A method for providing improved scalable cloud-based storage to a client computer system, the method comprising:

receiving, using a programmed hardware processor of a gateway, a data storage request associated with a data file, wherein the data storage request is generated by an application running on the client computer system;

provisioning, using the programmed hardware processor, a storage volume for the client computer system as a virtual disk, wherein locations that comprise the provisioned storage volume are hidden from the client computer system, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the remote storage devices and one or more local storage devices;

including, using the programmed hardware processor, the data file in one or more cluster blocks, each cluster block of the one or more cluster blocks (1) comprising a plurality of data blocks that are to be written simultaneously to the cloud library over a plurality of channels and (2) has a predetermined size determined based on one or more criteria selected from the group consisting of cloud-access latency, a total size of the data file, a file type of the data file, and a total capacity of the local cache memory;

causing, using the programmed hardware processor, the one or more cluster blocks to be stored in the local cache memory;

controlling, using the programmed hardware processor, a data transfer rate to the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache memory, thereby avoiding storage overflow of the local cache memory; and

in response to detecting a change in the one or more cluster blocks, causing, using the programmed hardware processor, the one or more cluster blocks to be transparently stored to the remote storage devices using the plurality of channels.

2. The method of claim 1, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

3. The method of claim 1, wherein the including the data file in the one or more cluster blocks comprises:

generating, using a device mapper module, one or more sequential identifiers for a subset of data blocks generated from the data file; and

updating a block map for the one or more cluster blocks to associate the one or more sequential identifiers with the data file.

4. The method of claim 1, further comprising removing a respective cluster block from the local cache memory in response to detecting that the respective cluster block is stored to the cloud library.

5. The method of claim 1, further comprising removing a selected cluster block from the local cache memory in response to detecting that an available storage space of the local cache memory is less than or equal to a predetermined threshold.

6. The method of claim 5, further comprising selecting the selected cluster block for removal in response to determining that the selected cluster block is a least recently used cluster block in the local cache memory.

7. The method of claim 5, further comprising selecting the selected cluster block for removal in response to determining that the selected cluster block is a least frequently used cluster block in the local cache memory.

8. The method of claim 1, further comprising transparently increasing a total capacity of the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache memory.

9. The method of claim 1, wherein each cluster block of the one or more cluster blocks comprises a plurality of data blocks obtained from a plurality of separate data files.

10. The method of claim 1, wherein the remote storage devices are geographically separated.

11. The method of claim 1, further comprising updating an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting a change in the respective one of the one or more cluster blocks in the local cache memory.

12. The method of claim 1, further comprising updating an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting that the respective one of the one or more cluster blocks is stored to the cloud library.

13. The method of claim 1, wherein:

the causing the one or more cluster blocks to be stored in the local cache memory further comprises applying a first cryptographic operation to the one or more cluster blocks; and

the causing the one or more cluster blocks to be transparently stored to the remote storage devices further comprises applying a second cryptographic operation to the one or more cluster blocks.

14. The method of claim 13, wherein:

the applying the first cryptographic operation comprises encrypting the one or more cluster blocks using a first encryption key; and

the applying the second cryptographic operation comprises encrypting the one or more cluster blocks using a second encryption key different from the first encryption key.

15. The method of claim 14, further comprising storing the first encryption key, the second encryption key, or both in a separate storage location from the one or more cluster blocks.

16. The method of claim 1, wherein the causing the one or more cluster blocks to be transparently stored to the remote storage devices comprises causing the one or more cluster blocks to be distributed in a plurality of data shares located in the remote storage devices, each share of the plurality of data shares comprising a portion of each cluster block in a subset of the one or more cluster blocks.

17. The method of claim 16, wherein the causing the one or more cluster blocks to be distributed in the plurality of data shares comprises causing each cluster block of the one or more cluster blocks to be shuffled into a single data share of the plurality of data shares.

18. The method of claim 16, wherein the causing the one or more cluster blocks to be distributed in a plurality of data shares comprises splitting each cluster block of the one or more cluster blocks into a plurality of secondary data units and causing each secondary data unit of the plurality of secondary data units to be placed into one of the plurality of data shares, wherein each cluster block of the one or more cluster blocks is restorable by recombining a subset less than all of the plurality of secondary data units from the plurality of data shares.

19. The method of claim 18, wherein the plurality of secondary data units are placed into the plurality of data shares using a key generated based on a random or pseudo-random number.

20. A gateway system for providing improved scalable cloud-based storage to a client computer system, the gateway system comprising:

a programmed hardware processor configured to execute a program that causes;

a user communication interface to receive a data storage request associated with a data file, wherein the data storage request is generated by an application running on the client computer system;

a storage volume manager, communicatively coupled to the user communication interface, to provision a storage volume for the client computer system as a virtual disk, wherein locations that comprise the provisioned storage volume are hidden from the client computer system, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the remote storage devices and one or more local storage devices;

a transparent data storage layer coupled to the storage volume manager and configured to;

include the data file in one or more cluster blocks, each cluster block of the one or more cluster blocks (1) comprising a plurality of data blocks that are grouped to be written simultaneously to the cloud library over a plurality of channels and (2) has a predetermined size determined based on one or more criteria selected from the group consisting of cloud-access latency, a total size of the data file, a file type of the data file, and a total capacity of the local cache memory, and

cause the one or more cluster blocks to be stored in the local cache memory;

control a data transfer rate to the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache memory, thereby avoiding storage overflow of the local cache memory; and

a cloud interface module configured to cause the one or more cluster blocks to be transparently stored to the remote storage devices using the plurality of channels based on an upload status indicator associated with each cluster block of the one or more cluster blocks, in response to detecting a change in the one or more cluster blocks.

21. The gateway system of claim 20, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

22. The gateway system of claim 20, wherein the transparent data storage layer is further configured to:

generate, using a device mapper module, one or more sequential identifiers for a subset of data blocks generated from the data file; and

update a block map for the one or more cluster blocks to associate the one or more sequential identifiers with the data file.

23. The gateway system of claim 20, wherein the transparent data storage layer is further configured to remove a respective cluster block from the local cache memory in response to detecting that the respective cluster block is stored to the cloud library.

24. The gateway system of claim 20, wherein the transparent data storage layer is further configured to remove a selected cluster block from the local cache memory in response to detecting that an available storage space of the local cache memory is less than or equal to a predetermined threshold.

25. The gateway system of claim 24, wherein the transparent data storage layer is further configured to select the selected cluster block for removal in response to determining that the selected cluster block is a least recently used cluster block in the local cache memory.

26. The gateway system of claim 24, wherein the transparent data storage layer is further configured to select the selected cluster block for removal in response to determining that the selected cluster block is a least frequently used cluster block in the local cache memory.

27. The gateway system of claim 20, wherein the transparent data storage layer is further configured to transparently increase a total capacity of the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache memory.

28. The gateway system of claim 20, wherein each cluster block of the one or more cluster blocks comprises a plurality of data blocks obtained from a plurality of separate data files.

29. The gateway system of claim 20, wherein the remote storage devices are geographically separated.

30. The gateway system of claim 20, wherein the transparent data storage layer is further configured to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting a change in the respective one of the one or more cluster blocks in the local cache memory.

31. The gateway system of claim 20, wherein the transparent data storage layer is further configured to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting that the respective one of the one or more cluster blocks is stored to the cloud library.

32. The gateway system of claim 20, wherein:

the transparent data storage layer is further configured to apply a first cryptographic operation to the one or more cluster blocks to be stored in the local cache memory; and

the cloud interface module is further configured to apply a second cryptographic operation to the one or more cluster blocks to be transparently stored to the remote storage devices.

33. The gateway system of claim 32, wherein:

the applying the first cryptographic operation comprises encrypting the one or more cluster blocks using a first encryption key; and

the applying the second cryptographic operation comprises encrypting the one or more cluster blocks using a second encryption key different from the first encryption key.

34. The gateway system of claim 33, wherein the transparent data storage layer or the cloud interface module is configured to store the first encryption key, the second encryption key, or both in a separate storage location from the one or more cluster blocks.

35. The gateway system of claim 20, wherein the cloud interface module is configured to cause the one or more cluster blocks to be transparently stored to the remote storage devices by causing the one or more cluster blocks to be distributed in a plurality of data shares located in the remote storage devices, each share of the plurality of data shares comprising a portion of each cluster block in a subset of the one or more cluster blocks.

36. The gateway system of claim 35, wherein the cloud interface module is configured to cause the one or more cluster blocks to be distributed in the plurality of data shares by causing each cluster block of the one or more cluster blocks to be shuffled into a single data share of the plurality of data shares.

37. The gateway system of claim 35, wherein the cloud interface module is configured to cause the one or more cluster blocks to be distributed in the plurality of data shares by splitting each cluster block of the one or more cluster blocks into a plurality of secondary data units and causing each secondary data unit of the plurality of secondary data units to be placed into one of the plurality of data shares, wherein each cluster block of the one or more cluster blocks is restorable by recombining a subset less than all of the plurality of secondary data units from the plurality of data shares.

38. The gateway system of claim 37, wherein the plurality of secondary data units are placed into the plurality of data shares using a key generated based on a random or pseudo-random number.

39. A non-transitory computer-readable medium storing processor-executable instructions for providing improved scalable cloud-based storage to a client computer system, comprising instructions executable by one or more computer processors of a gateway to:

receive a data storage request associated with a data file, wherein the data storage request is generated by an application running on the client computer system;

provision a storage volume for the client computer system as a virtual disk, wherein locations that comprise the provisioned storage volume are hidden from the client computer system, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the remote storage devices and one or more local storage devices;

include the data file in one or more cluster blocks, each cluster block of the one or more cluster blocks (1) comprising a plurality of data blocks that are grouped to be written simultaneously to the cloud library over a plurality of channels and (2) has a predetermined size determined based on one or more criteria selected from the group consisting of cloud-access latency, a total size of the data file, a file type of the data file, and a total capacity of the local cache memory;

cause the one or more cluster blocks to be stored in the local cache memory;

control a data transfer rate to the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache memory, thereby avoiding storage overflow of the local cache memory; and

in response to detecting a change in the one or more cluster blocks, cause the one or more cluster blocks to be transparently stored to the remote storage devices using the plurality of channels.

40. The non-transitory computer-readable medium of claim 39, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

41. The non-transitory computer-readable medium of claim 39, further comprising instructions executable by the one or more computer processors to:

generate one or more sequential identifiers for a subset of data blocks generated from the data file; and

update a block map for the one or more cluster blocks to associate the one or more sequential identifiers with the data file.

42. The non-transitory computer-readable medium of claim 39, further comprising instructions executable by the one or more computer processors to remove a respective cluster block from the local cache memory in response to detecting that the respective cluster block is stored to the cloud library.

43. The non-transitory computer-readable medium of claim 39, further comprising instructions executable by the one or more computer processors to remove a selected cluster block from the local cache memory in response to detecting that an available storage space of the local cache memory is less than or equal to a predetermined threshold.

44. The non-transitory computer-readable medium of claim 43, further comprising instructions executable by the one or more computer processors to select the selected cluster block for removal in response to determining that the selected cluster block is a least recently used cluster block in the local cache memory.

45. The non-transitory computer-readable medium of claim 43, further comprising instructions executable by the one or more computer processors to select the selected cluster block for removal in response to determining that the selected cluster block is a least frequently used cluster block in the local cache memory.

46. The non-transitory computer-readable medium of claim 39, further comprising instructions executable by the one or more computer processors to transparently increase a total capacity of the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache memory.

47. The non-transitory computer-readable medium of claim 39, wherein each cluster block of the one or more cluster blocks comprises a plurality of data blocks obtained from a plurality of separate data files.

48. The non-transitory computer-readable medium of claim 39, wherein the remote storage devices are geographically separated.

49. The non-transitory computer-readable medium of claim 39, further comprising instructions executable by the one or more computer processors to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting a change in the respective one of the one or more cluster blocks in the local cache memory.

50. The non-transitory computer-readable medium of claim 39, further comprising instructions executable by the one or more computer processors to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting that the respective one of the one or more cluster blocks is stored to the cloud library.

51. The non-transitory computer-readable medium of claim 39, wherein:

the instructions executable by the one or more computer processors to cause the one or more cluster blocks to be stored in the local cache memory further comprise instructions executable by the one or more computer processors to apply a first cryptographic operation to the one or more cluster blocks; and

the instructions executable by the one or more computer processors to cause the one or more cluster blocks to be transparently stored to the remote storage devices further comprise instructions executable by the one or more computer processors to apply a second cryptographic operation to the one or more cluster blocks.

52. The non-transitory computer-readable medium of claim 51, wherein:

the instructions executable by the one or more computer processors to apply the first cryptographic operation comprise instructions executable by the one or more computer processors to encrypt the one or more cluster blocks using a first encryption key; and

the instructions executable by the one or more computer processors to apply the second cryptographic operation comprise instructions executable by the one or more computer processors to encrypt the one or more cluster blocks using a second encryption key different from the first encryption key.

53. The non-transitory computer-readable medium of claim 52, further comprising instructions executable by the one or more computer processors to store the first encryption key, the second encryption key, or both in a separate storage location from the one or more cluster blocks.

54. The non-transitory computer-readable medium of claim 39, wherein the instructions executable by the one or more computer processors to cause the one or more cluster blocks to be transparently stored to the remote storage devices comprise instructions executable by the one or more computer processors to cause the one or more cluster blocks to be distributed in a plurality of data shares located in the remote storage devices, each share of the plurality of data shares comprising a portion of each cluster block in a subset of the one or more cluster blocks.

55. The non-transitory computer-readable medium of claim 54, wherein the instructions executable by the one or more computer processors to cause the one or more cluster blocks to be distributed in a plurality of data shares comprise instructions executable by the one or more computer processors to cause each cluster block of the one or more cluster blocks to be shuffled into a single data share of the plurality of data shares.

56. The non-transitory computer-readable medium of claim 54, wherein the instructions executable by the one or more computer processors to cause the one or more cluster blocks to be distributed in a plurality of data shares comprise instructions executable by the one or more computer processors to split each cluster block of the one or more cluster blocks into a plurality of secondary data units and causing each secondary data unit of the plurality of secondary data units to be placed into one of the plurality of data shares, wherein each cluster block of the one or more cluster blocks is restorable by recombining a subset less than all of the plurality of secondary data units from the plurality of data shares.

57. The non-transitory computer-readable medium of claim 56, wherein the plurality of secondary data units are placed into the plurality of data shares using a key generated based on a random or pseudo-random number.