GATEWAY FOR CLOUD-BASED SECURE STORAGE

• US 20160150047A1
• Filed: 11/23/2015
• Published: 05/26/2016
• Est. Priority Date: 11/21/2014
• Status: Active Grant

• Current Assignee: Security First Innovations, LLC
• Original Assignee: Security First Corp
• Inventors: O'Hare, Mark S., Orsini, Rick L.
• Granted Patent: US 10,031,679 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2005 using redundant communication controllers
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2212/1052 Security improvement
• CPC Class Codes: G06F 2212/224 Disk storage
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/061 : for key exchange, e.g. in peer-to-peer networks cryptographic mechanisms or cryptographic arrangements for key agreement H04L9/0838
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 9/3297 : involving time stamps, e.g. generation of time stamps

Abstract

The systems and methods disclosed herein transparently provide an improved scalable cloud-based dynamically adjustable or configurable storage volume. In one aspect, a gateway provides a dynamically or configurably adjustable storage volume, including a local cache. The storage volume may be transparently adjusted for the amount of data that needs to be stored using available local or cloud-based storage. The gateway may use caching techniques and block clustering to provide gains in access latency compared to existing gateway systems, while providing scalable off-premises storage.

Claims

1. A method for providing improved scalable cloud-based storage to a client computer system, the method comprising:

receiving, using a programmed hardware processor, a data storage request associated with a data file, wherein the data storage request is generated by an application running on the client computer system;

provisioning a storage volume for the client computer system, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising one or more remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the one or more remote storage devices and one or more local storage devices;

including the data file in one or more cluster blocks, each cluster block comprising a plurality of data blocks that are to be written simultaneously to the cloud library;

causing the one or more cluster blocks to be stored in the local cache memory; and

in response to detecting a change in the one or more cluster blocks, causing the one or more cluster blocks to be transparently stored to the one or more remote storage devices.

2. The method of claim 1, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

3. The method of claim 1, wherein including the data file in one or more cluster blocks comprises:

generating, using a device mapper module, one or more sequential identifiers for a subset of data blocks generated from the data file; and

updating a block map for the one or more cluster blocks to associate the sequential identifiers with the data file.

4. The method of claim 1, further comprising removing the respective cluster block from the local cache memory in response to detecting that the respective cluster block is stored to the cloud library.

5. The method of claim 1, further comprising removing a selected cluster block from the local cache memory in response to detecting that an available storage space of the local cache memory is less than or equal to a predetermined threshold.

6. The method of claim 5, further comprising selecting the selected cluster block for removal in response to determining that the selected cluster block is the least recently used cluster block in the local cache memory.

7. The method of claim 5, further comprising selecting the selected cluster block for removal in response to determining that the selected cluster block is the least frequently used cluster block in the local cache memory.

8. The method of claim 1, further comprising transparently increasing a total capacity of the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache.

9. The method of claim 1, further comprising controlling a data transfer rate to the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache, thereby avoiding storage overflow of the local cache memory.

10. The method of claim 1, wherein each cluster block has a predetermined size determined based on one or more criteria selected from the group consisting of cloud-access latency, a total size of the data file, a file type of the data file, and a total capacity of the local cache memory.

11. The method of claim 1, wherein each cluster block comprises a plurality of data blocks obtained from a plurality of separate data files.

12. The method of claim 1, wherein the one or more remote storage devices are geographically separated.

13. The method of claim 1, further comprising updating an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting a change in the respective cluster block in the local cache memory.

14. The method of claim 1, further comprising updating an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting that the respective cluster block is stored to the cloud library.

15. The method of claim 1, wherein:

causing the one or more cluster blocks to be stored in the local cache memory further comprises applying a first cryptographic operation to the one or more cluster blocks; and

causing the one or more cluster blocks to be transparently stored to the one or more remote storage devices further comprises applying a second cryptographic operation to the one or more cluster blocks.

16. The method of claim 15, wherein:

applying the first cryptographic operation comprises encrypting the one or more cluster blocks using a first encryption key; and

applying the second cryptographic operation comprises encrypting the one or more cluster blocks using a second encryption key different from the first encryption key.

17. The method of claim 16, further comprising storing the first encryption key, the second encryption key, or both in a separate storage location from the one or more cluster blocks.

18. The method of claim 1, wherein causing the one or more cluster blocks to be transparently stored to the one or more remote storage devices comprises causing the one or more cluster blocks to be distributed in a plurality of data shares located in the one or more remote storage devices, each share comprising a portion of each cluster block in a subset of the cluster blocks.

19. The method of claim 18, wherein causing each share to be distributed in a plurality of data shares comprises causing each cluster block to be shuffled into a single data share of the plurality of data shares.

20. The method of claim 18, wherein causing each share to be distributed in a plurality of data shares comprises splitting each cluster block into a plurality of secondary data units and causing each secondary data unit to be placed into one of the plurality of data shares, wherein each cluster block is restorable by recombining a subset less than all of the secondary data units from the plurality of data shares.

21. The method of claim 20, wherein the secondary data units are placed into the plurality of data shares using on a key generated based on a random or pseudo-random number.

22. A method for providing improved scalable cloud-based storage to a client computer system, the method comprising:

presenting to the client computer system a virtual disk associated with a provisioned storage volume that stores one or more data files, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising one or more remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the one or more remote storage devices and one or more local storage devices;

receiving, from the client computer system, a request to access a selected data file from the one or more data files;

identifying one or more cluster blocks associated with the selected data file based at least in part on a cluster block map that relates information associated with the selected data file to information maintained by the cluster block map for the cluster blocks;

in response to determining that the one or more cluster blocks are stored in the local cache memory, transparently retrieving the selected data file from the one or more cluster blocks;

in response to determining that at least one of the one or more cluster blocks is missing from the local cache memory, transparently retrieving the at least one cluster block from a storage location in the cloud library, wherein the storage location of the at least one cluster block is hidden from the client computer system; and

providing the selected data file to the client computer system.

23. The method of claim 22, further comprising updating a usage counter associated with the one or more cluster blocks.

24. The method of claim 22, wherein retrieving the at least one cluster block from the cloud library comprises recombining a threshold number of secondary data units of the at least cluster block, the threshold number being less than all of the secondary data units of the cluster block.

25. The method of claim 24, wherein the secondary data units of the at least one cluster block are stored ion data shares located at geographically separated locations.

26. The method of claim 24, wherein the at least one cluster block comprises encrypted data.

27. The method of claim 26, wherein said recombining is performed without decrypting the encrypted data in the at least one cluster block.

28. The method of claim 22, further comprising storing the at least one cluster block in the local cache memory in response to determining that the at least one cluster block is missing from the local cache memory.

29. The method of claim 22, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

30. A system for providing improved scalable cloud-based storage to a client computer system, the system comprising:

a user communication interface to receive a data storage request associated with a data file, wherein the data storage request is generated by an application running on the client computer system;

a storage volume manager, communicatively coupled to the user communication interface, to provision a storage volume for the client computer system, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising one or more remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the one or more remote storage devices and one or more local storage devices;

a transparent data storage layer coupled to the storage volume manager and configured to;

include the data file in one or more cluster blocks, each cluster block comprising a plurality of data blocks that are grouped to be written simultaneously to the cloud library, and

cause the one or more cluster blocks to be stored in the local cache memory; and

a cloud interface module configured to cause the one or more cluster blocks to be transparently stored to the one or more remote storage devices based on the upload status indicator associated with each cluster block, in response to detecting a change in the one or more cluster blocks.

31. The system of claim 30, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

32. The system of claim 30, wherein the transparent data storage layer is further configured to:

generate, using a device mapper module, one or more sequential identifiers for a subset of data blocks generated from the data file; and

update a block map for the one or more cluster blocks to associate the sequential identifiers with the data file.

33. The system of claim 30, wherein the transparent data storage layer is further configured to remove the respective cluster block from the local cache memory in response to detecting that the respective cluster block is stored to the cloud library.

34. The system of claim 30, wherein the transparent data storage layer is further configured to remove a selected cluster block from the local cache memory in response to detecting that an available storage space of the local cache memory is less than or equal to a predetermined threshold.

35. The system of claim 34, wherein the transparent data storage layer is further configured to select the selected cluster block for removal in response to determining that the selected cluster block is the least recently used cluster block in the local cache memory.

36. The system of claim 34, wherein the transparent data storage layer is further configured to select the selected cluster block for removal in response to determining that the selected cluster block is the least frequently used cluster block in the local cache memory.

37. The system of claim 30, wherein the transparent data storage layer is further configured to transparently increase a total capacity of the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache.

38. The system of claim 30, wherein the transparent data storage layer is further configured to control a data transfer rate to the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache, thereby avoiding storage overflow of the local cache memory.

39. The system of claim 30, wherein each cluster block has a predetermined size determined based on one or more criteria selected from the group consisting of cloud-access latency, a total size of the data file, a file type of the data file, and a total capacity of the local cache memory.

40. The system of claim 30, wherein each cluster block comprises a plurality of data blocks obtained from a plurality of separate data files.

41. The system of claim 30, wherein the one or more remote storage devices are geographically separated.

42. The system of claim 30, wherein the transparent data storage layer is further configured to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting a change in the respective cluster block in the local cache memory.

43. The system of claim 30, wherein the transparent data storage layer is further configured to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting that the respective cluster block is stored to the cloud library.

44. The system of claim 30, wherein:

the transparent data storage layer is further configured to apply a first cryptographic operation to the one or more cluster blocks to be stored in the local cache memory; and

the cloud interface module is further configured to apply a second cryptographic operation to the one or more cluster blocks to be transparently stored to the one or more remote storage devices.

45. The system of claim 44, wherein:

applying the first cryptographic operation comprises encrypting the one or more cluster blocks using a first encryption key; and

applying the second cryptographic operation comprises encrypting the one or more cluster blocks using a second encryption key different from the first encryption key.

46. The system of claim 45, wherein the transparent data storage layer or the cloud interface module is configured to store the first encryption key, the second encryption key, or both in a separate storage location from the one or more cluster blocks.

47. The system of claim 30, wherein the cloud interface module is configured to cause the one or more cluster blocks to be transparently stored to the one or more remote storage devices by causing the one or more cluster blocks to be distributed in a plurality of data shares located in the one or more remote storage devices, each share comprising a portion of each cluster block in a subset of the cluster blocks.

48. The system of claim 47, wherein the cloud interface module is configured to cause each share to be distributed in a plurality of data shares by causing each cluster block to be shuffled into a single data share of the plurality of data shares.

49. The system of claim 47, wherein the cloud interface module is configured to cause each share to be distributed in a plurality of data shares by splitting each cluster block into a plurality of secondary data units and causing each secondary data unit to be placed into one of the plurality of data shares, wherein each cluster block is restorable by recombining a subset less than all of the secondary data units from the plurality of data shares.

50. The system of claim 49, wherein the secondary data units are placed into the plurality of data shares using on a key generated based on a random or pseudo-random number.

51. A non-transitory computer-readable medium storing processor-executable instructions for providing improved scalable cloud-based storage to a client computer system, comprising instructions executable by one or more computer processors to:

receive a data storage request associated with a data file, wherein the data storage request is generated by an application running on the client computer system;

receive provision a storage volume for the client computer system, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising one or more remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the one or more remote storage devices and one or more local storage devices;

include the data file in one or more cluster blocks, each cluster block comprising a plurality of data blocks that are grouped to be written simultaneously to the cloud library;

cause the one or more cluster blocks to be stored in the local cache memory; and

in response to detecting a change in the one or more cluster blocks, cause the one or more cluster blocks to be transparently stored to the one or more remote storage devices.

52. The non-transitory computer-readable medium of claim 51, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

53. The non-transitory computer-readable medium of claim 51, further comprising instructions executable by the one or more computer processors to:

generate, using a device mapper module, one or more sequential identifiers for a subset of data blocks generated from the data file; and

update a block map for the one or more cluster blocks to associate the sequential identifiers with the data file.

54. The non-transitory computer-readable medium of claim 51, further comprising instructions executable by the one or more computer processors to remove the respective cluster block from the local cache memory in response to detecting that the respective cluster block is stored to the cloud library.

55. The non-transitory computer-readable medium of claim 51, further comprising instructions executable by the one or more computer processors to remove a selected cluster block from the local cache memory in response to detecting that an available storage space of the local cache memory is less than or equal to a predetermined threshold.

56. The non-transitory computer-readable medium of claim 55, further comprising instructions executable by the one or more computer processors to select the selected cluster block for removal in response to determining that the selected cluster block is the least recently used cluster block in the local cache memory.

57. The non-transitory computer-readable medium of claim 55, further comprising instructions executable by the one or more computer processors to select the selected cluster block for removal in response to determining that the selected cluster block is the least frequently used cluster block in the local cache memory.

58. The non-transitory computer-readable medium of claim 51, further comprising instructions executable by the one or more computer processors to transparently increase a total capacity of the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache.

59. The non-transitory computer-readable medium of claim 51, further comprising instructions executable by the one or more computer processors to control a data transfer rate to the local cache memory in response to detecting that a file size of the data file exceeds an available storage capacity of the local cache, thereby avoiding storage overflow of the local cache memory.

60. The non-transitory computer-readable medium of claim 51, wherein each cluster block has a predetermined size determined based on one or more criteria selected from the group consisting of cloud-access latency, a total size of the data file, a file type of the data file, and a total capacity of the local cache memory.

61. The non-transitory computer-readable medium of claim 51, wherein each cluster block comprises a plurality of data blocks obtained from a plurality of separate data files.

62. The non-transitory computer-readable medium of claim 51, wherein the one or more remote storage devices are geographically separated.

63. The non-transitory computer-readable medium of claim 51, further comprising instructions executable by the one or more computer processors to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting a change in the respective cluster block in the local cache memory.

64. The non-transitory computer-readable medium of claim 51, further comprising instructions executable by the one or more computer processors to update an upload status indicator associated with a respective one of the one or more cluster blocks in response to detecting that the respective cluster block is stored to the cloud library.

65. The non-transitory computer-readable medium of claim 51, wherein:

instructions executable by the one or more computer processors to cause the one or more cluster blocks to be stored in the local cache memory further comprises instructions executable by the one or more computer processors to apply a first cryptographic operation to the one or more cluster blocks; and

instructions executable by the one or more computer processors to cause the one or more cluster blocks to be transparently stored to the one or more remote storage devices further comprises instructions executable by the one or more computer processors to apply a second cryptographic operation to the one or more cluster blocks.

66. The non-transitory computer-readable medium of claim 65, wherein:

instructions executable by the one or more computer processors to apply the first cryptographic operation comprises instructions executable by the one or more computer processors to encrypt the one or more cluster blocks using a first encryption key; and

instructions executable by the one or more computer processors to apply the second cryptographic operation comprises instructions executable by the one or more computer processors to encrypt the one or more cluster blocks using a second encryption key different from the first encryption key.

67. The non-transitory computer-readable medium of claim 66, further comprising instructions executable by the one or more computer processors to store the first encryption key, the second encryption key, or both in a separate storage location from the one or more cluster blocks.

68. The non-transitory computer-readable medium of claim 51, wherein instructions executable by the one or more computer processors to cause the one or more cluster blocks to be transparently stored to the one or more remote storage devices comprises instructions executable by the one or more computer processors to cause the one or more cluster blocks to be distributed in a plurality of data shares located in the one or more remote storage devices, each share comprising a portion of each cluster block in a subset of the cluster blocks.

69. The non-transitory computer-readable medium of claim 68, wherein instructions executable by the one or more computer processors to cause each share to be distributed in a plurality of data shares comprises instructions executable by the one or more computer processors to cause each cluster block to be shuffled into a single data share of the plurality of data shares.

70. The non-transitory computer-readable medium of claim 68, wherein instructions executable by the one or more computer processors to cause each share to be distributed in a plurality of data shares comprises instructions executable by the one or more computer processors to split each cluster block into a plurality of secondary data units and causing each secondary data unit to be placed into one of the plurality of data shares, wherein each cluster block is restorable by recombining a subset less than all of the secondary data units from the plurality of data shares.

71. The non-transitory computer-readable medium of claim 70, wherein the secondary data units are placed into the plurality of data shares using on a key generated based on a random or pseudo-random number.

72. A system for providing improved scalable cloud-based storage to a client computer system, the system comprising:

a storage volume manager to present to the client computer system a virtual disk associated with a provisioned storage volume that stores one or more data files, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising one or more remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the one or more remote storage devices and one or more local storage devices;

a user communication interface, communicatively coupled to the storage volume manager, to receive, from the client computer system, a request to access a selected data file from the one or more data files;

a transparent data storage layer, communicatively coupled to the storage volume manager and configured to;

identify one or more cluster blocks associated with the selected data file based at least in part on a cluster block map that relates information associated with the selected data file to information maintained by the cluster block map for the cluster blocks, and

in response to determining that the one or more cluster blocks are stored in the local cache memory, transparently retrieve the selected data file from the one or more cluster blocks;

a cloud interface communicatively coupled to the transparent data storage layer, and configured to;

in response to determining that at least one of the one or more cluster blocks is missing from the local cache memory, transparently retrieve the at least one cluster block from a storage location in the cloud library, wherein the storage location of the at least one cluster block is hidden from the client computer system, wherein the selected data file is provided to the client computer system via the user communication interface.

73. The system of claim 72, wherein the transparent data storage layer is further configured to update a usage counter associated with the one or more cluster blocks.

74. The system of claim 72, wherein the transparent data storage layer is further configured to recombine a threshold number of secondary data units of the at least cluster block, the threshold number being less than all of the secondary data units of the cluster block.

75. The system of claim 74, wherein the secondary data units of the at least one cluster block are stored ion data shares located at geographically separated locations.

76. The system of claim 72, wherein the at least one cluster block comprises encrypted data.

77. The system of claim 76, wherein the transparent data storage layer is further configured to perform said recombining without decrypting the encrypted data in the at least one cluster block.

78. The system of claim 72, wherein the transparent data storage layer is further configured to store the at least one cluster block in the local cache memory in response to determining that the at least one cluster block is missing from the local cache memory.

79. The system of claim 72, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.

80. A non-transitory computer-readable medium storing processor-executable instructions for providing improved scalable cloud-based storage to a client computer system, comprising instructions executable by one or more computer processors to:

present to the client computer system a virtual disk associated with a provisioned storage volume that stores one or more data files, wherein the provisioned storage volume (1) comprises a local cache memory communicatively coupled to the client computer system and a cloud library comprising one or more remote storage devices in one or more clouds, and (2) is dynamically or configurably adjustable by transparently including or excluding a subset of the one or more remote storage devices and one or more local storage devices;

receive, from the client computer system, a request to access a selected data file from the one or more data files;

identify one or more cluster blocks associated with the selected data file based at least in part on a cluster block map that relates information associated with the selected data file to information maintained by the cluster block map for the cluster blocks;

in response to determining that the one or more cluster blocks are stored in the local cache memory, transparently retrieve the selected data file from the one or more cluster blocks;

in response to determining that at least one of the one or more cluster blocks is missing from the local cache memory, transparently retrieve the at least one cluster block from a storage location in the cloud library, wherein the storage location of the at least one cluster block is hidden from the client computer system; and

provide the selected data file to the client computer system.

81. The non-transitory computer-readable medium of claim 80, further comprising instructions executable by the one or more computer processors to update a usage counter associated with the one or more cluster blocks.

82. The non-transitory computer-readable medium of claim 80, further comprising instructions executable by the one or more computer processors to retrieve the at least one cluster block from the cloud library by recombining a threshold number of secondary data units of the at least cluster block, the threshold number being less than all of the secondary data units of the cluster block.

83. The non-transitory computer-readable medium of claim 82, wherein the secondary data units of the at least one cluster block are stored ion data shares located at geographically separated locations.

84. The non-transitory computer-readable medium of claim 82, wherein the at least one cluster block comprises encrypted data.

85. The non-transitory computer-readable medium of claim 84, wherein said recombining is performed without decrypting the encrypted data in the at least one cluster block.

86. The non-transitory computer-readable medium of claim 80, further comprising instructions executable by the one or more computer processors to store the at least one cluster block in the local cache memory in response to determining that the at least one cluster block is missing from the local cache memory.

87. The non-transitory computer-readable medium of claim 80, wherein the local cache memory comprises non-volatile memory located within the client computer system or in a gateway server within a local network of the client computer system.