More Elegant Exastore Apparatus and Method of Operation

• US 20120017096A1
• Filed: 09/27/2011
• Published: 01/19/2012
• Est. Priority Date: 03/20/2009
• Status: Active Grant

• Current Assignee: Barracuda Networks Incorporated
• Original Assignee: Barracuda Networks Incorporated
• Inventors: SNIDER, LINDSAY EUGENE
• Granted Patent: US 8,788,831 B2
• Time in Patent Office: Days
• US Class Current: 713/189
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 21/60 Protecting data
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 9/0894 Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage
• CPC Class Codes: H04L 9/3236 using cryptographic hash functions

Abstract

An apparatus to scale for multiple petabyte backup in redundant locations. Workload is automatically shared among many servers by a characteristic derived from the content itself. Duplicate storage is eliminated by checking for the existence or absence of simple files and appending client identification to files shared among multiple subscribers. Replication depends on simple atomic file operations rather than use of tables or databases. An efficient storage method for much larger quantities of data than conventional services.

Claims

1. A computer-implemented method for control of an apparatus comprising a processor configured by instructions stored on a non-transitory computer readable medium, the method comprising:

receiving from a backup client of a plurality of backup clients distinguishable by a client identifier, a substantially unique characteristic derived from a shard;

determining when the shard is a previously stored shard or a new shard to be stored,

when the shard is a previously stored shard,

appending the client identifier of the backup client to an end of a previously stored uniquely named flat file.

2. The method of claim 1 wherein the substantially unique characteristic derived from a shard is a plurality of cryptographic hashes concatenated into a delimited string.

3. The method of claim 1 wherein a shard is a portion of a data file.

4. The method of claim 1 further comprising:

when the shard is a new shard to be stored,

requesting and storing the shard into an assigned server, and

creating and storing a new uniquely named flat file which contains an encryption key and the client identifier of the backup client

wherein the name of the new uniquely named flat file comprises the substantially unique characteristic derived from the shard.

5. The method of claim 4 wherein determining when the shard is a previously stored shard comprises determining when a flat file already exists which has a name comprising the substantially unique characteristic derived from the shard.

6. The method of claim 5 further comprising:

creating a token file to request replication of a shard and creating a token file to request replication of a uniquely named flat file wherein replication comprises storing to at least one non-transitory computer-readable medium on an other apparatus.

7. The method of claim 6 further comprising

deleting the token file when a shard of a uniquely named flat file has been successfully replicated.

8. The method of claim 7 further comprising

assigning the shard and the substantially named flat file to a server according to an except from the substantially unique characteristic derived from the shard.

9. The method of claim 8 wherein the except is a range of characters in the substantially unique characteristic.

10. The method of claim 9 further comprising

assigning the shard to a group of servers based on the client identifier of the backup client.

11. A method comprising,

receiving a shard and meta-data of a shard from a backup client,

assigning to a server based on the first n bytes of a key, and

appending to a file for the shard, the client id of the backup client.

12. The method of claim 11 wherein the meta-data comprises a key for the shard.

13. The method of claim 12 further comprising

creating a file containing the encryption key and the client id of the first client,

storing the file, and

storing the shard.

14. The method of claim 13 wherein the file containing the encryption key and the client ids of backup clients referencing the shard is named with the metadata of the shard.

15. The method of claim 14 wherein storing the file and storing the shard comprises

adding an action request file to the queue of the selected server and its replication server(s).

16. The method of claim 15 further comprising

assigning to a group of servers based on the backup client ID.

17. The method of claim 11 wherein a key is a hash computed on the shard.

18. An apparatus comprising

a file management circuit to determine if a uniquely named flat file exists,

when the uniquely named flat file exists, to append a client identifier to the end of the file,

when the uniquely named flat file does not exist, to create and store a flat file comprising an encryption key and a client identifier and to request and store a shard with the substantially unique characteristic as the name of the flat file.

19. The apparatus of claim 18 further comprising

a receiver circuit configured to receive from a backup client a substantially unique characteristic derived from a shard and a client identifier, to assign the shard to a server according to an excerpt from the substantially unique characteristic, to assign the shard to a group of servers according to the client identifier, and request and receive a shard when the shard has been determined not to have been previously stored.

20. The apparatus of claim 19 further comprising

a file write and delete circuit configured to

append an additional client identifier to an end of a stored uniquely named flat file,

create and store a new uniquely named flat file containing an encryption key and a client identifier when the flat file does not already exist,

create at least one token file to request replication of a shard or a uniquely named flat file, and

delete the token file when the requested replication has succeeded.