System and method of implementing an object storage infrastructure for cloud-based services

• US 9,135,269 B2
• Filed: 12/07/2012
• Issued: 09/15/2015
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Egnyte Incorporated
• Original Assignee: Egnyte Incorporated
• Inventors: Sankar, Krishna, Jassal, Amrit, Patel, Kalpesh, Sharma, Shishir, Shetty, Sachin
• Primary Examiner(s): Nguyen, Tammy
• Application Number:
  US13/708,040
• Publication Number:
  US 20140149794A1
• Time in Patent Office: 1,012 Days
• Field of Search: 709/219, 709/226, 455/521, 714/20
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/142 Reconfiguring to eliminate the error group management mechanisms in a peer-to-peer network H04L67/1044
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/141 : Setup of application sessions admission control or resource allocation in data switching networks H04L47/70

Abstract

A method for storing objects in an object storage system includes the steps of establishing a network connection with a client over an inter-network, receiving an upload request indicating an object to be uploaded by the client, selecting at least two storage nodes on which the object will be stored, receiving the object from the client via the network connection, and streaming the object to each of the selected storage nodes such that the object is stored on each of the selected storage nodes. The method can also include writing an object record associating the object and the selected storage nodes to a shard of an object database and generating a Universally Unique Identifier (UUID). The UUID indicates the shard and the object ID of the object record, such that the object record can be quickly retrieved. Object storage infrastructures are also disclosed.

Claims

1. An object storage system comprising:

a plurality of storage nodes for storing digital objects associated with clients;

at least one client application server being operative to

establish a network connection with a client over an inter-network,

receive an indication of an object to be uploaded by said client,

generate an upload request associated with said object, and

receive said object from said client via said inter-network;

at least one cloud application server being operative to

receive said upload request from said client application server,

generate a Universally Unique Identifier (UUID) associated with said object,

select at least two storage nodes from said plurality of storage nodes,

receive said object from said client application server, and

stream said object to each of said selected storage nodes such that said object is stored on each of said selected storage nodes;

a file system database storing information defining a virtual file system associated with said client and accessible to said client application server, said UUID being stored in said file system database; and

at least one cloud object server; and

wherein

said client application server is further operative to

receive a download command for said object from said client,

retrieve said UUID from said file system database, and

provide said UUID to said cloud object server.

2. The system of claim 1, wherein each of said plurality of storage nodes exposes a HyperText Transfer Protocol (HTTP) interface.

3. The system of claim 1, wherein the number of said selected storage nodes is configurable.

4. The system of claim 1, further comprising:

an object database storing a plurality of object records each associating a stored object and multiple ones of said storage nodes; and

said cloud application server is further operative to cause an object record associating said object and said selected storage nodes to be written to said object database.

5. The system of claim 4, wherein said cloud application server is further operative to:

receive an indication that said object is to be deleted; and

modify said object record to indicate that said object is marked for deletion.

6. The system of claim 5, wherein said cloud application server is further operative to purge said object marked for deletion from each of said selected storage nodes.

7. The system of claim 4, wherein:

said object database is sharded into a plurality of shards; and

each shard includes some of said plurality of object records.

8. The system of claim 7, wherein:

said object database further includes a plurality of shard records; and

each of said shard records includes a shard identifier uniquely identifying one of said shards.

9. The system of claim 7, wherein said UUID includes:

a first field containing data identifying one of said plurality of shards containing said object record associated with said object; and

a second field containing an object identifier identifying said object record among said plurality of object records.

10. The system of claim 4, wherein:

said client application server is further operative to

generate a download request associated with said object; and

said cloud object server is operative to

receive said download request,

request said cloud application server to provide storage node information facilitating the retrieval of said object from any of said selected storage nodes,

request said object from any one of said selected storage nodes, and

provide said object to said client application server.

11. The system of claim 10, wherein:

said object database is sharded into a plurality of shards;

each shard includes some of said plurality of object records; and

said download request includes said UUID, said UUID comprising a shard identifier identifying one of said plurality of shards containing said object record.

12. The system of claim 11, wherein:

said cloud object server is operative to provide said UUID to said cloud application server as part of said request for said storage node information; and

said cloud application server is further operative to use said UUID to query said object database to return said storage node information.

13. The system of claim 1, wherein said cloud object server, responsive to receiving said UUID, is operative to use said UUID to obtain information facilitating the retrieval of said object from any of said selected storage nodes.

14. The system of claim 1, wherein

said client application server is further operative to store a virtual object record associated with said object in said file system database.

15. The system of claim 14, wherein said client application server is further operative to:

receive a delete command from said client requesting that said object be deleted; and

modify said virtual object record to indicate that said object is to be deleted.

16. The system of claim 14, wherein:

said file system database includes a plurality of tables defining said virtual file system;

said plurality of tables is sharded into a plurality of shards; and

said data defining said virtual file system associated with said client is stored in only one of said plurality of shards.

17. The system of claim 1, further comprising:

a plurality of cloud object servers each being operative to retrieve said object from one of said selected storage nodes and serve said object to said client application server responsive to receiving a download request for said object;

a plurality of said cloud application servers; and

a load balancer operative to

receive said upload request from said client application server and provide it to a selected one of said plurality of cloud application servers and

receive said download request from said client application server and provide it to a selected one of said plurality of cloud object servers.

18. The system of claim 1, further comprising a storage node recovery module operative to rebuild one of said plurality of storage nodes, including said digital objects stored on said one of said plurality of storage nodes, when said one of said storage nodes fails.

19. The system of claim 18, further comprising:

an object database storing a plurality of object records associated with a plurality of stored objects, each said object record indicating multiple ones of said storage nodes storing an associated stored object; and

wherein

said storage node recovery module is operative to rebuild said failed storage node as a rebuilt storage node by using said plurality of object records in said object database to identify said stored objects stored on said failed storage node and copy said identified stored objects from other ones of said plurality of storage nodes to said rebuilt storage node.

20. The system of claim 18, further comprising:

a database storing a plurality of object records associated with a plurality of stored objects, each said object record indicating multiple ones of said storage nodes storing an associated stored object; and

wherein

said plurality of object records are sharded by storage node; and

said storage node recovery module is operative to rebuild said failed storage node as a rebuilt storage node by identifying a shard associated with said failed storage node and copying objects associated with the object records contained in said identified shard from other ones of said plurality of storage nodes to said rebuilt storage node.

21. The system of claim 1, further comprising a rebalance module operative to move objects stored on said one of said plurality of storage nodes to other ones of said plurality of storage nodes.

22. The system of claim 21, further comprising:

an object database storing a plurality of object records associated with a plurality of stored objects, each said object record indicating multiple ones of said storage nodes storing an associated stored object; and

wherein

said rebalance module is operative to modify at least some of said plurality of object records.

23. The system of claim 1, further comprising an object auditor operative to determine whether said object stored on one or more of said selected storage nodes has degraded.

24. The system of claim 1, wherein:

said client application server is further operative to receive a second object from said client via said inter-network; and

said cloud application server is further operative to

select at least two storage nodes from said plurality of storage nodes for said second object that are different from said at least two storage nodes selected for said object and

store said second object on each of said storage nodes selected for said second object.

25. The system of claim 1, wherein multiple ones of said plurality of storage nodes are selected on an object-by-object basis for additional objects uploaded by said client.

26. An object storage system comprising:

a plurality of storage nodes for storing digital objects associated with clients;

at least one client application server being operative to

establish a network connection with a client over an inter-network,

receive an indication of an object to be uploaded by said client,

generate an upload request associated with said object, and

receive said object from said client via said inter-network;

at least one cloud application server being operative to

receive said upload request from said client application server,

generate a Universally Unique Identifier (UUID) associated with said object, and

receive said object from said client application server;

means for selecting at least two storage nodes from said plurality of storage nodes in response to said upload request;

means for streaming said object from said client application server to each of said selected storage nodes such that said object is stored on each of said selected storage nodes;

a file system database storing information defining a virtual file system associated with said client and accessible to said client application server, said UUID being stored in said file system database;

at least one cloud object server; and

wherein

said client application server is further operative to

receive a download command for said object from said client,

retrieve said UUID from said file system database, and

provide said UUID to said cloud object server.

27. The system of claim 26, wherein each of said plurality of storage nodes exposes a HyperText Transfer Protocol (HTTP) interface.

28. The system of claim 26, wherein the number of said selected storage nodes is configurable.

29. The system of claim 26, further comprising:

an object database storing a plurality of object records each associating a stored object and multiple ones of said storage nodes; and

said cloud application server is further operative to cause an object record associating said object and said selected storage nodes to be written to said object database.

30. The system of claim 29, wherein said cloud application server is further operative to:

receive an indication that said object is to be deleted; and

modify said object record to indicate that said object is marked for deletion.

31. The system of claim 30, wherein said cloud application server is further operative to purge said object marked for deletion from each of said selected storage nodes.

32. The system of claim 29, wherein:

said object database is sharded into a plurality of shards; and

each shard includes some of said plurality of object records.

33. The system of claim 32, wherein:

said object database further includes a plurality of shard records; and

each of said shard records includes a shard identifier uniquely identifying one of said shards.

34. The system of claim 32, wherein said UUID includes:

a first field containing data identifying one of said plurality of shards containing said object record associated with said object; and

a second field containing an object identifier identifying said object record among said plurality of object records.

35. The system of claim 29, wherein:

said client application server is further operative to generate a download request associated with said object; and

said cloud object server is operative to

receive said download request,

request said cloud application server to provide storage node information facilitating the retrieval of said object from any of said selected storage nodes,

request said object from any one of said selected storage nodes, and

provide said object to said client application server.

36. The system of claim 35, wherein:

said object database is sharded into a plurality of shards;

each shard includes some of said plurality of object records; and

said download request includes said UUID, said UUID comprising a shard identifier identifying one of said plurality of shards containing said object record.

37. The system of claim 36, wherein:

said cloud object server is operative to provide said UUID to said cloud application server as part of said request for said storage node information; and

said cloud application server is further operative to use said UUID to query said object database to return said storage node information.

38. The system of claim 26, wherein said cloud object server, responsive to receiving said UUID, is operative to use said UUID to obtain information facilitating the retrieval of said object from any of said selected storage nodes.

39. The system of claim 26, wherein said client application server is further operative to store a virtual object record associated with said object in said file system database.

40. The system of claim 39, wherein said client application server is further operative to:

receive a delete command from said client requesting that said object be deleted; and

modify said virtual object record to indicate that said object is to be deleted.

41. The system of claim 39, wherein:

said file system database includes a plurality of tables defining said virtual file system;

said plurality of tables is sharded into a plurality of shards; and

said data defining said virtual file system associated with said client is stored in only one of said plurality of shards.

42. The system of claim 26, further comprising:

a plurality of cloud object servers each being operative to retrieve said object from one of said selected storage nodes and serve said object to said client application server responsive to receiving a download request for said object;

a plurality of said cloud application servers; and

a load balancer operative to

receive said upload request from said client application server and provide it to a selected one of said plurality of cloud application servers and

receive said download request from said client application server and provide it to a selected one of said plurality of cloud object servers.

43. The system of claim 26, further comprising a storage node recovery module operative to rebuild one of said plurality of storage nodes, including said digital objects stored on said one of said plurality of storage nodes, when said one of said storage nodes fails.

44. The system of claim 43, further comprising:

an object database storing a plurality of object records associated with a plurality of stored objects, each said object record indicating multiple ones of said storage nodes storing an associated stored object; and

wherein

said storage node recovery module is operative to rebuild said failed storage node as a rebuilt storage node by using said plurality of object records in said object database to identify said stored objects stored on said failed storage node and copy said identified stored objects from other ones of said plurality of storage nodes to said rebuilt storage node.

45. The system of claim 43, further comprising:

a database storing a plurality of object records associated with a plurality of stored objects, each said object record indicating multiple ones of said storage nodes storing an associated stored object; and

wherein

said plurality of object records are sharded by storage node; and

said storage node recovery module is operative to rebuild said failed storage node as a rebuilt storage node by identifying a shard associated with said failed storage node and copying objects associated with the object records contained in said identified shard from other ones of said plurality of storage nodes to said rebuilt storage node.

46. The system of claim 26, further comprising a rebalance module operative to move objects stored on said one of said plurality of storage nodes to other ones of said plurality of storage nodes.

47. The system of claim 46, further comprising:

an object database storing a plurality of object records associated with a plurality of stored objects, each said object record indicating multiple ones of said storage nodes storing an associated stored object; and

wherein

said rebalance module is operative to modify at least some of said plurality of object records.

48. The system of claim 26, further comprising an object auditor operative to determine whether said object stored on one or more of said selected storage nodes has degraded.

49. The system of claim 26, wherein:

said client application server is further operative to receive a second object from said client via said inter-network;

said means for selecting selects at least two storage nodes from said plurality of storage nodes for said second object that are different from said at least two storage nodes selected for said object; and

said means for streaming is operative to stream said second object to each of said storage nodes selected for said second object.

50. The system of claim 26, wherein multiple ones of said plurality of storage nodes are selected on an object-by-object basis for additional objects uploaded by said client.