Distributed object storage system comprising performance optimizations
First Claim
Patent Images
1. A distributed object storage system which, according to a first storage and retrieval option, comprising:
- a plurality of redundant storage elements, configured to store and retrieve a data object comprising a data object identifier in form of a predetermined number of redundant sub blocks comprising said data object identifier, said predetermined number of redundant sub blocks corresponding to a predetermined multiple of a desired spreading width, said predetermined multiple of the desired spreading width consisting of the sum of;
(a) a minimal spreading requirement, corresponding to a minimal number of storage elements that must store sub blocks of said data object and are not allowed to fail; and
(b) a maximal concurrent failures tolerance, corresponding to a number of storage elements that must store sub blocks of said data object and are allowed to fail concurrently;
each one of said redundant sub blocks comprising;
(a) encoded data of equal size of the data object divided by a factor equal to said predetermined multiple of said minimal spreading requirement; and
(b) decoding data, enabling said data object to be decoded from any combination of a minimum number of said redundant sub blocks, the minimum number corresponding to said predetermined multiple of said minimal spreading requirement;
a plurality of storage nodes each comprising a share of said plurality of redundant storage elements; and
at least one controller node, operably connected to or comprised within said plurality of storage nodes when storing or retrieving said data object, comprising;
(a) an encoding module configured to disassemble said data object into said predetermined number of redundant sub blocks;
(b) a spreading module configured to store said predetermined number of said redundant sub blocks on a number of said storage elements being larger or equal to said desired spreading width;
(c) a clustering module configured to retrieve at least said predetermined multiple of said minimal spreading requirement of said redundant sub blocks from a plurality of said storage elements; and
(d) a decoding module configured to assemble said data object from any combination of a minimum number of said redundant sub blocks, the minimum number corresponding to said predetermined multiple of said minimal spreading requirement;
wherein, according to a second storage and retrieval option of said distributed object storage system;
(a) said plurality of redundant storage elements, are further configured to store and retrieve said data object comprising a data object identifier in form of a predetermined plurality of replication copies of said data object comprising said data object identifier, said predetermined plurality corresponding to said desired spreading width, each one of said replication copies comprising an exact copy of said data object;
(b) said encoding module is further configured to replicate said data object into said predetermined plurality of said replication copies;
(c) said spreading module is further configured to store said predetermined plurality of said replication copies on a corresponding plurality of said storage elements;
(d) said clustering module is further configured to retrieve at least one of said predetermined plurality of said replication copies from said corresponding plurality of said storage elements; and
(e) said decoding module is further configured to provide said data object from any one of said replication copies; and
wherein said distributed object storage system is operated according to said first storage and retrieval option if the size of said data object is equal to or larger than a predetermined lower data object size threshold, and is operated according to said second storage and retrieval option if the size of said data object is smaller than said predetermined lower data object size threshold.
8 Assignments
0 Petitions
Accused Products
Abstract
A distributed object storage system includes several performance optimizations with respect to storing very small data objects, very large data objects and CRC calculations.
44 Citations
20 Claims
-
1. A distributed object storage system which, according to a first storage and retrieval option, comprising:
-
a plurality of redundant storage elements, configured to store and retrieve a data object comprising a data object identifier in form of a predetermined number of redundant sub blocks comprising said data object identifier, said predetermined number of redundant sub blocks corresponding to a predetermined multiple of a desired spreading width, said predetermined multiple of the desired spreading width consisting of the sum of; (a) a minimal spreading requirement, corresponding to a minimal number of storage elements that must store sub blocks of said data object and are not allowed to fail; and (b) a maximal concurrent failures tolerance, corresponding to a number of storage elements that must store sub blocks of said data object and are allowed to fail concurrently; each one of said redundant sub blocks comprising; (a) encoded data of equal size of the data object divided by a factor equal to said predetermined multiple of said minimal spreading requirement; and (b) decoding data, enabling said data object to be decoded from any combination of a minimum number of said redundant sub blocks, the minimum number corresponding to said predetermined multiple of said minimal spreading requirement; a plurality of storage nodes each comprising a share of said plurality of redundant storage elements; and at least one controller node, operably connected to or comprised within said plurality of storage nodes when storing or retrieving said data object, comprising; (a) an encoding module configured to disassemble said data object into said predetermined number of redundant sub blocks; (b) a spreading module configured to store said predetermined number of said redundant sub blocks on a number of said storage elements being larger or equal to said desired spreading width; (c) a clustering module configured to retrieve at least said predetermined multiple of said minimal spreading requirement of said redundant sub blocks from a plurality of said storage elements; and (d) a decoding module configured to assemble said data object from any combination of a minimum number of said redundant sub blocks, the minimum number corresponding to said predetermined multiple of said minimal spreading requirement; wherein, according to a second storage and retrieval option of said distributed object storage system; (a) said plurality of redundant storage elements, are further configured to store and retrieve said data object comprising a data object identifier in form of a predetermined plurality of replication copies of said data object comprising said data object identifier, said predetermined plurality corresponding to said desired spreading width, each one of said replication copies comprising an exact copy of said data object; (b) said encoding module is further configured to replicate said data object into said predetermined plurality of said replication copies; (c) said spreading module is further configured to store said predetermined plurality of said replication copies on a corresponding plurality of said storage elements; (d) said clustering module is further configured to retrieve at least one of said predetermined plurality of said replication copies from said corresponding plurality of said storage elements; and (e) said decoding module is further configured to provide said data object from any one of said replication copies; and wherein said distributed object storage system is operated according to said first storage and retrieval option if the size of said data object is equal to or larger than a predetermined lower data object size threshold, and is operated according to said second storage and retrieval option if the size of said data object is smaller than said predetermined lower data object size threshold. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. A method for storage and retrieval of a data object on a distributed object storage system, the data object having a size, the method comprising:
-
if the size of the data object is greater than or equal to a predetermined lower data object size threshold; disassembling the data object into a predetermined number of redundant sub blocks, the predetermined number of redundant sub blocks corresponding to a predetermined multiple of a desired spreading width, the predetermined multiple of the desired spreading width consisting of the sum of;
(a) a minimal spreading requirement corresponding to a minimal number of storage elements that must store sub blocks of the data object and are not allowed to fail, and (b) a maximal concurrent failures tolerance corresponding to a number of storage elements that must store sub blocks of the data object and are allowed to fail concurrently;
each one of the redundant sub blocks comprising;
(a) encoded data of equal size of the data object divided by a factor equal to the predetermined multiple of the minimal spreading requirement; and
(b) decoding data, enabling the data object to be decoded from any combination of a particular number of the redundant sub blocks, the particular number corresponding to the predetermined multiple of the minimal spreading requirement;storing the predetermined number of redundant sub blocks on a number of redundant storage elements, the number of redundant storage elements being greater than or equal to the desired spreading width; retrieving any combination of the particular number of the redundant sub blocks from the number of storage elements; and assembling the data object from the retrieved redundant sub blocks; otherwise, if the size of the data object is less than the predetermined lower data object size threshold; replicating the data object into a predetermined number of replication copies corresponding to the desired spreading width, each one of the replication copies comprising an exact copy of the data object; storing the predetermined number of replication copies on a corresponding plurality of storage elements; retrieving any of the replication copies from the corresponding plurality of the storage elements; and providing the data object from the replication copy. - View Dependent Claims (14, 15, 16, 17, 18, 19)
-
-
20. A computer device for storage and retrieval of a data object on a distributed object storage system, the computer device comprising:
-
a processor; and memory having stored thereon instructions that when processed by the processor cause a method for storage and retrieval of a data object on a distributed object storage system to be performed, the method comprising; if a size of the data object is greater than or equal to a predetermined lower data object size threshold; disassembling the data object into a predetermined number of redundant sub blocks, the predetermined number of redundant sub blocks corresponding to a predetermined multiple of a desired spreading width, the predetermined multiple of the desired spreading width consisting of the sum of;
(a) a minimal spreading requirement corresponding to a minimal number of storage elements that must store sub blocks of the data object and are not allowed to fail, and (b) a maximal concurrent failures tolerance corresponding to a number of storage elements that must store sub blocks of the data object and are allowed to fail concurrently;
each one of the redundant sub blocks comprising;
(a) encoded data of equal size of the data object divided by a factor equal to the predetermined multiple of the minimal spreading requirement; and
(b) decoding data, enabling the data object to be decoded from any combination of a particular number of the redundant sub blocks, the particular number corresponding to the predetermined multiple of the minimal spreading requirement;storing the predetermined number of redundant sub blocks on a number of redundant storage elements, the number of redundant storage elements being greater than or equal to the desired spreading width; retrieving any combination of the particular number of the redundant sub blocks from the number of storage elements; and assembling the data object from the retrieved redundant sub blocks; otherwise, if the size of the data object is less than the predetermined lower data object size threshold; replicating the data object into a predetermined number of replication copies corresponding to the desired spreading width, each one of the replication copies comprising an exact copy of the data object; storing the predetermined number of replication copies on a corresponding plurality of storage elements; retrieving any of the replication copies from the corresponding plurality of the storage elements; and providing the data object from the replication copy.
-
Specification