DISTRIBUTED STORAGE SYSTEM AND DISTRIBUTED STORAGE METHOD

• US 20130346365A1
• Filed: 03/08/2012
• Published: 12/26/2013
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Kan, Masaki, Torii, Takashi
• Granted Patent: US 9,342,574 B2
• Time in Patent Office: Days
• US Class Current: 707/610
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/2308 Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 16/2336 Pessimistic concurrency control approaches, e.g. locking or multiple versions without time stamps
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

A distributed storage system of the present invention includes a plurality of data nodes coupled via a network and respectively including data storage units. At least two of the data nodes hold in the respective data storage units thereof replicas of a plurality of types of data structures that are logically identical but are physically different between the data nodes.

Claims

1. A distributed storage system, comprising:

a plurality of data nodes coupled via a network, the data nodes including data storage units, respectively, wherein

the data nodes of replication destination of data, include at least two data nodes that hold, in the respective data storage units thereof, data structures logically identical but physically different between the at least two data nodes.

2. The distributed storage system according to claim 1, wherein each of the data nodes of replication destinations performs conversion to a target data structure asynchronously with reception of replication data.

3. The distributed storage system according to claim 2, wherein each of the data nodes of replication destinations holds the replication data in an intermediate data retention structure to return a response, and asynchronously converts a data structure held in the intermediate data retention structure to the target data structure.

4. The distributed storage system according to claim 2, comprising:

a unit configured to variably control the data node of a data arrangement destination, data structure in an arrangement destination, and data partitioning, for each table set in advance.

5. The distributed storage system according to claim 1, comprising:

a unit configured to determine the data node to arrange data by consistent hashing.

6. The distributed storage system according to claim 1, wherein, in replication of data executed at a time of a data update, the data node of a replication destination converts the data to be updated to a target data structure specified and stores the converted data in the data storage unit, at the time, the data node temporarily holding data to be updated in an intermediate data retention structure to return a response for the update, and converting the data to be updated to a specified target data structure asynchronously with an update request to store the converted data in the data storage unit.

7. The distributed storage system according to claim 1, comprising:

a structure information management apparatus including a structure information retention unit configured to store and manage data structure management information provided corresponding to a number of types of the data structures and data arrangement specifying information,

the data structure management information each including, in association with a table identifier that is an identifier to identify data to be stored, a replica identifier to identify a replica, data structure information to identify a type of the data structure associated with the replica identifier, and update trigger information that is information indicating a period to be taken for the replica to be converted to the data structure specified,

the data arrangement specifying information including, in association with the table identifier, the replica identifier, data node information indicating the data nodes of one or a plurality of data arrangement destinations associated with the replica identifier;

a client function implementation unit including a data access unit configured to identify an access destination for each of an update process and a reference process by referring to the data structure management information and the data arrangement specifying information; and

a plurality of the data nodes, each of the data nodes including the data storage unit, and being connected to the structure information management apparatus and the client function implementation unit;

the data nodes including;

a data management and processing unit configured to hold data in the intermediate data retention structure to return the response to the client function implementation unit, when the update process is performed, based on an access request from the client function implementation unit; and

a data structure conversion unit configured to perform a process of converting the data held in the intermediate data retention structure to the data structure specified in the data structure management information, by referring to the data structure management information, according to an update trigger specified.

8. The distributed storage system according to claim 7, wherein the intermediate data retention structure holds the data until the data is stored in the data storage unit with the target data structure specified.

9. The distributed storage system according to claim 7, wherein the client function implementation unit selects the data node of an access destination according to content of the update process or the reference process, based on the data structure management information and the data arrangement specifying information.

10. The distributed storage system according to claim 7, wherein the client function implementation unit obtains the data arrangement specifying information held in the structure information retention unit of the structure information management apparatus or the data arrangement specifying information held in a structure information cache holding unit configured to cache the information held in the structure information retention unit, and issues an access instruction to the data node of a data arrangement destination.

11. The distributed storage system according to claim 7, wherein the data node includes an access accept unit, an access process unit, and a data structure conversion unit, wherein

the data storage unit of the data node includes a classified-by-structure data storage unit, wherein

the access accept unit receives the update request from the client function implementation unit, and then transfers the update request to the data node specified in association with the replica identifier in the data arrangement specifying information,

the access process unit of the data node performs a process associated with the received update request, executes the update process by referring to information in the data structure management information, converts the update data to the data structure specified in the data structure management information and updates the classified-by-structure data storage unit when the update trigger for the data node is zero,

the access process unit temporarily writes the update data to the immediate data retention structure and makes a response indicating completion of the process when the update trigger is not zero;

the access accept unit makes the response to the client function implementation unit upon receipt of a notification of the completion from the access process unit and a notification of completion from the data node of the replication destination, and

the data structure conversion unit converts the data in the immediate data retention structure to the data structure specified in the data structure management information and stores the converted data in the classified-by-structure data structure unit for the converted data.

12. The distributed storage system according to claim 7, wherein the client function implementation unit, in the case of a reference access, after selecting the data structure suited to the data access for the data node, and identifying the replica identifier, computes the data node to be accessed, issues an access request to the selected data node, and then receives a result of the access process from the data node.

13. The distributed storage system according to claim 7, wherein the client function implementation unit is provided within the data node.

14. The distributed storage system according to claim 13, wherein the client function implementation unit comprises a structure information cache holding unit configured to cache the information in the structure information retention unit.

15. The distributed storage system according to claim 14, comprising:

a structure information synchronization unit configured to synchronize the structure information in the structure information cache holding unit of the client function implementation unit and the structure information held in the structure information retention unit of the structure information management apparatus.

16. The distributed storage system according to claim 7, wherein the data structure management information includes, in association with the replica identifier a number of partitions indicating a number of partitionings of data for storage in a plurality of the data nodes;

the data arrangement specifying information includes the plurality of the data nodes as the nodes for arrangement corresponding to the replica identifier associated with the number of partitions of two or more in the data structure management information; and

when arrangement destinations of the partitioned data extend across a plurality of the data nodes, the access accept unit of the data node that has received the access request issues an access request to the access process unit of each of the plurality of data nodes different from the data node that has received the access request.

17. The distributed storage system according to claim 7, wherein on reception of the access request, the data structure conversion unit of the data node, when the update trigger is zero, issues an access request to the data structure conversion unit of the data node different from the data node that has received the access request.

18. The distributed storage system according to claim 7, comprising:

a log recording unit configured to record a log of the access request; and

a change determination unit configured to determine whether or not to convert the data structure, using information on the log recorded in the log recording unit.

19. The distributed storage system according to claim 18, wherein the change determination unit outputs a conversion request to a structure information changing unit of the structure information management apparatus, when the change determination unit determines that data structure conversion is necessary;

the structure information changing unit of the structure information management apparatus alters the information in the structure information retention unit, and outputs a conversion request to the data structure conversion unit of the data node; and

the data structure conversion unit of the data node converts the data structure held in the data storage unit of the data node.

20. A distributed storage method in a system comprising a plurality of data nodes coupled via a network, the plurality of data nodes respectively including data storage units, the method comprising:

at least two of the data nodes of data replication destinations, receiving replication data; and

storing the data in respective data storage units thereof, in data structures that are logically identical but are physically different between the at least two of the data nodes.

21. The distributed storage method according to claim 20, comprising:

the data node of replication destinations performing conversion to a target data structure asynchronously with reception of replication data.

22. The distributed storage method according to claim 21, comprising:

the data nodes of replication destination holding the replication data in an intermediate data retention structure to return a response, and asynchronously converting a data structure held in the intermediate data retention structure to the target data structure.

23. The distributed storage method according to claim 21, comprising:

variably controlling the data node of a data arrangement destination, the data structure of the arrangement destination, and data partitioning, for each table set in advance.

24. The distributed storage system according to claim 20, comprising:

determining the data node to arrange data by consistent hashing.

25. The distributed storage method according to claim 20, comprising:

in replication of data executed at a time of a data update, the data node of a replication destination converting the data to be updated to a target data structure specified and storing the converted data in the data storage unit, at the time, the data node temporarily holding data to be updated in an intermediate data retention structure to return a response for the update, and converting the data to be updated to a target data structure specified, asynchronously with an update request to store the converted data in the data storage unit.

26. The distributed storage method according to claim 25, comprising:

a structure information management unit storing and managing structure information including data structure management information and data arrangement specifying information, the data structure management information being provided corresponding to a number of types of the data structures and each including, in association with a table identifier that is an identifier to identify data to be stored, a replica identifier to identify a replica, data structure information to identify a type of the data structure associated with the replica identifier, and update trigger information that is information indicating a period to be taken for the replica to be converted to the data structure that has been specified, the data arrangement specifying information including, in association with the table identifier, the replica identifier, data node information indicating the data nodes of one or a plurality of data arrangement destinations associated with the replica identifier;

a client identifying an access destination for each of an update process and a reference process by referring to the data structure management information and the data arrangement specifying information; and

the data node holding data in the intermediate data retention structure to return the response to the client, when the update process is performed, based on an access request from the client, and converting the data held in the intermediate data structure to the data structure specified in the data structure management information, by referring to the data structure management information, according to an update trigger specified.

27. The distributed storage method according to claim 26, wherein the data structure management information includes, in association with the replica identifier a number of partitions indicating a number of partitionings of data for storage in a plurality of the data nodes;

the data arrangement specifying information includes the plurality of the data nodes as the nodes for arrangement corresponding to the replica identifier associated with the number of partitions of two or more in the data structure management information; and

the method comprises;

by the data node that has received the access request, issuing an access request to each of the plurality of data nodes different from the data node that has received the access request, when arrangement destinations of the partitioned data extend across a plurality of the data nodes.

28. The distributed storage method according to claim 26, comprising:

determining whether or not to convert the data structure, using information on a log recorded in a log recording unit configured to record the log of the access request, and converting the structure information and further converting the data structure of the data node when the conversion is needed.

29. A data node apparatus comprising

a data storage unit,

the data node apparatus coupled to other data nodes via a network, a plurality of data nodes forming a distributed storage system, the data node apparatus comprising

a unit that, in of replication of data to be updated to a plurality of the data nodes, receives replication data and stores the data in the data storage unit thereof in a data structure that is logically identical to but is physically different from at least one of other data nodes.

30. The data node apparatus according to claim 29, wherein the data node apparatus temporarily holds the data to be updated in an intermediate data retention structure to return a response to an update request, converts the data structure to a specified data structure asynchronously with the update request, and then stores the converted data structure in the data storage unit.