Distributed storage system and distributed storage method

• US 9,342,574 B2
• Filed: 03/08/2012
• Issued: 05/17/2016
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Kan, Masaki, Torii, Takashi
• Primary Examiner(s): Alam, Shahid
• Assistant Examiner(s): Ho, Andrew N
• Application Number:
  US14/003,658
• Publication Number:
  US 20130346365A1
• Time in Patent Office: 1,531 Days
• Field of Search: 707/610
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/2308 Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 16/2336 Pessimistic concurrency control approaches, e.g. locking or multiple versions without time stamps
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

A distributed storage system of the present invention includes a plurality of data nodes coupled via a network and respectively including data storage units. At least two of the data nodes hold in the respective data storage units thereof replicas of a plurality of types of data structures that are logically identical but are physically different between the data nodes.

Claims

1. A distributed storage system comprising:

a plurality of data nodes coupled via a network, the data nodes including data storage units, respectively, wherein the data nodes of replication destination of data, include at least two data nodes that hold, in the respective data storage units thereof, data structures logically identical but physically different between the at least two data nodes;

a structure information management apparatus including a structure information retention unit configured to store and manage data structure management information provided corresponding to a number of types of the data structures and data arrangement specifying information,

the data structure management information each including, in association with a table identifier that is an identifier to identify data to be stored, a replica identifier to identify a replica, data structure information to identify a type of the data structure associated with the replica identifier, and update trigger information that is information indicating a period to be taken for the replica to be converted to the data structure specified,

the data arrangement specifying information including, in association with the table identifier, the replica identifier, data node information indicating the data nodes of one or a plurality of data arrangement destinations associated with the replica identifier;

a client function implementation unit including a data access unit configured to identify an access destination for each of an update process and a reference process by referring to the data structure management information and the data arrangement specifying information; and

a plurality of the data nodes, each of the data nodes including the data storage unit, and being connected to the structure information management apparatus and the client function implementation unit;

the data nodes including;

a data management and processing unit configured to hold data in the intermediate data retention structure to return the response to the client function implementation unit, when the update process is performed, based on an access request from the client function implementation unit; and

a data structure conversion unit configured to perform a process of converting the data held in the intermediate data retention structure to the data structure specified in the data structure management information, by referring to the data structure management information, according to an update trigger specified.

2. The distributed storage system according to claim 1, wherein the intermediate data retention structure holds the data until the data is stored in the data storage unit with the target data structure specified.

3. The distributed storage system according to claim 1, wherein the client function implementation unit selects the data node of an access destination according to content of the update process or the reference process, based on the data structure management information and the data arrangement specifying information.

4. The distributed storage system according to claim 1, wherein the client function implementation unit obtains the data arrangement specifying information held in the structure information retention unit of the structure information management apparatus or the data arrangement specifying information held in a structure information cache holding unit configured to cache the information held in the structure information retention unit, and issues an access instruction to the data node of a data arrangement destination.

5. The distributed storage system according to claim 1, wherein the data node includes an access accept unit, an access process unit, and a data structure conversion unit, wherein

the data storage unit of the data node includes a classified-by-structure data storage unit, wherein

the access accept unit receives the update request from the client function implementation unit, and then transfers the update request to the data node specified in association with the replica identifier in the data arrangement specifying information,

the access process unit of the data node performs a process associated with the received update request, executes the update process by referring to information in the data structure management information, converts the update data to the data structure specified in the data structure management information and updates the classified-by-structure data storage unit when the update trigger for the data node is zero,

the access process unit temporarily writes the update data to the immediate data retention structure and makes a response indicating completion of the process when the update trigger is not zero;

the access accept unit makes the response to the client function implementation unit upon receipt of a notification of the completion from the access process unit and a notification of completion from the data node of the replication destination, and

the data structure conversion unit converts the data in the immediate data retention structure to the data structure specified in the data structure management information and stores the converted data in the classified-by-structure data structure unit for the converted data.

6. The distributed storage system according to claim 1, wherein the client function implementation unit, in the case of a reference access, after selecting the data structure suited to the data access for the data node, and identifying the replica identifier, computes the data node to be accessed, issues an access request to the selected data node, and then receives a result of the access process from the data node.

7. The distributed storage system according to claim 1, wherein the client function implementation unit is provided within the data node.

8. The distributed storage system according to claim 7, wherein the client function implementation unit comprises a structure information cache holding unit configured to cache the information in the structure information retention unit.

9. The distributed storage system according to claim 8, comprising:

a structure information synchronization unit configured to synchronize the structure information in the structure information cache holding unit of the client function implementation unit and the structure information held in the structure information retention unit of the structure information management apparatus.

10. The distributed storage system according to claim 1, wherein the data structure management information includes, in association with the replica identifier a number of partitions indicating a number of partitionings of data for storage in a plurality of the data nodes;

the data arrangement specifying information includes the plurality of the data nodes as the nodes for arrangement corresponding to the replica identifier associated with the number of partitions of two or more in the data structure management information; and

when arrangement destinations of the partitioned data extend across a plurality of the data nodes, the access accept unit of the data node that has received the access request issues an access request to the access process unit of each of the plurality of data nodes different from the data node that has received the access request.

11. The distributed storage system according to claim 1, wherein on reception of the access request, the data structure conversion unit of the data node, when the update trigger is zero, issues an access request to the data structure conversion unit of the data node different from the data node that has received the access request.

12. The distributed storage system according to claim 1, comprising:

a log recording unit configured to record a log of the access request; and

a change determination unit configured to determine whether or not to convert the data structure, using information on the log recorded in the log recording unit.

13. The distributed storage system according to claim 12, wherein the change determination unit outputs a conversion request to a structure information changing unit of the structure information management apparatus, when the change determination unit determines that data structure conversion is necessary;

the structure information changing unit of the structure information management apparatus alters the information in the structure information retention unit, and outputs a conversion request to the data structure conversion unit of the data node; and

the data structure conversion unit of the data node converts the data structure held in the data storage unit of the data node.

14. A distributed storage method in a system comprising a plurality of data nodes coupled via a network, the plurality of data nodes respectively including data storage units, the method comprising:

at least two of the data nodes of data replication destinations, receiving replication data; and

storing the data in respective data storage units thereof, in data structures that are logically identical but are physically different between the at least two of the data nodes, the method comprising;

in replication of data executed at a time of a data update, the data node of a replication destination converting the data to be updated to a target data structure specified and storing the converted data in the data storage unit, at the time, the data node temporarily holding data to be updated in an intermediate data retention structure to return a response for the update, and converting the data to be updated to a target data structure specified, asynchronously with an update request to store the converted data in the data storage unit, the method further comprising;

a structure information management unit storing and managing structure information including data structure management information and data arrangement specifying information, the data structure management information being provided corresponding to a number of types of the data structures and each including, in association with a table identifier that is an identifier to identify data to be stored, a replica identifier to identify a replica, data structure information to identify a type of the data structure associated with the replica identifier, and update trigger information that is information indicating a period to be taken for the replica to be converted to the data structure that has been specified, the data arrangement specifying information including, in association with the table identifier, the replica identifier, data node information indicating the data nodes of one or a plurality of data arrangement destinations associated with the replica identifier;

a client identifying an access destination for each of an update process and a reference process by referring to the data structure management information and the data arrangement specifying information; and

the data node holding data in the intermediate data retention structure to return the response to the client, when the update process is performed, based on an access request from the client, and converting the data held in the intermediate data structure to the data structure specified in the data structure management information, by referring to the data structure management information, according to an update trigger specified.

15. The distributed storage method according to claim 14, wherein the data structure management information includes, in association with the replica identifier a number of partitions indicating a number of partitionings of data for storage in a plurality of the data nodes;

the data arrangement specifying information includes the plurality of the data nodes as the nodes for arrangement corresponding to the replica identifier associated with the number of partitions of two or more in the data structure management information; and

the method comprises;

by the data node that has received the access request, issuing an access request to each of the plurality of data nodes different from the data node that has received the access request, when arrangement destinations of the partitioned data extend across a plurality of the data nodes.

16. The distributed storage method according to claim 14, comprising:

determining whether or not to convert the data structure, using information on a log recorded in a log recording unit configured to record the log of the access request, and converting the structure information and further converting the data structure of the data node when the conversion is needed.