Information processing device, data processing method, and recording medium

• US 10,657,121 B2
• Filed: 08/18/2015
• Issued: 05/19/2020
• Est. Priority Date: 08/21/2014
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Kadowaki, Takayuki
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US15/505,276
• Publication Number:
  US 20170270155A1
• Time in Patent Office: 1,736 Days
• Field of Search: 707691, 707690, 707698
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 9/30 Arrangements for executing machine instructions, e.g. instruction decode for executing microinstructions G06F9/22

Abstract

Distributed batch processing on an eventually consistent storage system is efficiently performed. A control node includes an execution control unit and a re-execution control unit. The execution control unit causes a processing node to execute predetermined processing that includes reading of data for a key stored in a distributed data store. The re-execution control unit determines, after causing the processing node to execute the predetermined processing, presence or absence of a possibility that inconsistency occurred on data for the key stored in the distributed data store at a time of execution of the predetermined processing, based on a representative value of the data. Then, the re-execution control unit causes, when it is determined that there is a possibility that the inconsistency occurred, the processing node to re-execute the predetermined processing at a time point when resolution of the inconsistency is verified.

Claims

1. An information processing device comprising:

a memory storing instructions; and

one or more processors configured to execute the instructions to;

execute predetermined processing that includes reading of data for each of a plurality of keys stored in a distributed storage system including a plurality of data store nodes;

determine, after executing the predetermined processing for each of the plurality of keys, presence or absence of a possibility that inconsistency between the data store nodes occurred on data for any key among the plurality of keys stored in the distributed storage system at a time of execution of the predetermined processing, based on a representative value of the data for all of the plurality of keys; and

when it is determined that there is a possibility that the inconsistency occurred, identify all of the keys among the plurality of keys having a possibility that the inconsistency occurred, re-execute the predetermined processing for all of the keys identified at a time point when resolution of the inconsistency for all of the keys identified is verified.

2. The information processing device according to claim 1, wherein,

in the distributed storage system, data for each of the plurality of keys is duplicated and stored in first and second regions, and

using a representative value of data for each of the plurality of keys and a representative value concerning all data for the plurality of keys which are generated for each of the first region and the second region, presence or absence of a possibility that inconsistency occurred on data for any of the plurality of keys is determined and a key having a possibility of the inconsistency is identified.

3. The information processing device according to claim 2, wherein

when a representative value concerning all data for the plurality of keys in the first region and a representative value concerning all data for the plurality of keys in the second region, prior to execution of the predetermined processing, are different, it is determined that there is a possibility that inconsistency occurred on data for any of the plurality of keys, and

out of the plurality of keys, a key for which data in the first region and data in the second region have different representative values is identified, as a key having a possibility of the inconsistency.

4. The information processing device according to claim 1, wherein

the one or more processors are further configured to execute the instructions to execute predetermined pre-processing that includes writing of data into the distributed storage system, and

the one or more processors are configured to execute the predetermined processing after the predetermined pre-processing is executed.

5. A data processing method comprising:

executing predetermined processing that includes reading of data for each of a plurality of keys stored in a distributed storage system including a plurality of data store nodes;

determining, after executing the predetermined processing for each of the plurality of keys, presence or absence of a possibility that inconsistency between the data store nodes occurred on data for any key among the plurality of keys stored in the distributed storage system at a time of execution of the predetermined processing, based on a representative value of the data for all of the plurality of keys; and

when it is determined that there is a possibility that the inconsistency occurred, identify all of the keys among the plurality of keys having possibility that the inconsistency occurred, re-executing the predetermined processing for all of the keys identified at a time point when resolution of the inconsistency for all of the keys identified is verified.

6. The data processing method according to claim 5, wherein,

in the distributed storage system, data for each of the plurality of keys is duplicated and stored in first and second regions, and

using a representative value of data for each of the plurality of keys and a representative value concerning all data for the plurality of keys which are generated for each of the first region and the second region, presence or absence of a possibility that inconsistency occurred on data for any of the plurality of keys is determined and a key having a possibility of the inconsistency is identified.

7. The data processing method according to claim 6, wherein,

when a representative value concerning all data for the plurality of keys in the first region and a representative value concerning all data for the plurality of keys in the second region, prior to execution of the predetermined processing, are different, it is determined that there is a possibility that inconsistency occurred on data for any of the plurality of keys, and

out of the plurality of keys, a key for which data in the first region and data in the second region have different representative values is identified, as a key having a possibility of the inconsistency.

8. A non-transitory computer readable storage medium recording thereon a program causing a computer to perform a method comprising:

executing predetermined processing that includes reading of data for each of a plurality of keys stored in a distributed storage system including a plurality of data store nodes;

determining, after executing the predetermined processing for each of the plurality of keys, presence or absence of a possibility that inconsistency between the data store nodes occurred on data for any key among the plurality of keys stored in the distributed storage system at a time of execution of the predetermined processing, based on a representative value of the data for all of the plurality of keys; and

when it is determined that there is a possibility that the inconsistency occurred, identify all of the keys among the plurality of keys having a possibility that the inconsistency occurred, re-executing the predetermined processing for all of the keys identified at a time point when resolution of the inconsistency for all of the keys identified is verified.