Computer system, data management method, and computer

• US 9,870,385 B2
• Filed: 04/01/2013
• Issued: 01/16/2018
• Est. Priority Date: 04/01/2013
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Kawamoto, Shinichi, Hanaoka, Miyuki
• Primary Examiner(s): Peach, Polina
• Assistant Examiner(s): Abraham, Ahmed
• Application Number:
  US14/428,805
• Publication Number:
  US 20150234882A1
• Time in Patent Office: 1,751 Days
• Field of Search: 707634
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2201/805 Real-time

Abstract

A computer system comprising a plurality of computers and a storage system, further comprising: a first data processing module configured to process data transmitted from an apparatus; a memory store; a database; a memory store control module; a second data processing module configured to execute predetermined processing using processed data included in a first data range; and a third data processing module configured to execute predetermined processing using the processed data included in a second data range, the memory store control module being configured to: specify a third data range, which is a time range of the processed data to be written to the database; and specify deletable processed data based on the first data range and the third data range.

Claims

1. A computer system configured to process data transmitted from an external apparatus, the computer system comprising a plurality of computers and a storage system,

each of the plurality of computers including a processor, a memory coupled to the processor, and a network interface coupled to the processor,

the storage system including a controller and at least one storage medium,

the data transmitted from the external apparatus including a time stamp,

the computer system further comprising:

a first data processing module configured to process the data transmitted from the external apparatus;

a memory store constructed on a storage area of the memory included in at least one of the plurality of computers, the memory store being configured to temporarily store processed data processed by the first data processing module;

a database constructed on a storage area of the at least one storage medium included in the storage system, the database being configured to store the processed data;

a memory store control module configured to manage the processed data stored in the memory store and to control read processing and write processing to and from the memory store;

a database management module configured to manage the database and to control read processing and write processing to and from the database;

a second data processing module configured to execute predetermined processing using the processed data included in a first data range of the processed data stored in the memory store; and

a third data processing module configured to execute predetermined processing using the processed data included in a second data range of the processed data stored in the database, the second data range having a larger data range than the first data range,

the memory store control module being configured to;

specify a third data range, which is a time range of the processed data to be written to the database;

obtain the processed data included in the specified third data range from the memory store;

transmit to the database management module a write request to write the obtained processed data to the database; and

specify deletable processed data of the processed data temporarily stored in the memory store based on the first data range and the third data range,

wherein the first data range is a time range extending a predetermined period back in time from the newest time stamp of the processed data,

wherein the third data range is a time range advanced by a predetermined period from a time point to be a predetermined reference point,

wherein the memory store includes, for each of a plurality of the third data ranges, storage data management information for managing a first state indicating a write state of the processed data to the database and a second state indicating a usage state of the processed data by the second data processing module, and

wherein the memory store control module is configured to;

refer to the storage data management information; and

specify, based on the first state and the second state, the deletable processed data of the processed data temporarily stored in the memory store for each of the plurality of the third data ranges,

wherein the memory store includes third data range management information for managing each of the plurality of the third data ranges, and

wherein the memory store control module is configured to;

receive an input of a first parameter for specifying a write timing of the processed data to the database and the third data range to be written;

refer, after the write timing has passed, to the third data range management information based on the first parameter to specify the third data range to be written from the plurality of the third data ranges;

obtain the processed data included in the specified third data range from the memory store;

transmit to the database management module the write request including the obtained processed data;

update the first state corresponding to the specified third data range to a state indicating that writing of the processed data to the database is complete;

refer to the storage data management information to retrieve the third data range for which writing of the processed data to the database is complete, and that is not being used by the second data processing module; and

delete from the memory store the processed data included in the retrieved third data range,

wherein the memory store includes first data range management information for managing a starting point time stamp, which is a time stamp of the processed data serving as a starting point of the first data range, and a finishing point time stamp, which is a time stamp of the processed data serving as a finishing point of the first data range, and

wherein the memory store control module is configured to;

receive an input of a second parameter for designating the predetermined time for specifying the first data range;

update the starting point time stamp of the first data range management information to the time stamp of the processed data included in the write request to write the processed data of the write request received from the first data processing module;

specify the oldest time stamp from the processed data included in the time range extending a predetermined period back in time from the starting point time stamp based on the starting point time stamp and the second parameter in a case of being received a read request to read the processed data from the second data processing module;

update the finishing point time stamp of the first data range management information to the specified oldest time stamp;

refer to the first data range management information to determine whether the third data range including the processed data corresponding to the pre-update finishing point time stamp and the third data range including the processed data corresponding to the updated finishing point time stamp match;

refer to the storage data management information when there is a determination that the third data range including the processed data corresponding to the pre-update finishing point time stamp and the third data range including the processed data corresponding to the updated finishing point time stamp do not match;

update the second state of the third data range including the processed data corresponding to the pre-update finishing point time stamp to a state indicating that the processed data included in the third data range is not being used;

refer to the storage data management information to retrieve the third data range for which writing of the processed data to the database is complete, and that is not being used by the second data processing module; and

delete from the memory store the processed data included in the retrieved third data range.

2. The computer system according to claim 1,

wherein the computer system receives a plurality of types of data from the external apparatus, and

wherein the memory store control module holds the first data range management information, the third data range management information, and the storage data management information for each of the plurality of types of data.

3. A data management method performed in a computer system configured to process data transmitted from an external apparatus, the computer system including a plurality of computers and a storage system,

each of the plurality of computers including a processor, a memory coupled to the processor, and a network interface coupled to the processor,

the storage system including a controller and at least one storage medium,

the data transmitted from the external apparatus including a time stamp,

the computer system further including:

a first data processing module configured to process the data transmitted from the external apparatus;

a memory store constructed on a storage area of the memory included in at least one of the plurality of computers, the memory store being configured to temporarily store processed data processed by the first data processing module;

a database constructed on a storage area of the at least one storage medium included in the storage system, the database being configured to store the processed data;

a memory store control module configured to manage the processed data stored in the memory store and to control read processing and write processing to and from the memory store;

a database management module configured to manage the database and to control read processing and write processing to and from the database;

a second data processing module configured to execute predetermined processing using the processed data included in a first data range of the processed data stored in the memory store; and

a third data processing module configured to execute predetermined processing using the processed data included in a second data range of the processed data stored in the database, the second data range having a larger data range than the first data range,

the data management method including;

a first step of specifying, by the memory store control module, a third data range, which is a time range of the processed data to be written to the database;

a second step of obtaining, by the memory store control module, the processed data included in the specified third data range from the memory store;

a third step of transmitting, by the memory store control module, to the database management module, a write request to write the obtained processed data to the database; and

a fourth step of specifying, by the memory store control module, deletable processed data of the processed data temporarily stored in the memory store based on the first data range and the third data range,

wherein the first data range is a time range extending a predetermined period back in time from the newest time stamp of the processed data,

wherein the third data range is a time range advanced by a predetermined period from a time point to be a predetermined reference point,

wherein the memory store including, for each of a plurality of the third data ranges, storage data management information for managing a first state indicating a write state of the processed data to the database and a second state indicating a usage state of the processed data by the second data processing module, and

wherein the fourth step includes;

referring to the storage data management information; and

specifying, based on the first state and the second state, the deletable processed data of the processed data temporarily stored in the memory store for each of the plurality of the third data ranges,

wherein the memory store includes third data range management information for managing each of the plurality of the third data ranges, and

the data management method further includes;

receiving, by the memory store control module, an input of a first parameter for specifying a write timing of the processed data to the database and the third data range to be written;

referring, by the memory store control module, after the write timing has passed, to the third data range management information based on the first parameter to specify the third data range to be written from the plurality of the third data ranges;

obtaining, by the memory store control module, the processed data included in the specified third data range from the memory store;

transmitting, by the memory store control module, to the database management module, the write request including the obtained processed data;

updating, by the memory store control module, the first state corresponding to the specified third data range to a state indicating that writing of the processed data to the database is complete;

referring, by the memory store control module, to the storage data management information to retrieve the third data range for which writing of the processed data to the database is complete, and that is not being used by the second data processing module; and

deleting, by the memory store control module, from the memory store, the processed data included in the retrieved third data range,

wherein the memory store includes first data range management information for managing a starting point time stamp, which is a time stamp of the processed data serving as a starting point of the first data range, and a finishing point time stamp, which is a time stamp of the processed data serving as a finishing point of the first data range, and

the data management method further includes;

receiving, by the memory store control module, an input of a second parameter for designating the predetermined time for specifying the first data range;

updating, by the memory store control module, the starting point time stamp of the first data range management information to the time stamp of the processed data included in the write request to write the processed data of the write request received from the first data processing module;

specifying, by the memory store control module, the oldest time stamp from the processed data included in the time range extending a predetermined period back in time from the starting point time stamp based on the starting point time stamp and the second parameter of a read request to read the processed data from the second data processing module;

updating, by the memory store control module, the finishing point time stamp of the first data range management information to the specified oldest time stamp;

referring, by the memory store control module, to the first data range management information to determine whether or not the third data range including the processed data corresponding to the pre-update finishing point time stamp and the third data range including the processed data corresponding to the updated finishing point time stamp match;

referring, by the memory store control module, to the storage data management information when there is a determination that the third data range including the processed data corresponding to the pre-update finishing point time stamp and the third data range including the processed data corresponding to the updated finishing point time stamp do not match;

updating, by the memory store control module, the second state of the third data range including the processed data corresponding to the pre-update finishing point time stamp to a state indicating that the processed data included in the third data range is not being used;

referring, by the memory store control module, to the storage data management information to retrieve the third data range for which writing of the processed data to the database is complete, and that is not being used by the second data processing module; and

deleting, by the memory store control module, from the memory store, the processed data included in the retrieved third data range.

4. The data management method according to claim 3,

wherein the computer system receives a plurality of types of data from the external apparatus, and

wherein the memory store control module holds the first data range management information, the third data range management information, and the storage data management information for each of the plurality of types of data.

5. A computer, comprising:

a processor;

a memory coupled to the processor;

a network interface coupled to the processor;

a memory store constructed on a storage area of the memory, the memory store being configured to temporarily store data including a time stamp; and

a memory store control module configured to manage the data stored in the memory store and to control read processing and write processing to and from the memory store,

the memory store control module being configured to;

specify a first data range, which is a time range of data to be written to an external database, from the data stored in the memory store;

obtain the data included in the specified first data range from the memory store;

transmit to another computer configured to manage the database a write request to write the obtained data to the database; and

specify deletable data of the data temporarily stored in the memory store based on the first data range and a second data range, which is a time range of data handled by the another computer accessing the memory store,

wherein the first data range is a time range advanced by a predetermined period from a time point to be a predetermined reference point,

wherein the second data range is a time range extending a predetermined period back in time from the newest time stamp of the data,

wherein the memory store includes, for each of a plurality of the first data ranges, storage data management information for managing a first state indicating a write state of the data to the database and a second state indicating a usage state of the data by the another computer accessing the memory store, and

wherein the memory store control module is configured to;

refer to the storage data management information, and

specify, based on the first state and the second state, the deletable data of the data temporarily stored in the memory store for each of the plurality of the first data ranges,

wherein the memory store includes first data range management information for managing each of the plurality of the first data ranges, and

wherein the memory store control module is configured to;

receive an input of a first parameter for specifying a write timing of the data to the database and the first data range to be written;

refer, after the write timing has passed, to the first data range management information based on the first parameter to specify the first data range to be written from the plurality of the first data ranges;

obtain the data included in the specified first data range from the memory store;

transmit to the another computer configured to manage the database the write request including the obtained data;

update the first state corresponding to the specified first data range to a state indicating that writing of the data to the database is complete;

refer to the storage data management information to retrieve the first data range for which writing of the data to the database is complete, and that is not being used by the another computer accessing the memory store; and

delete from the memory store the data included in the retrieved first data range,

wherein the memory store comprises second data range management information for managing a starting point time stamp, which is a time stamp of the data serving as a starting point of the second data range, and a finishing point time stamp, which is a time stamp of the data serving as a finishing point of the first data range, and

wherein the memory store control module is configured to;

receive an input of a second parameter for designating the predetermined time for specifying the second data range;

update the starting point time stamp of the second data range management information to the time stamp of the data included in the write request to write the data of the write request;

specify the oldest time stamp from the data included in the time range extending a predetermined period back in time from the starting point time stamp based on the starting point time stamp and the second parameter in a case of being received a read request to read the data stored in the memory store;

update the finishing point time stamp of the second data range management information to the specified oldest time stamp;

refer to the second data range management information to determine whether or not the first data range including the data corresponding to the pre-update finishing point time stamp and the first data range including the data corresponding to the updated finishing point time stamp match;

refer to the storage data management information in a case where it is determined that the first data range including the data corresponding to the pre-update finishing point time stamp and the first data range including the data corresponding to the updated finishing point time stamp do not match;

update the second state of the first data range including the data corresponding to the pre-update finishing point time stamp to a state indicating that the data included in the first data range is not being used;

refer to the storage data management information to retrieve the first data range for which writing of the data to the database is complete, and that is not being used by the another computer accessing the memory store; and

delete from the memory store the data included in the retrieved first data range.

6. The computer according to claim 5, wherein the memory store control module holds the first data range management information, the second data range management information, and the storage data management information for each of types of data.