Data management apparatus and method for efficiently generating a blocked transposed file and converting that file using a stored compression method

• US 6,725,225 B1
• Filed: 05/18/2000
• Issued: 04/20/2004
• Est. Priority Date: 09/29/1999
• Status: Expired due to Term

• Current Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Inventors: Kori, Mitsunori
• Primary Examiner(s): Homere, Jean R
• Assistant Examiner(s): Wong, Leslie
• Application Number:
  US09/572,938
• Time in Patent Office: 1,433 Days
• Field of Search: 707/101, 707/1, 707/205, 708/203, 345/716-726
• US Class Current: 707/693
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0674 Disk device
• CPC Class Codes: H03M 7/40 Conversion to or from variable length codes, e.g. Shannon-Fano code, Huffman code, Morse code
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A data management apparatus and method provide for a virtual conversion by repeatedly reading at least one record from an input file having a plurality of records, each record including a plurality of fields, and adding the at least one record to a buffer. The records in the buffer are then converted into a post-conversion block on a field-by-field basis until data size of post-conversion blocks for each kind of field of the records in the buffer exceeds a threshold size. The number of records in the post-conversion blocks can be determined by counting the records in the buffer during a process immediately preceding a time when the data size exceeds the threshold size, and storing that number. The conversion is completed by reading out records of the stored number of records from the input file, converting the records read out into post-conversion blocks on a field-by-field basis, and storing the post-conversion blocks.

Claims

1. A data management apparatus comprising:

number of records in group determining means for virtual conversion by repeatedly reading at least one record from an input file having a plurality of records, each record including a plurality of fields, adding the at least one record to a buffer, and converting records in the buffer into a post-conversion block on a field-by-field basis until data size of post-conversion blocks for each kind of field of the records in the buffer exceeds a threshold size, wherein the number of records in group determining means determines the number of records of the post-conversion block by counting the records in the buffer, during a process immediately preceding a time when the data size exceeds the threshold size, and stores that number;

first conversion means for generating a first block by dividing at least one record including a plurality of fields into the fields and combining fields of like kind; and

second conversion means for converting the first block into a second block using a stored data conversion method, and for storing the second block, wherein the second conversion means selects one data compression algorithm that provides highest compression efficiency from a plurality of stored data compression algorithms in accordance with properties of the first block, and converts the first block into the second block using the selected data compression algorithm.

2. The data management apparatus according to claim 1, wherein the second conversion means stores type information characterizing the data compression algorithm that has been used to convert the first block into the second block.

3. The data management apparatus according to claim 2, wherein the second conversion means stores the type information and the second block in one storing means.

4. The data management apparatus according to claim 2, wherein the second conversion means stores data size of the second block.

5. The data management apparatus according to claim 2, further comprising inverse conversion means for inversely converting the second block into the first block by referring to the type information that has been stored.

8. The data management apparatus according to claim 1, wherein the second block has a variable length.

6. A data management method comprising:

a virtual conversion, repeatedly reading at least one record from an input file having a plurality of records, each record including a plurality of fields, adding the at least one record to a buffer, and converting records in the buffer into a post-conversion block on a field-by-field basis until data size of post-conversion blocks for each kind of field of the records in the buffer exceeds a threshold size;

determining the number of records of the post-conversion block by counting the records in the buffer during a process immediately preceding a time when the data size exceeds the threshold size, and storing that number; and

reading out records of the stored number of records from the input file, converting the records read out into post-conversion blocks on a field-by-field basis, and storing the post-conversion blocks.

7. The data management method according to claim 6, wherein the virtual conversion comprises:

generating a first block by converting a record in the buffer into a converted record having a fixed-length field format, dividing the converted record into fields, and combining fields of like kind; and

converting the first block into a second block using a stored data conversion method.