System and method for FBA formatted disk mapping and variable-length CKD formatted data record retrieval

• US 5,664,144 A
• Filed: 06/18/1996
• Issued: 09/02/1997
• Est. Priority Date: 09/24/1990
• Status: Expired due to Term

• Current Assignee: EMC Corporation (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Vishlitzky, Natan, Alterescu, Bruno, Yanai, Moshe, Castel, Daniel
• Primary Examiner(s): Kim, Matthew M.
• Application Number:
  US08/665,607
• Time in Patent Office: 441 Days
• Field of Search: 395/440, 395/439, 395/441, 395/415-419, 395/421.02
• US Class Current: 711/113
• CPC Class Codes: G06F 11/2066 Optimisation of the communication load
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2079 Bidirectional techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 2003/0697 device management, e.g. handlers, drivers, I/O schedulers
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G11B 20/1217 on discs
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 69/26 Special purpose or proprietary protocols or architectures network applications for proprietary or special purpose networking environments H04L67/12
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

An apparatus and method for disk mapping and data retrieval includes a data storage medium on which has been stored a plurality of data records. Each record includes at least a record identification portion, for uniquely identifying each record from among the plurality of data records. The apparatus builds a record locator table or high speed semiconductor memory which comprises the unique record identifiers for the records on the storage medium as well as a record locator index generated by the apparatus, which indicates the address of the data record on the storage medium. Data retrieval is facilitated by first searching the record locator table in high speed semiconductor memory for a requested data record. Utilizing the record locator index associated with the requested data record, the system directly accesses the requested data record on the storage medium thereby minimizing storage medium search time. Also disclosed is an apparatus and method for converting CKD formatted data records to FBA formatted disk drives and for building and compressing the "count" portion of the CKD data formatted record into a record locator table.

Claims

1. Apparatus for retrieving a requested variable-length data record stored on fixed block architecture (FBA) formatted storage medium comprising:

means for receiving a plurality of variable-length data records, each of said variable-length data records including at least a record identification portion and a data portion;

means, responsive to said means for receiving, for transforming and storing said plurality of variable-length data records on said FBA formatted storage medium;

means, responsive to said means for transforming and storing, for generating a corresponding plurality of record locator indices, each of said corresponding plurality of record locator indices being associated with one of said plurality of variable-length data records and a corresponding record identification portion, for uniquely identifying the location of each of said plurality of variable-length data records stored on said FBA formatted storage medium;

memory storage means, for storing at least said plurality of record locator indices and the associated record identification portions in a record locator table;

means for requesting access to said requested variable-length data record, and for providing the record identification portion corresponding to said requested variable-length data record;

means, responsive to said means for requesting access, for searching said record locator table stored in said memory storage means, for locating an associated one of said record locator indices corresponding to said requested variable-length data record;

record retrieval means, responsive to said means for searching, for retrieving from said FBA formatted storage medium said requested variable-length data record as directed by said associated one of said record locator indices corresponding to said requested variable-length data record; and

wherein said means for generating includes means for transforming and encoding a combination of said record locator indices and associated record identification portions to produce encoded information stored in said record locator table in said memory storage means, so that the encoded information in said record locator table is reduced in length in comparison to said record locator indices and associated record identification portions.

2. The apparatus as claimed in claim 1, wherein said means for receiving includes means for receiving said plurality of variable-length data records in count-key-data (CKD) format.

3. The apparatus as claimed in claim 1, wherein said FBA formatted storage medium has a plurality of fixed-length blocks of storage locations, and said means for transforming and storing said plurality of variable-length data records on said FBA formatted storage medium includes means for aligning a beginning of each of said data portions of each of said variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations.

4. The apparatus as claimed in claim 3, wherein said associated one of said record locator indices corresponding to said requested variable-length data record specifies the respective one of said fixed-length blocks of storage locations storing a beginning portion of said requested variable-length data record.

5. The apparatus as claimed in claim 4, wherein said record retrieval means includes means responsive to information in said record locator table for computing a number of the fixed-length blocks of storage locations that are read to retrieve all of the data portion for said requested variable-length data record.

6. The apparatus as claimed in claim 1, wherein the record identification portion contains a record number portion that usually identifies a record number from among a number of sequentially numbered data records, and wherein the system includes means for determining whether the record number portion of the record identification portion identifies a record number from among a number of sequentially numbered data records.

7. The apparatus as claimed in claim 6, including means for incrementing a record number for a next expected data record by one.

8. The apparatus as claimed in claim 1 wherein said record locator table includes a track-level record locator table, and wherein said means for searching said record locator table includes means for computing an address for said requested variable-length data record from length information stored in said track-level record locator table for intervening variable-length data records in a track including said requested variable-length data record, said intervening variable-length data records occurring in the track before said requested variable-length data record.

9. The apparatus as claimed in claim 1 wherein said memory storage means includes semiconductor memory, said FBA formatted storage medium includes a plurality of FBA formatted disk drives, and said record retrieval means and said means for transforming and storing retrieves and stores at least one variable-length data record in a cache in said semiconductor memory;

one of said record locator indices is associated with said at least one variable-length data record and includes means for indicating that the variable-length data record is in cache, and includes a pointer to a memory address of said at least one variable-length data record in cache, and includes means for indicating that the variable-length data record in cache is awaiting writing to said FBA formatted storage medium.

10. The apparatus as claimed in claim 1, wherein said record locator table includes a track-level record locator table, and wherein said means for transforming and encoding encodes said associated one of said record locator indices corresponding to said requested variable-length data record in portions of said encoded information associated with intervening variable-length data records in a track including said requested data record, said intervening variable-length data records occurring in the track before said requested variable-length data record.

11. The apparatus as claimed in claim 1, wherein said FBA formatted storage medium has a plurality of fixed-length blocks of storage locations, said means for transforming and storing said plurality of variable-length data records on said FBA formatted storage medium includes means for aligning a beginning of each of said data portions of each of said plurality of variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations;

and a number of the fixed-length blocks of storage locations that are used to store the data portion of each variable-length data record is specified by the portion of the encoded information associated with said each intervening variable-length data record.

12. The apparatus as claimed in claim 1, wherein said means for receiving includes means for receiving said plurality of variable-length data records in count-key-data (CKD) format, each of said variable-length data records in CKD format having a count portion including said record identification portion and length information, and wherein said means for transforming and encoding transforms and encodes the count portion for each of said variable-length data records so that the count portion is reduced in length and the entire count portion for each of said variable-length data records can be decoded from the encoded information.

13. The apparatus as claimed in claim 1, wherein said means for transforming and encoding includes means for producing a bit mask of bits indicating respective bytes of a count portion for a variable-length CKD data record that are different from corresponding bytes of an expected count portion.

14. The apparatus as claimed in claim 1, wherein said means for transforming and encoding encodes whether or not a variable-length data record having a preceding record in a track has the same length information as the preceding record in the track.

15. The apparatus as claimed in claim 1, wherein said means for transforming and encoding includes means for substituting in said encoded information a pattern identifier representing length information which is the same for a multiplicity of said variable-length data records.

16. The apparatus as claimed in claim 1, wherein said record retrieval means retrieves the requested variable-length data record from said FBA formatted storage medium with no intervening search of said FBA formatted storage medium.

17. The apparatus as claimed in claim 1, wherein said FBA formatted storage medium includes an FBA formatted disk drive.

18. The apparatus as claimed in claim 1, wherein said FBA formatted storage medium includes semiconductor memory.

19. Apparatus for transforming and mapping variable-length CKD formatted data records onto fixed block disk drives, and for retrieving a requested data record stored on a fixed block disk drive, comprising:

means for receiving a plurality of variable-length CKD formatted data records, each of said variable-length CKD formatted data records including at least a record identification portion and a data portion;

means, responsive to said means for receiving, for transforming the plurality of variable-length CKD formatted data records to a fixed block format, and for storing the plurality of fixed block format data records on one or more fixed block disk drives;

means, responsive to said means for transforming and storing, for generating a plurality of record locator indices, each of said plurality of record locator indices associated with one of said plurality of data records, for uniquely identifying the location of each of said plurality of data records stored on said one or more fixed block disk drives, and also including means for transforming and encoding said plurality of record locator indices and record identification portions to produce encoded information reduced in length in comparison to the length of said record identification portions and record locator indices;

semiconductor memory, for storing said encoded information in a record locator table;

means for requesting access to said requested data record, and for providing a record identification portion corresponding to said requested data record;

mean, responsive to said means for requesting access to said requested data record, for searching said record locator table stored in said semiconductor memory, for decoding from said encoded information an associated one of said record locator indices corresponding to said requested data record; and

record retrieval means, responsive to said means for searching, for retrieving from said one or more fixed block disk drives, said requested data record as directed by said associated one of said record locator indices corresponding to said requested data record.

20. The apparatus as claimed in claim 19, wherein each of said fixed block disk drives has a plurality of fixed-length blocks of storage locations, and said means for transforming and storing includes means for aligning a beginning of each of said data portions of each of said variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations;

said associated one of said record locator indices corresponding to said requested data record specifies the respective one of said fixed-length blocks of storage locations storing a beginning portion of said requested data record; and

said record retrieval means includes means responsive to information in said record locator table for computing a number of the fixed-length blocks of storage locations that are read to retrieve all of the data portion for said requested data record.

21. The apparatus as claimed in claim 19, wherein the record identification portion contains a record number portion that usually identifies a record number from among a number of sequentially numbered data records, and wherein the system includes means for determining whether the record number portion of the record identification portion identifies a record number from among a number of sequentially numbered data records.

22. The apparatus as claimed in claim 21, including means for incrementing a record number for a next expected data record by one.

23. The apparatus as claimed in claim 19, wherein said record locator table includes a track-level record locator table, and wherein said means for searching said record locator table includes means for computing an address for said requested data record from length information stored in said track-level record locator table for intervening data records in a track including said requested data record, said intervening data records occurring in the track before said requested data record.

24. The apparatus claimed in claim 19, wherein said record retrieval means and said means for transforming and storing retrieves and stores at least one variable-length data record in a cache in said semiconductor memory;

and

wherein at least one of said record-locator indices further includes means for indicating that the variable-length data record associated with said at least one of said record locator indices is stored in said cache;

said at least one of said record locator indices further includes a pointer to a memory address of the associated data record stored in said cache; and

said at least one of said record locator indices further includes means for indicating that the associated variable-length data record stored in said cache is awaiting writing to one or more of said fixed-block disk drives.

25. The apparatus as claimed in claim 19, wherein said record locator table includes a track-level record locator table, and wherein said means for producing encoded information encodes said associated one of said record locator indices corresponding to said requested data record in portions of said encoded information associated with intervening variable-length data records in a track including said requested data record, said intervening variable-length data records occurring in the track before said requested data record.

26. The apparatus as claimed in claim 19, wherein each of said fixed block disk drives has a of fixed-length blocks of storage locations, said means for transforming and storing said plurality of variable-length data records on said fixed block disk drives includes means for aligning a beginning of each of said data portions of each of said variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations;

and a number of the fixed-length blocks of storage locations that are used to store the data portion of each variable-length data record is specified by the portion of the encoded information associated with said each variable-length data record.

27. The apparatus as claimed in claim 19, wherein each of said variable-length records in CKD format has a count portion including said record identification portion and length information, and wherein the means for transforming and encoding transforms and encodes the count portion for each of said variable-length records in CKD format so that the count portion is reduced in length and the entire count portion can be decoded from the encoded information.

28. The apparatus as claimed in claim 19, wherein said means for transforming and encoding includes means for producing a bit mask of bits indicating respective bytes of a count portion for a variable-length CKD record that are different from corresponding bytes of an expected count portion.

29. The apparatus as claimed in claim 19, wherein said means for transforming and encoding includes means for encoding whether or not a variable-length record having a preceding record in a track has the same length information as the preceding record in the track.

30. The apparatus as claimed in claim 19, wherein said means for producing the encoded information includes means for substituting in said encoded information identifier representing length information which is the same for a multiplicity of the variable-length data records.

31. The apparatus as claimed in claim 19, wherein said record retrieval means retrieves the requested data record from said one or more fixed-block disk drives with no intervening search of said one or more fixed-block disk drives.

32. A method for transforming and storing variable-length CKD formatted data records onto one or more fixed block architecture (FBA) disk drives, and for retrieving a requested data record stored on one or more of said fixed block architecture disk drives, comprising the steps of:

receiving a plurality of variable-length CKD formatted data records, each of said variable-length CKD formatted data records including at least a record identification portion and a data portion;

storing to said one or more fixed block architecture (FBA) disk drives, said plurality of variable-length CKD formatted data records, said step of storing including;

transforming said plurality of variable-length CKD formatted data records from a CKD format to an FBA format, and storing the plurality of data records in the FBA format on said one or more FBA disk drives;

generating a plurality of record locator indices;

each of the plurality of record locator indices corresponding to one of the plurality of data records, for uniquely identifying the location of each of the plurality of data records stored on said one or more FBA disk drives;

transforming and encoding the record locator indices and record identification portions to produce encoded information in a record locator table, said encoded information being reduced in length in comparison to said record locator indices and associated record identification portions;

storing said record locator table in memory;

searching said record locator table stored in memory, for locating one of the record locator indices corresponding to said requested data record; and

retrieving from said one or more FBA disk drives, said requested data record as directed by said one of the record locator indices corresponding to said requested data record.

33. The method as claimed in claim 32, wherein said record locator table includes a track-level record locator table, and wherein said searching said record locator table includes computing an address for said requested data record from length information stored in said track-level record locator table for intervening data records in a track including said requested data record, said intervening data records occurring in the track before said requested data record.

34. The method as claimed in claim 32, wherein said FBA disk drives have fixed-length blocks of storage locations, and said transforming said plurality of variable-length data records from a CKD format to an FBA format includes aligning a beginning of each of said data portions of each of said variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations.

35. The method as claimed in claim 32, wherein said requested data record is retrieved from said one or more FBA disk drives with no intervening search of said one or more FBA disk drives.

36. A method for transforming and storing variable-length data records onto fixed block architecture (FBA) formatted storage medium, and for retrieving a requested data record stored on said FBA formatted storage medium, comprising the steps of:

receiving a plurality of variable-length data records, each of said variable-length data records including at least a record identification portion and a data portion;

storing to said FBA formatted storage medium, said plurality of variable-length data records, said step of storing including;

transforming said plurality of variable-length data records to an FBA format, and storing the plurality of data records in the FBA format on said FBA formatted storage medium;

generating a plurality of record locator indices;

each of the plurality of record locator indices corresponding to one of the plurality of data records, for uniquely identifying the location of each of the plurality of data records stored on said FBA formatted storage medium;

transforming and encoding the record locator indices and record identification portions to produce encoded information in a record locator table, said encoded information being reduced in length in comparison to said record locator indices and associated record identification portions; and

storing said record locator table in memory;

searching said record locator table stored in memory, for locating one of the record locator indices corresponding to said requested data record; and

retrieving from said FBA formatted storage medium, said requested data record as directed by said one of the record locator indices corresponding to said requested data record.

37. The method as claimed in claim 36, wherein said record locator table includes a track-level record locator table, and wherein said searching said record locator table includes computing an address for said requested data record from length information stored in said track-level record locator table for intervening data records in a track including said requested data record, said intervening data records occurring in the track before said requested data record.

38. The method as claimed in claim 36, wherein said FBA formatted storage medium has fixed-length blocks of storage locations, and said transforming said plurality of variable-length data records to an FBA format includes aligning a beginning of each of said data portions of each of said variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations.

39. The method as claimed in claim 36, wherein said requested data record is retrieved from said FBA formatted storage medium with no intervening search of said EBA formatted storage medium.

40. The method as claimed in claim 36, wherein said transforming and encoding includes producing a bit mask of bits indicating respective bytes of a count portion for a variable-length CKD data record that are different from corresponding bytes of an expected count portion.

41. The method as claimed in claim 36, wherein said transforming and encoding encodes whether or not a variable-length data record having a preceding record in a track has the same length information as the preceding record in the track.

42. The method as claimed in claim 36, wherein said transforming and encoding substitutes in said encoded information a pattern identifier representing length information which is the same for a multiplicity of said variable-length data records.

43. Apparatus for retrieving a requested variable-length data record stored on fixed block architecture (FBA) formatted storage medium comprising:

means for receiving a plurality of variable-length data records;

means, responsive to said means for receiving, for transforming and storing said plurality of variable-length data records on said FBA formatted storage medium;

means, responsive to said means for transforming and storing, for generating a corresponding plurality of record locator indices, each of said corresponding plurality of record locator indices being associated with one of said plurality of variable-length data records, for uniquely identifying the location of each of said plurality of variable-length data records stored on said FBA formatted storage medium;

mapping means, for storing at least said plurality of record locator indices in a record locator table, and in response to a request for access to said requested variable-length data record, for searching said record locator table for obtaining an associated one of said record locator indices corresponding to said requested variable-length data record;

record retrieval means for retrieving from said FBA formatted storage medium said requested variable-length data record as directed by said associated one of said record locator indices corresponding to said requested variable-length data record; and

wherein said means for generating includes means for transforming and encoding a combination of said record locator indices and associated record identification portions to produce encoded information stored in said record locator table, so that the encoded information in said record locator table is reduced in length in comparison to said record locator indices and associated record identification portions.

44. The apparatus as claimed in claim 43, wherein said FBA formatted storage medium has a plurality of fixed-length blocks of storage locations, and said means for-transforming and storing said plurality of variable-length data records on said FBA formatted storage medium includes means for aligning a beginning of a data portion of each of said variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations.

45. The apparatus as claimed in claim 44, wherein said associated one of said record locator indices corresponding to said requested variable-length data record specifies the respective one of said fixed-length blocks of storage locations storing a beginning portion of said requested variable-length data record.

46. The apparatus as claimed in claim 44, wherein said record retrieval means includes means responsive to information in said record locator table for computing a number of the fixed-length blocks of storage locations that are read to retrieve all data for said requested variable-length data record.

47. The apparatus as claimed in claim 43, wherein said record locator table includes a track-level record locator table, and wherein said mapping means includes means for computing an address for said requested variable-length data record from length information stored in said track-level record locator table for intervening variable-length data records in a track including said requested variable-length data record, said intervening variable-length data records occurring in the track before said requested variable-length data record.

48. Apparatus for transforming, storing, and mapping variable-length data records on fixed block architecture (FBA) formatted storage medium comprising:

means for receiving a plurality of variable-length data records, each of said variable-length data records including at least a record identification portion and a data portion;

means, responsive to said means for receiving, for transforming and storing said plurality of variable-length data records on said FBA formatted storage medium;

means, responsive to said means for transforming and storing, for generating a corresponding plurality of record locator indices, each of said corresponding plurality of record locator indices being associated with one of said plurality of variable-length data records and a corresponding record identification portion, for uniquely identifying the location of each of said plurality of variable-length data records stored on said FBA formatted storage medium;

memory storage means, for storing at least said plurality of record locator indices and the associated record identification portions in a record locator table;

means, responsive to a request for access to a specified variable-length data record, for searching said record locator table for obtaining an associated one of said record locator indices corresponding to said specified variable-length data record, and retrieving from said FBA formatted storage medium said specified variable-length data record as directed by said associated one of said record locator indices corresponding to said specified variable-length data record; and

wherein said means for generating includes means for transforming and encoding a combination of said record locator indices and associated record identification portions to produce recorded information stored in said record locator table in said memory storage means, so that the encoded information in said record locator table is reduced in length in comparison to said record locator indices and associated record identification portions.

49. The apparatus as claimed in claim 48, wherein said FBA formatted storage medium has a plurality of fixed-length blocks of storage locations, and said means for transforming and storing said plurality of variable-length data records on said FBA formatted storage medium includes means for aligning a beginning of each of said data portions of each of said variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations.

50. The apparatus as claimed in claim 48, wherein the record identification portion contains a record number portion that usually identifies a record number from among a number of sequentially numbered data records, and wherein the system includes means for determining whether the record number portion of the record identification portion identifies a record number from among a number of sequentially numbered data records.

51. The apparatus as claimed in claim 50, including means for incrementing a record number for a next expected data record by one.

52. The apparatus as claimed in claim 48, wherein said record locator table includes a track-level record locator table, and wherein said means for transforming and encoding encodes said associated one of said record locator indices corresponding to said requested variable-length data record in portions of said encoded information associated with intervening variable-length data records in a track including said requested data record, said intervening variable-length data records occurring in the track before said requested variable-length data record.

53. The apparatus as claimed in claim 52, wherein said FBA formatted storage medium has a plurality of fixed-length blocks of storage locations, said means for transforming and storing said plurality of variable-length data records on said FBA formatted storage medium includes means for aligning a beginning of each of said data portions of each of said plurality of variable-length data records on a beginning of a respective one of said fixed-length blocks of storage locations;

and a number of the fixed-length blocks of storage locations that are used to store the data portion of each variable-length data record is specified by the portion of the encoded information associated with said each intervening variable-length data record.

54. The apparatus as claimed in claim 48, wherein said means for receiving includes means for receiving said plurality of variable-length data records in count-key-data (CKD) format, each of said variable-length data records in CKD format having a count portion including said record identification portion and length information, and wherein said means for transforming and encoding transforms and encodes the count portion for each of said variable-length data records so that the count portion is reduced in length and the entire count portion for each of said variable-length data records can be decoded from the encoded information.

55. The apparatus as claimed in claim 48, wherein said means for transforming and encoding includes means for producing a bit mask of bits indicating respective bytes of a count portion for a variable-length CKD data record that are different from corresponding bytes of an expected count portion.

56. The apparatus as claimed in claim 48, wherein said means for transforming and encoding encodes whether or not a variable-length data record having a preceding record in a track has the same length information as the preceding record in the track.

57. The apparatus as claimed in claim 48, wherein said means for transforming and encoding includes means for substituting in said encoded information a pattern identifier representing length information which is the same for a multiplicity of said variable-length data records.