Optimized disk repository for the storage and retrieval of mostly sequential data

• US 7,882,081 B2
• Filed: 08/30/2002
• Issued: 02/01/2011
• Est. Priority Date: 08/30/2002
• Status: Expired due to Fees

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Chang, Yafen Peggy, Johnston, Craig Anthony, Trimmer, Don Alvin, Lau, Jerry Kai, Stager, Roger Keith
• Primary Examiner(s): Trujillo; James
• Assistant Examiner(s): Black; Linh
• Application Number:
  US10/231,617
• Publication Number:
  US 20040044705A1
• Time in Patent Office: 3,077 Days
• Field of Search: 707/8, 707/204, 707/205, 707/202, 707/4, 707 1-1041, 707/691, 707/703, 711/112, 711/114, 711/162, 714/6, 714/4
• US Class Current: 707/691
• CPC Class Codes: G06F 16/10 File systems; File servers

Abstract

A system for the storage and retrieval of data files includes a data disk, a configuration disk and a controller. Each data file includes one or more data blocks, each data block comprising a data section. The data disk stores the data blocks and is divided into a plurality of block groups. The configuration disk stores a copy of at least a portion of the corresponding meta-data. The controller controls both the data disk and the configuration disk.

Claims

1. A method comprising:

receiving a data container to be stored to a first nonvolatile storage device, wherein the first nonvolatile storage device contains a plurality of block groups;

selecting a block group from the plurality of block groups, wherein each of the plurality of block groups allows sequential allocation of data blocks for storing the data container to the first nonvolatile storage device;

storing data and associated metadata of the data container together to the selected block group of the plurality of block groups as a single write to contiguous data blocks within the selected block group;

storing a transactional record to a second nonvolatile storage device physically separate from the first nonvolatile storage device, wherein the transactional record corresponds to the single write and contains a header and a corresponding trailer, wherein the header and trailer contain a copy of a subset of the metadata of the data container, wherein the transactional record enables storage of a subset of the metadata in the second nonvolatile storage medium in addition to storage of a complete copy of the metadata in the first non-volatile storage medium, wherein the header is stored to the separate second nonvolatile storage device at a time before the storing of the data container to the first nonvolatile storage device, and the trailer is stored to the separate second nonvolatile storage device at a time after the storing of the data container to the first nonvolatile storage device, and wherein successfully storing the trailer to the second nonvolatile storage device indicates the write has completed without errors;

checking to determine whether the trailer was successfully stored in the second nonvolatile storage device;

committing the storing of the data container to the first nonvolatile only when the trailer has been determined to be successfully stored in the second nonvolatile storage device; and

nullifying the storing of the data container to the first nonvolatile storage device upon failure to successfully store the trailer to the second nonvolatile storage device, wherein failure to store the trailer to the second nonvolatile storage device indicates at least part of the data container has failed to be stored to the selected block group of the plurality of data blocks without errors.

2. The method as recited in claim 1, further comprising:

during disaster recovery, upon detecting the transactional record having the header without the corresponding trailer, rolling-back the storing of the data container to the plurality of data blocks to restore the first nonvolatile storage device to a consistent state.

3. The method as recited in claim 1, further comprising:

retrieving the plurality of data blocks in a single data stream by using a sequential change number (SCN) stored in each of the plurality of data blocks.

4. The method as recited in claim 1, wherein the block group is selected round-robin from the plurality of block groups.

5. The method as recited in claim 1, wherein the block group contains metadata of the first nonvolatile storage device, and the metadata can be used for recovery of the first nonvolatile storage device.

6. The method as recited in claim 1, wherein each of the plurality of block groups contains metadata of the first nonvolatile storage device allowing recovery of the first nonvolatile storage device upon recovery of the metadata from any one of the plurality of block groups.

7. The method as recited in claim 1, wherein the block group contains an identifier for deleted data containers, and the identifier identifies data blocks that have been previously used by the deleted data containers.

8. The method as recited in claim 1, wherein the header and a trailer are used for verifying the integrity of the corresponding data block.

9. The method as recited in claim 8, wherein the header and the trailer of the corresponding data block contain metadata capable of being used for recovery of the corresponding data block.

10. The method as recited in claim 8, wherein the header and the trailer of the corresponding data block contain metadata capable of being used for recovery of the data container stored on the corresponding data block.

11. The method as recited in claim 1, wherein data of the data container and metadata of the data container are stored in the corresponding data block in a single contiguous write transaction.

12. The method as recited in claim 1, wherein at least some of the metadata of the first nonvolatile storage device is duplicated in the configuration disk.

13. The method as recited in claim 1, wherein the second nonvolatile storage device stores data in a circular buffer fashion by overwriting outdated data with most recent data.

14. The method as recited in claim 1, wherein the header and the trailer of the transactional record contain metadata of the data container capable of being used for recovery of the data container.

15. The method as recited in claim 1, wherein the data container is a data file.

16. A system comprising:

a data disk for storing a data container, wherein the data disk contains a plurality of block groups, each of the plurality of block groups allows sequential allocation of data blocks for storing the data container to the data disk;

a configuration disk physically separate from the data disk for storing a transactional record corresponding to the storing of the data container to the data disk, wherein the transactional record contains a header and a corresponding trailer to be stored in the configuration disk separate from data of the data container stored in the data disk, wherein the header is stored to the separate configuration disk at a time before the storing of the data container to the data disk, and the trailer is stored to the separate configuration disk at a time after the storing of the data container to the data disk, wherein successfully storing the trailer to the configuration disk indicates .the write has completed without errors; and

a controller configured to;

select one of the plurality of block groups from the data disk;

sequentially allocate a plurality of data blocks from the selected block group;

to store data and associated metadata of the data container together to the plurality of data blocks as a single write to contiguous blocks within the selected one of the plurality of block groups;

to store the transactional record to the configuration disk wherein the transactional record corresponds to the single write and contains a header and a corresponding trailer, wherein the header and trailer contain a copy of a subset of the metadata of the data container, wherein the transactional record enables storage of a subset of the metadata in the configuration disk in addition to storage of a complete copy of the metadata in the separate data disk;

to check whether the trailer was successfully stored in the configuration disk; and

to commit the storing of the data container to the data disk only when the trailer of the has been successfully stored to the separate configuration disk, and

to nullify the storing of the data container to the data disk upon failure to successfully store the trailer to the configuration disk, wherein failure to successfully store the trailer to the configuration disk indicates at least part of the data container has failed to be stored to the selected block group of the plurality of data blocks without errors.

17. The system as recited in claim 16, wherein during disaster recovery, the controller rolls back the storing of the data container to the plurality of data blocks upon detecting the transactional record having the header without the corresponding trailer.

18. The system as recited in claim 16, wherein each of the plurality of block groups contains metadata of the data disk, and the metadata can be used for recovery of the data disk.

19. The system as recited in claim 16, wherein each of the plurality of data blocks contains metadata of the data container, and the metadata can be used for recovery of the data container.

20. The system as recited in claim 16, wherein the configuration disk contains metadata of the data disk, and the metadata can be used for recovery of the data disk.

21. A method comprising:

receiving a data container to be stored to a first non-volatile storage medium, wherein the first non-volatile storage medium contains a plurality of block groups;

selecting a block group from the plurality of block groups, wherein each of the plurality of block groups allows sequential allocation of data blocks for storing the data container to the first non-volatile storage medium;

storing data of the data container and associated metadata together to contiguous data blocks in the selected block group of the first non-volatile storage medium as a single contiguous write, the metadata including one or more of a UUID associated with the first non-volatile storage medium, an offset indicator associated with the data container, a sequence change number associated with the data container, a resource fork length associated with the storage container, or an indicator of unused memory within a given block group;

storing a transactional record to a second nonvolatile storage medium, wherein the second nonvolatile storage medium is physically separate from the first non-volatile storage medium and has access characteristics different from those of the first nonvolatile storage medium, wherein the transactional record contains a header and a corresponding trailer associated with the write, wherein the header and trailer contain a copy of a subset of the metadata of the data container, wherein the transactional record enables storage of a subset of the metadata in the second nonvolatile storage medium in addition to storage of a complete copy of the metadata in the first non-volatile storage medium, wherein the header is stored to the second nonvolatile storage medium at a time before the storing of the data container to the first nonvolatile storage medium and the trailer is stored to the second nonvolatile storage medium at a time after the storing of the data container to the first nonvolatile storage medium, and wherein successfully storing the trailer to the second nonvolatile storage medium indicates the write has completed without error;

checking to determine whether the trailer was successfully stored in the second nonvolatile storage medium; and

committing the storing of the data container and associated metadata to the first non-volatile storage medium only when the transactional record was successfully stored to the second non-volatile storage medium; and

nullifying the storing of the data container to the data disk upon failure to store the trailer to the second nonvolatile storage medium at the time after the storing of the data container to the data disk, wherein failure to store the trailer to the second nonvolatile storage medium indicates at least part of the data container has failed to be stored to the selected block group of the plurality of data blocks without errors.

22. The method as recited in claim 21, further comprising, during disaster recovery and upon detecting the transactional record having the header without the corresponding trailer, rolling-back the storing of the data container to the plurality of data blocks to restore the first non-volatile storage medium to a consistent state.

23. The method as recited in claim 22, wherein each of the plurality of block groups contains metadata of the first non-volatile storage medium allowing recovery of the first non-volatile storage medium upon recovery of the metadata from any one of the plurality of block groups.

24. The method as recited in claim 22, wherein the block group contains an identifier for deleted data containers, and the identifier identifies data blocks that have been previously used by the deleted data containers.

25. The method as recited in claim 22, wherein the header and the corresponding trailer are used for recovery of the data container stored in the corresponding data block.

26. The method as recited in claim 22, wherein the second non-volatile storage medium is a configuration disk that stores data in a circular buffer by overwriting outdated data with most recent data.

27. The method as recited in claim 22, wherein the data container is a data file.