Partial recall of deduplicated files

• US 8,645,335 B2
• Filed: 12/16/2010
• Issued: 02/04/2014
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Abhishek, Kalach, Ran, Cheung, Chun Ho, Benton, James Robert, Pfenning, Joerg-Thomas
• Primary Examiner(s): LE, UYEN T
• Application Number:
  US12/970,848
• Publication Number:
  US 20120158675A1
• Time in Patent Office: 1,146 Days
• Field of Search: None
• US Class Current: 707/692
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641

Abstract

The subject disclosure is directed towards changing a file from a fully deduplicated state to a partially deduplicated state in which some of the file data is deduplicated in a chunk store, and some is recalled into the file, that is, in the file'"'"'s storage volume. A partial recall mechanism such as in a file system filter tracks (e.g., via a bitmap in a file reparse point) whether file data is maintained in the chunk store or has been recalled to the file. Data is recalled from the chunk store as needed, and committed (e.g., flushed) to the file. Also described is efficiently returning the file to a fully deduplicated state by using the tracking information to determine which parts of the file are already deduplicated into the chunk store so as to avoid their further deduplication processing.

Claims

1. In a computing environment, a method performed at least in part on at least one processor, comprising, changing a state of a file of a storage volume from a fully deduplicated state to a partially deduplicated state, including reading from a chunk store one or more chunks of the file'"'"'s data, committing at least part of the one or more chunks to a stable storage as one or more recalled ranges of data of that file, and maintaining information in association with the file that tracks which range or ranges of data have been recalled, and which range or ranges of data reside in the chunk store.

2. The method of claim 1 wherein changing the state comprises recalling the one or more ranges of data for modification by a program or user write.

3. The method of claim 1 wherein reading from the chunk store obtains more data than is needed to satisfy a request, and wherein committing comprises writing the data that is needed to satisfy the request plus additional data to fill space allocated by the file system up to beginning and ending allocation boundaries.

4. The method of claim 1 wherein reading from the chunk store obtains one or more chunks than are needed to satisfy a request, and wherein committing comprises writing the chunks that are needed to satisfy the request plus additional data from one or more unneeded chunks to fill space allocated by the file system up to beginning and ending allocation boundaries.

5. The method of claim 1 wherein the file is divided into partitions corresponding to the recalled ranges and the ranges that reside in the chunk store, and wherein maintaining the information in association with the file comprises maintaining a bitmap-related structure in an auxiliary structure associated with that file, the bitmap-related structure having data that indicates whether each partition corresponds to a recalled range or corresponds to a range that resides in the chunk store.

6. The method of claim 5 further comprising, updating the bitmap-related structure after a recalled range is committed, and committing the file to commit the bitmap-related structure after updating.

7. The method of claim 5 further comprising, representing a plurality of partitions with a single bit in the bitmap structure.

8. The method of claim 5 further comprising, compressing the data of the bitmap structure.

9. The method of claim 8 further comprising, reading at least one chunk into the file as a recalled chunk to increase compression efficiency.

10. The method of claim 1 further comprising receiving write data, and wherein reading from the chunk store comprises obtaining zero or more bytes of data before the write data as needed to append data to the request with respect to one recall boundary, and obtaining zero or more bytes of data after the write data up as needed to append data to the request with respect to another recall boundary.

11. The method of claim 1 further comprising receiving a write request comprising write data, providing the write request to the file system, waiting for the write request to complete at the file system, obtaining zero or more bytes of data before the write data as needed to append data to the request with respect to one recall boundary, and obtaining zero or more bytes of data after the write data up as needed to append data to the request with respect to another recall boundary, committing the write data and any appended data to the stable storage, and updating the information that tracks which range or ranges of data have been recalled.

12. The method of claim 1 further comprising, changing the state of the file from the partially deduplicated state back to a fully deduplicated state, including using the information maintained in association with the file to determine which range or ranges of data reside in the chunk store and do not need to be processed for deduplication.

13. The method of claim 1 further comprising, preparing for committing the recalled data, including detecting an end-of-file boundary, and using a paging I/O write for committing the recalled data so as not to increase file size.

14. In a computing environment, a system comprising at least one processor, a memory communicatively coupled to the at least one processor and including components, comprising, a partial recall mechanism configured to access associated tracking data that indicates which range or ranges of a file have been recalled to the file, and which ranges reside on a chunk store and are referenced by the file, the partial recall mechanism further configured to access one or more chunks of file data in the chunk store, commit one or more ranges corresponding to at least part of the one or more chunks to the file as one or more recalled ranges, and update the tracking data to indicate when a range has been committed as a recalled range.

15. The system of claim 14 wherein the partial recall mechanism is incorporated into a file system filter.

16. The system of claim 14 wherein the tracking data comprises a bitmap-related structure corresponding to a bitmap containing values that indicate whether each range is a recalled range or a range that resides in the chunk store.

17. The system of claim 14 wherein the bitmap-related structure is maintained in a metadata store, or on the file in a reparse point buffer, or on the file in a named stream, or in another file metadata buffer provided by the file system in association with the file.

18. The system of claim 14 wherein the tracking data is accessed during re-deduplication processing to determine whether a range of data already resides as a deduplicated range in the chunk store.

19. One or more computer-readable storage media having computer-executable instructions, which when executed perform steps, comprising:

receiving a request to access a range of file data of a file at a given offset, in which at least some of the requested file data resides in a deduplication chunk store;

reading from the chunk store, and writing to storage, until at least the range of file data at the given offset is obtained;

committing at least part of the file data to the file as one or more partially recalled ranges;

updating tracking information that indicates that the one or more partially recalled ranges have been committed to the file; and

forwarding the request towards a file system.

20. The one or more computer-readable storage media of claim 19 having further computer-executable instructions comprising, detecting that the entire file data has been recalled to the file, and changing the state of the file to a non-deduplicated file, including deleting the tracking information, and marking the file as non-sparse if the file was not originally a sparse file.