File system image transfer

• US 7,174,352 B2
• Filed: 05/10/2001
• Issued: 02/06/2007
• Est. Priority Date: 06/03/1993
• Status: Expired due to Fees

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Hitz, David, Kleiman, Steven R., Malcolm, Michael, Rakitzis, Byron, Lau, James, O'Malley, Sean W., Harris, Guy
• Primary Examiner(s): LE, UYEN T
• Application Number:
  US09/854,187
• Publication Number:
  US 20020049718A1
• Time in Patent Office: 2,098 Days
• Field of Search: 707/203, 707/204, 707/205, 707/100, 707/101, 707/102, 707/103.R, 707/103.Y, 707/104.1, 711/156, 711/161, 711/162
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99954 Version management

Abstract

The invention provides a method and system for duplicating all or part of a file system while maintaining consistent copies of the file system. The file server maintains a set of snapshots, each indicating a set of storage blocks making up a consistent copy of the file system as it was at a known time. Each snapshot can be used for a purpose other than maintaining the coherency of the file system, such as duplicating or transferring a backup copy of the file system to a destination storage medium. In a preferred embodiment, the snapshots can be manipulated to identify sets of storage blocks in the file system for incremental backup or copying, or to provide a file system backup that is both complete and relatively inexpensive. Also in a preferred embodiment, shadow snapshots can be maintained, with a shadow snapshot including a set of member storage blocks that formed a consistent file system other than an active file system, with a set of selected member storage blocks removed from the consistent file system.

Claims

1. A file system that stores and retrieves information in a plurality of storage blocks, said file system including:

mass storage having said plurality of storage blocks; and

a controller that controls storage and retrieval of information, said information including data and a plurality of snapshots each including a plurality of bits associated with each one of said plurality of storage blocks, said controller using at least one of said plurality of bits in a first snapshot to identify whether said one storage block was part of said file system at a time earlier than a current consistent version of said file system, and said controller using at least a second one of said plurality of bits in a second snapshot to identify whether said one storage block was part of said file system at a second time earlier than a current consistent version of said file system.

2. A file system as in claim 1, including an element disposed for selecting storage blocks in response to said one bit and said second one bit associated with said selected storage blocks.

3. A file system as in claim 2, including an element disposed for copying said selected storage blocks to a destination.

4. A file system as in claim 3, wherein said destination includes:

a tape, a disk, a data structure in a second file system, a set of network messages, or a destination distributed over a plurality of file systems.

5. A file system as in claim 1, including an element disposed for selecting storage blocks in response to said one bit associated with said selected storage blocks.

6. A file system as in claim 5, including an element disposed for copying said selected storage blocks to a destination.

7. A file system as in claim 6, wherein said destination includes:

a tape, a disk, a data structure in a second file system, a set of network messages, or a destination distributed over a plurality of file systems.

8. A file system having a plurality of storage blocks, said file system including a snapshot including a plurality of bits associated with a set of member storage blocks selected from said plurality of storage blocks, said member storage blocks forming a consistent file system other than an active file system, said snapshot being disposed as an object in said file system, wherein said file system is responsive to at least one file system request with regard to said snapshot.

9. A file system as in claim 8, including

a mark on allocate image of a set of member storage blocks selected from said plurality, said member storage blocks having been added to said snapshot;

and

a storage image defined based on said snapshot and said mark on allocate image, said storage image indicating a set of member storage blocks selected from said plurality.

10. A file system as in claim 9, wherein said storage image is defined with regard to a logical sum operation on said snapshot and said mark on allocate image.

11. A file system as in claim 8, including

a mark on deallocate image of a set of member storage blocks selected from said plurality, said member storage blocks having been removed from said snapshot;

and

a storage image defined based on said snapshot and said mark on deallocate image, said storage image indicating a set of member storage blocks selected from said plurality.

12. A file system as in claim 8, including

a shadow snapshot of a set of member storage blocks selected from said plurality, said member storage blocks having formed a consistent file system other than an active file system, with a set of selected member storage blocks removed from said consistent file system;

and

a storage image defined based on said snapshot and said shadow snapshot, said storage image indicating a set of member storage blocks selected from said plurality.

13. A file system as in claim 8, including an indicator of which ones of said member storage blocks have been copied.

14. A file system as in claim 8, including a plurality of said snapshots;

wherein said plurality of said snapshots are associated with an array of bits, said array having one set of bits for each storage block in said plurality of storage blocks, said set of bits having at least one bit for each said snapshot.

15. A file system as in claim 8, wherein said file system can manipulate said snapshot without having to traverse a hierarchy of file system objects within said snapshot.

16. A file system as in claim 8, wherein said snapshot includes a data structure disposed in a format allowing for a set management operation to be performed efficiently.

17. A file system as in claim 8, wherein said snapshot includes an array of bits, said array having one bit for each storage block in said plurality.

21. A file system as in claim 8, wherein said storage image is a result of a logical sum or difference performed on said set of member storage blocks for a first of said snapshots and said set of member storage blocks for a second said storage image.

22. A file system as in claim 8, wherein said storage image is a result of a set management operation on said set of member storage blocks for any of said snapshots.

23. A file system as in claim 8, wherein each of said snapshots includes a data structure disposed in a format allowing for a set management operation to be performed in O(n) time or less, where n is a number of storage blocks in said plurality, without reading any contents of said storage blocks in said plurality.

24. A file system as in claim 23, wherein said set management operation is a logical sum or difference.

25. A file system as in claim 8, wherein each of said snapshots includes a data structure identifying which storage blocks in said plurality are member storage blocks of that snapshot.

26. A file system as in claim 25, wherein said data structure uses no more than 1/100th of an amount of storage required by said storage blocks in said plurality.

27. A file system as in claim 25, wherein said data structure uses no more than four bytes per storage block in said plurality.

18. A file system having a plurality of storage blocks, said file system including:

a plurality of snapshots, each snapshot including a plurality of bits associated with a set of member storage blocks selected from said plurality of storage blocks, said member storage blocks forming a consistent file system other than an active file system, each said snapshot being disposed as an object in said file system; and

a storage image determined based on said plurality of snapshots, said storage image defining another set of member storage blocks selected from said plurality of storage blocks;

wherein said file system is responsive to at least one file system request with regard to said snapshots.

19. A file system as in claim 18, including an indicator of which ones of said storage blocks in said storage image have been copied.

20. A file system as in claim 18, wherein said storage image is a result of a logical sum or difference performed on said set of member storage blocks for a first of said snapshots and said set of member storage blocks for a second of said snapshots.

28. A method of operating a file system, said method including:

forming a first snapshot of a first consistent state of said file system at a selected time, said first snapshot including an indication of a set of storage blocks in said first consistent state;

forming a second snapshot of a second consistent state of said file system, said second snapshot including an indication of a set of storage blocks in said second consistent state; and

performing an operation on said first and second snapshots to form a storage image including an indication of at least some storage blocks in said file system.

29. A method as in claim 28, wherein said operation includes a logical sum or difference.

30. A method as in claim 28, wherein

said operation includes a logical sum or difference;

and

a purpose of said operation includes making a copy including or excluding a selected range of snapshots.

31. A method as in claim 28, wherein

said operation includes a logical sum or difference;

and

a purpose of said operation includes copying said storage image to a destination.

32. A method as in claim 30, wherein said destination includes a tape, a disk, a data structure in a second file system, a set of network messages, or a destination distributed over a plurality of file systems.

33. Apparatus including a file system that stores and retrieves information, said file system including:

mass storage having said plurality of storage blocks; and

a controller that controls storage and retrieval of information, said information including data and a plurality of snapshots of said file system, each representing an associated consistent state at an associated selected time, and each said snapshot including an indication of a set of storage blocks in said associated consistent state, said indication being recorded in at least one storage block in said associated consistent state, wherein said file system is responsive to at least one file system request with regard to said snapshots.

34. Apparatus as in claim 33, including a storage image defining at least some storage blocks in said file system, said storage image based on an operation on at least two of said snapshots.

35. In a file system having a plurality of storage blocks, a data structure including

a first snapshot of a set of member storage blocks selected from said plurality, said member storage blocks forming a consistent file system other than an active file system;

said first snapshot being represented as an object in said file system and having a set of storage blocks for recording said first snapshot;

whereby copying said member storage blocks in said first snapshot has the property of preserving at least one snapshot recorded in said file system at a time of said first snapshot.

36. A data structure as in claim 35, including

a second snapshot of a set of member storage blocks selected from said plurality, said member storage blocks forming a consistent file system other than an active file system;

said second snapshot being represented as an object in said file system and having a set of storage blocks for recording said second snapshot;

whereby copying said member storage blocks in said second snapshot has the property of preserving at least one snapshot recorded in said file system at a time of said second snapshot.

37. A data structure as in claim 35, including

an image stream including a set of storage blocks including both said first snapshot and said second snapshot;

whereby copying said member storage blocks in said image stream has the property of preserving both said first snapshot and said second snapshot.

38. In a file system having a plurality of storage blocks, a data structure including

a snapshot of a set of member storage blocks selected from said plurality, said member storage blocks forming a consistent file system other than an active file system;

said snapshot being represented as an object in said file system and having a set of storage blocks for recording said snapshot;

whereby a backup and restore operation on said file system has the property of preserving said snapshot within said file system.

39. A file system that stores and retrieves information in a plurality of storage blocks, said file system including:

mass storage having said plurality of storage blocks; and

a controller that controls storage and retrieval of information, said information including a data structure including a shadow snapshot of a set of member storage blocks selected from said plurality, said member storage blocks having formed a consistent file system other than an active file system, with a set of selected member storage blocks removed from said consistent file system;

wherein said shadow snapshot is disposed as a single object in said file system, whereby said controller can manipulate said shadow snapshot without having to traverse a hierarchy of file system objects associated with said shadow snapshot, and wherein said file system is responsive to at least one file system request with regard to said shadow snapshot.

40. A file system as in claim 39, wherein said shadow snapshot is disposed in a format allowing for a set management operation to be performed efficiently.

41. A file system as in claim 39, wherein said shadow snapshot uses, in addition to said member storage blocks, no more than 1/100th of an amount of storage required by said storage blocks in said plurality.

42. A file system as in claim 39, wherein said shadow snapshot uses, in addition to said member storage blocks, no more than one byte per storage block in said plurality.

43. A file system as in claim 39, wherein said removed member storage blocks are responsive to completion of a processing operation.

44. A file system as in claim 43, wherein said processing operation includes a file system operation.

45. A file system as in claim 43, wherein said processing operation includes reuse of said selected member storage blocks by said file system.

46. A file system as in claim 39, wherein said shadow snapshot is disposed in a format allowing for a set management operation to be performed in O(n) time or less, where n is a number of storage blocks in said plurality, without reading any contents of said storage blocks in said plurality.

47. A file system as in claim 46, wherein said set management operation is a logical sum or difference.

48. A file system that stores and retrieves information in a plurality of storage blocks, said file system including:

mass storage having said plurality of storage blocks; and

a controller that controls storage and retrieval of information, said information including a data structure that includes a mark on allocate image of a set of member storage blocks selected from said plurality, said member storage blocks having been added to a snapshot that originally formed a consistent file system;

wherein said mark on allocate storage image is disposed as a single object in said file system, whereby said controller can manipulate said storage image without having to traverse a hierarchy of file system objects associated with said storage image, and wherein said file system is responsive to at least one file system request with regard to said storage image.

49. A file system as in claim 48, wherein said mark on allocate image is disposed in a format allowing for a set management operation to be performed efficiently.

50. A file system as in claim 48, wherein said mark on allocate storage image uses no more than 1/100th of an amount of storage required by said storage blocks in said plurality.

51. A file system as in claim 48, wherein said mark on allocate image uses no more than four bytes per storage block in said plurality.

52. A file system as in claim 48, said member storage blocks having been selected responsive to completion of a processing operation.

53. A file system as in claim 52, wherein said processing operation includes a file system operation.

54. A file system as in claim 52, wherein said processing operation includes reuse of said selected member storage blocks by said file system.

55. A file system as in claim 48, wherein said mark on allocate image is disposed in a format allowing for a set management operation to be performed in O(n) time or less, where n is a number of storage blocks in said plurality, without reading any contents of said storage blocks in said plurality.

56. A file system as in claim 55, wherein said set management operation is a logical sum or difference.

57. A file system that stores and retrieves information in a plurality of storage blocks, said file system including:

mass storage having said plurality of storage blocks; and

a controller that controls storage and retrieval of information, said information including a data structure that includes a mark on deallocate image of a set of member storage blocks selected from said plurality, said member storage blocks having been removed from a snapshot that originally formed a consistent file system;

wherein said mark on deallocate storage image is disposed as a single object in said file system, whereby said controller can manipulate said storage image without having to traverse a hierarchy of file system objects associated with said storage image, and wherein said file system is responsive to at least one file system request with regard to said storage image.

58. A file system as in claim 57, wherein said mark on deallocate image uses no more than four bytes per storage block in said plurality.

59. A file system as in claim 57, wherein said mark on deallocate image is disposed in a format allowing for a set management operation to be performed efficiently.

60. A file system as in claim 57, wherein said mark on deallocate storage image uses no more than 1/100th of an amount of storage required by said storage blocks in said plurality.

61. A file system as in claim 57, wherein said mark on deallocate image is disposed in a format allowing for a set management operation to be performed in O(n) time or less, where n is a number of storage blocks in said plurality, without reading any contents of said storage blocks in said plurality.

62. A file system as in claim 61, wherein said set management operation is a logical sum or difference.