Unordered idempotent logical replication operations

• US 8,655,848 B1
• Filed: 02/26/2010
• Issued: 02/18/2014
• Est. Priority Date: 04/30/2009
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Leverett, Bruce, Kazar, Michael, Simharajan, Srishylam, Corbett, Peter Frank, Smith, Toby, Driscoll, Alan Stuart, Shah, Devang K.
• Primary Examiner(s): Pannala, Sathyanarayan
• Application Number:
  US12/713,744
• Time in Patent Office: 1,453 Days
• Field of Search: 711/173, 709/201, 709/203, 709/223, 707/660
• US Class Current: 707/660
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 3/065 Replication mechanisms

Abstract

A method and apparatus for replicating a data container of a source storage server at the logical level in an unordered stream of individual data units are described. In certain embodiments, the replication operations can be performed without regard to the differences in geometry of physical persistent storage of the source and destination storage servers and without regard to the differences in data container format of the source and destination storage servers. An initial replication operation of the source data container is performed by transferring the data objects within the source data container to a replica data container in the destination storage server as an unordered stream of individual data units while preserving data object identifiers of the source data objects. Afterwards, incremental replication operations can be performed to capture modifications to the source data container over time after the initial replication operation.

Claims

1. A method of replicating data in a file system, the method comprising:

creating a current snapshot of a file system stored within a source storage server at a point in time, the source file system configured to store a plurality of data objects, each of which is assigned a corresponding data object identifier;

scanning each data object within the source file system to identify changes in the source file system that have occurred since an earlier replication operation was performed by comparing the current snapshot with a previous snapshot corresponding to the earlier replication operation; and

transferring the changes in the source file system as a plurality of individual data units in a stream to the destination storage server while preserving the corresponding file system object identifiers to replicate the source file system at the destination storage server, wherein an ordering of the individual data units within the stream is independent of a hierarchical structure of the source file system;

wherein the changes in the source file system include updates to a data block of a file containing both modified and unmodified data blocks, wherein for the file, transferring only the modified data blocks to replicate the source file system at a destination storage server, wherein the changes in the source file system include updates to a directory entry of a directory containing both modified and unmodified directory entries, and wherein for the directory, transferring only the modified directory entries to replicate the source file system at the destination storage server;

wherein if the changes are for a directory, the stream of individual data units includes;

an inode number of a directory corresponding to a directory entry being replicated,

a generation number of the directory,

a name of the file system object referenced by the directory entry,

an inode number of the file system object referenced by the directory entry, and

a generation number of the file system object referenced by the directory entry.

2. The method of claim 1, wherein the replicating is performed without regard to differences in geometry of physical persistent storage of the source and destination storage servers and without regard to differences in file system format of the source and destination storage servers.

3. The method of claim 1, further comprising:

scanning each file system object within the source file system by examining each inode entry in an inode file of the source file system from beginning to end in a single pass through the inode file of the source file system, wherein when the inode is a directory, scanning each individual directory entry; and

transferring each file system object to the destination storage server without first determining where the file system object is located in a directory tree of the source file system.

4. The method of claim 3, wherein each of the file system object identifiers is an inode number of an inode entry in an inode file of the source storage server.

5. The method of claim 4, wherein the source inode file is a data structure representing a master list of source file system objects and each inode entry identifies a particular file system object within the source file system.

6. The method of claim 5, further comprising:

initiating a data operation when the file system object to be replicated is a file; and

initiating a directory operation when the file system object to be replicated is a directory or subdirectory.

7. The method of claim 6, wherein for a data operation the stream of individual data units includes:

a file block number of the data block being replicated;

a corresponding inode number for the data block corresponding to the file block number;

a block of data corresponding to the file block number; and

a generation number of the file.

8. The method of claim 6, wherein for a directory operation the stream of individual data units includes:

an inode number of a directory corresponding to a directory entry being replicated;

a generation number of the directory;

a name of the file system object referenced by the directory entry;

an inode number of the file system object referenced by the directory entry; and

a generation number of the file system object referenced by the directory entry.

9. The method of claim 6, further comprising one or more from the list consisting of:

transferring a file of the source file system to the destination storage server before transferring a directory in which the file is located;

transferring data blocks of a file of the source file system to the destination storage server before transferring metadata describing the file;

transferring individual directory entries out of order; and

transferring individual data blocks of a file out of order.

10. A method of replicating data in a file system, the method comprising:

receiving, at a destination storage server, a plurality of file system objects in a stream of individual data units from a source storage server, wherein an ordering of the individual data units within the stream is independent of a hierarchical structure of a source file system of the source storage server, the stream containing only the changes that have occurred in the source file system of the source storage server since a previous replication operation was performed, wherein each of the file system objects includes a corresponding file system object identifier, and wherein the changes in the source file system include updates to a data block of a file containing both modified and unmodified data blocks, wherein for the file, receiving only the modified data blocks to replicate the source file system at the destination storage server, wherein the changes in the source file system include updates to a directory entry of a directory containing both modified and unmodified directory entries, and wherein for the directory, receiving only the modified directory entries to replicate the source file system at the destination storage server;

replicating the plurality of file system objects at the destination storage server using the file system object identifiers; and

creating a current snapshot of the replicated file system of the destination storage server once the plurality of file system objects are replicated at the destination so that corresponding snapshots of the source and destination storage servers are identical;

wherein if the changes are for a file, the stream of individual data units includes;

a file block number of the data block being replicated,

a corresponding inode number for the data block corresponding to the file block number,

a block of data corresponding to the file block number, and

a generation number of the file.

11. The method of claim 10, wherein the replicating is performed without regard to differences in geometry of physical persistent storage of the source and destination storage servers and without regard to differences in file system format of the source and destination storage servers.

12. The method of claim 10, wherein each of the file system object identifiers is an inode number of an inode entry in an inode file of the source storage server.

13. The method of claim 12, wherein the source inode file is a data structure representing a master list of source file system objects and each inode entry identifies a particular file system object within the source file system.

14. The method of claim 13, wherein when the file system object is a file, the stream of individual data units includes:

a file block number of the data block being replicated;

a corresponding inode number for the data block corresponding to the file block number;

a block of data corresponding to the file block number; and

a generation number of the file.

15. The method of claim 13, wherein, when the file system object is a directory, the stream of individual data units includes:

an inode number of a directory corresponding to a directory entry being replicated;

a generation number of the directory;

a name of the file system object referenced by the directory entry;

an inode number of the file system object referenced by the directory entry; and

a generation number of the file system object referenced by the directory entry.

16. The method of claim 13, further comprising:

matching the inode number of the source file system object in the source inode file with a corresponding inode number in a replicated inode file at the destination storage server; and

adding the file system object to an inode entry in the replicated inode file at the destination storage server so that the inode number of the source inode file is preserved in the replicated inode file of the destination storage server.

17. The method of claim 16, further comprising:

storing data associated with each file system object to an unused block of data in physical persistent storage of the destination storage server; and

updating a buffer tree representing a logical block structure of the destination file system to reference the data of each file system object, wherein the buffer tree includes a hierarchical metadata structure containing pointers to logical blocks of data in the destination file system.

18. The method of claim 17, wherein updating the buffer tree comprises:

updating an indirect block in the buffer tree to include a pointer to the data of each file system object;

storing each indirect block to a different unused block of data in physical persistent storage of the destination storage server; and

repeating the updating and storing until all pointers to each data of the file system object in the destination buffer tree have been updated.

19. The method of claim 18, further comprising assigning the file block number to a highest-level indirect block in the destination buffer tree referencing the data of the file system object.

20. A source storage server apparatus comprising:

a processor;

a network interface coupled with the processor, the network interface through which the source storage server can communicate with a destination storage server;

a snapshot differential module configured to create a current snapshot of a file system stored within the source storage server at a point in time, the source file system configured to store a plurality of data objects, each of which is assigned a corresponding data object identifier; and

a replication engine coupled with the snapshot differential module, the replication engine configured to;

scan each data object within the source file system to identify changes in the source file system that have occurred since an earlier replication operation was performed by comparing the current snapshot with a previous snapshot corresponding to the earlier replication operation; and

transfer the changes in the source file system as a plurality of individual data units in a stream to the destination storage server while preserving the corresponding file system object identifiers to replicate the source file system at the destination storage server without first determining a relationship between the data units and a hierarchical structure of the source file system, wherein an ordering of the individual data units within the stream is independent of the hierarchical structure of the source file system;

wherein the changes in the source file system include updates to a data block of a file containing both modified and unmodified data blocks, wherein for the file, transmitting only the modified data blocks to replicate the source file system at a destination storage server, wherein the changes in the source file system include updates to a directory entry of a directory containing both modified and unmodified directory entries, and wherein for the directory, transmitting only the modified directory entries to replicate the source file system at the destination storage server;

wherein if the changes are for a file, the individual data units include;

a file block number of the data block being replicated,

a corresponding inode number for the data block corresponding to the file block number,

a block of data corresponding to the file block number, and

a generation number of the file.

21. The source storage server apparatus of claim 20, wherein replication is performed without regard to differences in geometry of physical persistent storage of the source and destination storage servers and differences in file system format of the source and without regard to destination storage servers.

22. The source storage server of claim 21, wherein the replication engine is further configured to:

scan each file system object within the source file system by examining each inode entry in an inode file of the source file system from beginning to end in a single pass through the inode file of the source file system, wherein when the inode is a directory, scanning each individual directory entry; and

transmit each file system object to the destination storage server without first determining where the file system object is located in a directory tree of the source file system.

23. The source storage server apparatus of claim 22, wherein each of the file system object identifiers is an inode number of an inode entry in an inode file of the source storage server, and wherein the source inode file includes a data structure representing a master list of source file system objects and each inode entry identifies a particular file system object within the source file system.

24. The source storage server apparatus of claim 23, wherein the replication engine is further configured to transmit one or more from the list consisting of:

a file of the source file system to the destination storage server before transmitting a directory in which the file is located;

data blocks of a file of the source file system to the destination storage server before transmitting metadata describing the file;

individual directory entries out of order; and

individual data blocks of a file out of order.

25. A destination storage server apparatus comprising:

a processor;

a network interface coupled with the processor, the network interface through which the destination storage server can communicate with a source storage server; and

a replication engine configured to;

receive, at a destination storage server, a plurality of file system objects in a stream of individual data units from a source storage server, wherein an ordering of the individual data units within the stream is independent of a hierarchical structure of a source file system of the source storage server, the stream containing only the changes that have occurred in the source file system of the source storage server since a previous replication operation was performed, wherein each of the file system objects includes a corresponding file system object identifier, wherein the changes in the source file system include updates to a data block of a file containing both modified and unmodified data blocks, wherein for the file, receiving only the modified data blocks to replicate the source file system at the destination storage server, wherein the changes in the source file system include updates to a directory entry of a directory containing both modified and unmodified directory entries, and wherein for the directory, receiving only the modified directory entries to replicate the source file system at the destination storage server; and

replicate the plurality of file system objects at the destination storage server using the file system object identifiers; and

a snapshot differential module configured to create a current snapshot of the replicated file system of the destination storage server once the plurality of file system objects are replicated at the destination so that corresponding snapshots of the source and destination storage servers are identical;

wherein if the changes are for a file, the stream of individual data units includes;

a file block number of the data block being replicated,

a corresponding inode number for the data block corresponding to the file block number,

a block of data corresponding to the file block number, and

a generation number of the file;

wherein if the changes are for a directory, the stream of individual data units includes;

an inode number of a directory corresponding to a directory entry being replicated,

a generation number of the directory,

a name of the file system object referenced by the directory entry,

an inode number of the file system object referenced by the directory entry, and

a generation number of the file system object referenced by the directory entry.

26. The destination storage server apparatus of claim 25, wherein the replicating is performed without regard to differences in geometry of physical persistent storage of the source and destination storage servers and without regard to differences in file system format of the source and destination storage servers.

27. The destination storage server apparatus of claim 26, wherein the stream of individual data units includes:

data associated with a file system object in the source file system; and

an inode number of an inode file of the source storage server assigned to the file system object.

28. The destination storage server apparatus of claim 27, wherein the source inode file includes a data structure representing a master list of file system objects of the source file system and each inode entry identifies a particular file system object within the source file system.

29. The destination storage server of apparatus claim 28, wherein the replication engine is further configured to:

match the inode number of the file system object in the source inode file with a corresponding inode number in a replicated inode file at the destination storage server; and

add the file system object to an inode entry in the replicated inode file at the destination storage server so that the inode number of the source inode file is preserved.

30. The destination storage server apparatus of claim 29, wherein the replication engine is further configured to:

store the data associated with the file system object to an unused block of data in physical persistent storage of the destination storage server; and

update a buffer tree representing a logical block structure of the destination file system to reference the data of the file system object, wherein the buffer tree includes a hierarchical metadata structure containing pointers to logical blocks of data in the file system.

31. The destination storage server apparatus of claim 30, wherein the destination replication engine is further configured to assign the file block number to a highest-level indirect block in the buffer tree pointing to the data of the file system object.