Method and apparatus for implementing a file system

• US 20050289152A1
• Filed: 06/10/2004
• Published: 12/29/2005
• Est. Priority Date: 06/10/2004
• Status: Abandoned Application

• Current Assignee: Agami Systems, Inc.
• Original Assignee: Agami Systems, Inc.
• Inventors: Sheehan, Kevin, Byrnes, Brian, Barszczak, Tomasz, Stirling, Patrick M., Earl, William J., Rai, Chetan
• Application Number:
  US10/866,229
• Publication Number:
  US 20050289152A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/184 implemented as replicated file system

Abstract

A system and method for efficiently implementing a local or distributed file system is disclosed. The system may include a distributed virtual file system (“dVFS”) that utilizes a persistent intent log (“PIL”) to record transactions to be applied to the file system. The PIL is preferably implemented in stable storage, so that a logical operation may be considered complete as soon as the log record has been made stable. This allows the dVFS to continue immediately, without waiting for the operation to be applied to a local or real file system. The dVFS may further incorporate replication to one or more remote file systems as an integral facility.

Claims

1. A file system comprising:

one or more front-end elements that provide access to the file system;

one or more back-end elements that communicate with the one or more front-end elements and provide persistent storage of data; and

a persistent log that stores file system operations communicated from the one or more front-end elements to the one or more back-end elements;

wherein the file system treats the file system operations as complete when the operations are stored in the log, thereby allowing the file system to continue operating without waiting for the operations to be applied to the one or more local file systems.

2. The file system of claim 1 wherein the file system is distributed over a plurality of computer systems.

3. The file system of claim 2 wherein the persistent log is stored in part on each of the plurality of computer systems.

4. The file system of claim 1 wherein the persistent log is implemented using stable storage.

5. The file system of claim 4 wherein the stable storage comprises battery-backed memory, flash memory, or a low-latency storage device.

6. The file system of claim 1 wherein the one or more front-end elements comprise a second log that buffers updates to be sent to the persistent log.

7. The file system of claim 6 wherein the one or more front-end elements confirm that there are sufficient resources for an operation to be performed on corresponding back-end elements before placing the operation in the second log.

8. The file system of claim 1 wherein the file system can apply operations from the persistent log to the back-end elements out of order.

9. The file system of claim 1 further comprising a lock manager that maintains data consistency in the file system.

10. The file system of claim 9 wherein the lock manager provides an exclusive lock that allows only a certain element to update files at a given time.

11. The file system of claim 9 wherein the lock manager provides a shared write lock function.

12. The file system of claim 1 wherein the persistent log maintains an index of pending operations, and wherein the one or more front-end elements check the index when receiving a request for data to ensure data coherency.

13. The file system of claim 12 wherein the one or more front-end elements satisfy a request for data using the persistent log whenever possible.

14. The file system of claim 1 wherein objects can be migrated from one back-end element to another.

15. The file system of claim 1 wherein the file system is adapted to provide for replication by transmitting entries from the persistent log to a second file system.

16. The file system of claim 15 wherein the replication is synchronous.

17. The file system of claim 15 wherein the replication is asynchronous.

18. An apparatus for implementing a file system including a plurality of front-end elements that provide access to the file system and one or more back-end elements that communicate with the front-end elements and provide persistent storage of data, the apparatus comprising:

a persistent log that stores file system operations communicated from the one or more front-end elements to the one or more back-end elements; and

a process that allows the file system to continue operating once the operations are stored in the log without waiting for the operations to be applied to the one or more back-end elements.

19. The apparatus of claim 18 wherein the persistent log is stored in part on each of a plurality of computer systems.

20. The apparatus of claim 18 wherein the persistent log is implemented using stable storage.

21. The apparatus of claim 20 wherein the stable storage comprises battery-backed memory, flash memory, or a low-latency storage device.

22. The apparatus of claim 18 further comprising:

a second log that buffers updates to be sent to the persistent log.

23. The apparatus of claim 22 wherein the second log is located in the one or more front-end elements.

24. The apparatus of claim 23 further comprising:

a second process that confirms that there are sufficient resources for an operation to be performed on corresponding back-end elements before placing the operation in the second log.

25. The apparatus of claim 18 wherein the process selectively applies operations from the persistent log to the back-end elements out of order.

26. The apparatus of claim 18 further comprising:

a lock manager that maintains data consistency in the file system.

27. The apparatus of claim 26 wherein the lock manager provides an exclusive lock that allows only a certain element to update files at a given time.

28. The apparatus of claim 26 wherein the lock manager provides a shared write lock function.

29. The apparatus of claim 18 further comprising:

a replication process for replicating the file system by transmitting entries from the persistent log to a second file system.

30. A method for implementing a file system having one or more front-end elements that provide access to the file system, and one or more back-end elements that communicate with the one or more front-end elements and provide persistent storage of data, the method comprising:

storing operations in a persistent log, wherein the operations comprise file system operations communicated from the one or more front-end elements to the one or more back-end elements; and

allowing the file system to continue operating once the operations are stored in the log without waiting for the operations to be applied to the one or more back-end elements.

31. The method of claim 30 wherein the file system is distributed over a plurality of computer systems.

32. The method of claim 31 wherein the log is stored in part on each of the plurality of computer systems.

33. The method of claim 30 wherein the persistent log is implemented using stable storage.

34. The method of claim 33 wherein the stable storage comprises battery-backed memory, flash memory, or a low-latency storage device.

35. The method of claim 30 further comprising:

buffering updates to be sent to the persistent log in a second log contained in the one or more front-end elements.

36. The method of claim 30 further comprising:

applying operations from the persistent log to the back-end elements out of order.

37. The method of claim 30 further comprising:

maintaining data consistency in the file system using a lock manager.

38. The method of claim 37 wherein the lock manager provides an exclusive lock that allows only a certain element to update files at a given time.

39. The method of claim 37 wherein the lock manager provides a shared write lock function.

40. The method of claim 30 further comprising:

maintaining an index of pending operations in the persistent log; and

checking the index before requesting data from the one or more back-end elements to ensure data coherency.

41. The method of claim 40 further comprising:

satisfying a request for data using just the persistent log.

42. The method of claim 40 further comprising:

migrating objects from one back-end element to another.

43. The method of claim 35 further comprising:

confirming that there are sufficient resources for an operation to be performed on corresponding back-end elements before placing the operation in the second log.

44. The method of claim 30 further comprising:

replicating the first file system by transmitting entries from the persistent log to a second file system.

45. The method of claim 44 wherein the replicating is synchronous.

46. The method of claim 44 wherein the replicating is asynchronous.

47. The method of claim 44 further comprising:

reviewing entries from the persistent log for compensating operations prior to transmitting the entries to the second file system; and

eliding compensating operations.