Low overhead methods and apparatus for shared access storage devices

• US 7,010,532 B1
• Filed: 10/13/2000
• Issued: 03/07/2006
• Est. Priority Date: 12/31/1997
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Stakutis, Christopher J., Stearns, Kevin M.
• Primary Examiner(s): Pardo, Thuy N.
• Application Number:
  US09/687,668
• Time in Patent Office: 1,971 Days
• Field of Search: 707/3, 707/8, 707/10, 707/2, 707/201, 707/203, 707/104, 707/100, 707/205, 709/213, 709/245, 709/219, 709/229, 709/203, 712/28
• US Class Current: 707/625
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 2003/0697 device management, e.g. handlers, drivers, I/O schedulers
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/99938 Concurrency, e.g. lock management in shared database
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Provided are implementations for improved access to information stored on a storage device. A plurality of first nodes and a second node are coupled to one another over a communications pathway, the second node being coupled to the storage device for determining meta data including block address maps to file data in the storage device, and the first nodes being configured for accessing file data from the storage device using said meta data, wherein at least one first node caches data including meta data for a file accessed by said first node. A file application on said first node is configured to get requested file data by accessing said cached data for the file. A file notification system sends a file change notification to said first node indicating changes affecting the cached data, wherein the file application on the first node inspects the change notification to determine whether to get the requested file data directly using said cached data, whereby file accesses may be effected for an extended time with data locally cached at first nodes of the system.

Claims

1. A digital data processing system with access to information stored on a storage device, the system comprising a plurality of first nodes and a second node coupled to one another over a communications pathway, the second node being coupled to the storage device for determining meta data including block address maps to file data in the storage device, and the first nodes being configured for accessing file data from the storage device using the meta data, wherein the system comprises:

at least one first node that caches data including meta data for a file accessed by the first node;

a file application on the first node configured to get requested file data by accessing the cached data for the file; and

a file notification system that sends a file charge notification to the first node indicating changes affecting the cached data wherein the file change notification identifies changes to data associated with the second node, and wherein the file change notification is communicated to the first node by publishing the file change notification via the second node, wherein the file application on the first node inspects the file change notification and based on the file change notification determines, whether to get the requested file data directly using the cached data or whether to get the requested file data from the storage device, wherein the storage device is a shared storage device between the first node and the second node, wherein file accesses may be effected for an extended time with data locally cached at the first nodes of the digital data processing system.

2. The digital data processing system of claim 1, wherein the file application on the first node determines whether requested file data is subject to the file change notification, and if so makes a further determination whether the cached data at the first node remains valid for the requested file data.

3. The digital data processing system of claim 1, wherein the file application on the first node:

i) determines whether requested file data is subject to the file change notification; and

ii) applies the cached meta data to directly mount the storage device to access the requested file when the cached data is not subject to the file change notification.

4. The digital data processing system of claim 2, wherein the file application on the first node further determines whether:

i) to directly access the file data by applying cached meta data associated with the file to directly mount the storage device, or

ii) to issue a file request to the second node for valid file access meta data or data.

5. The digital data processing system of claim 1, wherein the file notification system issues client-specific notifications limited to directories or portions of the file system that are to be accessed by each client.

6. The digital data processing system of claim 1, wherein the file notification system includes an interface layer with a storage system meta data controller for maintaining or acquiring administrative information pertaining to file size and storage locations.

7. The digital data processing system of claim 1, wherein the file notification system runs on the second node and interfaces with a file system meta data controller to detect changes in file system storage data, issuing a file change notice in response thereto.

8. The digital data processing system of claim 1, wherein the file notification system limits number of file change notifications for a given file to first n changes that occur, where n is a positive integer.

9. The digital data processing system of claim 1, wherein the file application on the first node implements a decision algorithm to determine whether to apply cached data for a requested file when the requested file is subject to the file change notification.

10. The digital data processing system of claim 1, wherein the file application on the first node intercepts reads and writes, and issues those directly to the storage device while exchanging messages over the communications pathway to permit coordinate file system management tasks performed by the second node.

11. The digital data processing system of claim 10, wherein the file system management tasks performed by the second node include publication of change data.

12. The digital data processing system of claim 1, wherein the file shared access coordination system runs on the second node and interfaces with or includes a file system meta data controller interceding in response to at least a first selected file access request applied thereto by a file application on a first node, and transferring data designated by that request between the first node and the peripheral device in accord with current meta data maintained by the file system pertaining to storage of that data on the storage device such that files may be directly transferred while maintaining file coherence and security.

13. The digital data processing system of claim 1, wherein if the file change notification indicates that changes that affect the cached data have been made, then the requested file data is obtained from the storage device coupled to the second node, wherein the storage device coupled to the second node is also coupled to the first node.

14. A digital data processing system, comprising:

a first node and a second node coupled for communication;

a storage device coupled for communication with at least the first node; and

a cache memory coupled to and associated with the first node, wherein the cache memory stores administrative data pertaining to files on the storage device, wherein the second node notifies the first node of changes to the administrative data pertaining to files for which the cache memory stores the administrative data, wherein a change notification identifies changes to data associated with the second node, and wherein the change notification is communicated to the first node by publishing the change notification via the second node.

15. A digital data processing system according to claim 14, wherein the storage device is any of a disk drive, a “

jukebox,”

other mass storage device or other mapped device.

16. A digital data processing system according to claim 14, wherein the administrative data stored by the cache includes any of a physical storage map and at least a portion of a directory pertaining to files on the storage device.

17. A digital data processing system according to claim 14, wherein the digital data processing system comprises a network having a file management system, and a file application on the first node applies administrative data in the cache memory pertaining to a file to directly mount the storage device.

18. A digital data processing system according to claim 14, wherein the digital data processing system comprises a network having a file management system, and a file application applies a notification of a change of administrative data pertaining to a given file by passing a request for that file by the first node to the file management system.

19. The digital data processing system of claim 14, wherein in response to receiving, at the first node, a notification of the changes to the administrative data, the first node obtains a requested file data from the storage device, wherein the storage device is coupled to the first node and the second node.

20. A method of sharing storage access in a digital data processing system having a first node and a second node coupled for communication and a storage device coupled for communication with at least the first node, the method comprising:

caching in a cache memory coupled to and associated with the first node, administrative data pertaining to files on the storage device;

communicating, to the first node, changes to the administrative data pertaining to files for which the cache memory stores the administrative data, wherein a change notification identifies changes to data associated with the second node, and wherein the change notification is communicated to the first node by publishing the change notification via the second node; and

determining, in the first node, whether to apply the cached data to reduce network communications for accessing a file.

21. The method of claim 20, wherein in response to receiving a communication of the changes to the administrative data, the first node accesses the file from the storage device if the first node determines not to apply the cached data for accessing the file, wherein the storage device is coupled to the first node and the second node.

22. A digital data processing method for access to information in a system that includes a storage device, a plurality of first nodes and a second node communicating over a communications pathway, the second node being coupled to the storage device for determining meta data for accessing file data in the storage device and the first nodes being configured for accessing file data from the storage device using the meta data, wherein the method comprises:

caching meta data for a file accessed by the first node in a cache memory of the first node;

providing a file application on the first node configured to get requested file data utilizing the cached data;

storing file change notifications at the first node indicating changes that may affect the cached data, wherein the file change notifications identify changes to data associated with the second node, and wherein the file change notifications are communicated to the first node by publishing the file change notifications via the second node; and

determining, based on the file change notifications, whether the file application may utilize the cached data for a requested file or whether the file application accesses the requested file from the storage device, wherein the storage device is a shared storage device between the first node and the second node.

23. The method of claim 22, wherein the storing of file change notifications is effected by receiving a change list published by the second node and storing the change list.

24. The method of claim 22, wherein the file application

i) determines whether requested file data is subject to a file change notification, and

ii) applies the cached meta data to directly mount the storage device to access requested file when the cached data is not subject to the file change notification.

25. The method of claim 22, wherein the file application on the first node operates

i) to directly access the file data by applying cached meta data associated with the file to directly mount the storage device when the cached data is not subject to a file change notification, or

ii) to issue a file request to the second node when the cached data is subject to the file change notification.

26. The method of claim 22, wherein the file change notifications are client-node-specific notifications limited to directories or portions of the file system that are to be accessed by each client.

27. The method of claim 22, wherein a file notification system runs on the second node and operates with a file system meta data controller to detect changes in file system storage data and issue the file change notifications in response thereto.

28. The method of claim 27, wherein the second node limits number of file change notifications for a given file to first n changes that occur, where n is a positive integer.

29. The method of claim 22, wherein the file application on the first node implements a decision algorithm to determine whether to apply cached data for a requested file.

30. The method of claim 22, wherein if the file change notifications indicate that changes that affect the cached data have been made, then the requested file is obtained from the storage device coupled to the second node, wherein the storage device coupled to the second node is also coupled to the first node.

31. A digital data processing system with access to information stored on a storage device, the system comprising a plurality of first nodes and a second node coupled to one another over a communications pathway, the second node being coupled to the storage device for determining meta data including block address maps to file data in the storage device, wherein the storage device is a shared storage device between the first node and the second node, and the first nodes being configured for accessing file data from the storage device using the meta data, wherein the system comprises:

at least one first node that caches data including meta data for a file accessed by the first node;

a file application on the first node configured to get requested file data by accessing the cached data for the file; and

a file notification system that communicates a file change notification to the first node indicating changes affecting the cached data, wherein the file change notification is communicated to the first node by publishing the file change notification via the second node, wherein the file application on the first node inspects the file change notification and if the file change notification indicates that changes that affect the cached data have been made, then the requested file data is obtained from the storage device, otherwise the requested file data is directly obtained using the cached data, and wherein file accesses may be effected for an extended time with data cached at the first nodes of the digital data processing system.

32. The digital data processing system of claim 31, wherein the file notification system runs on the second node and sends the file change notification to the first node, wherein the first node is registered with the second node for receiving the file change notification, wherein the file change notification includes changes to the data and the meta data.

33. A digital data processing system, comprising:

a first node and a second node coupled for communication;

a storage device coupled for communication with at least the first node; and

a cache memory coupled to and associated with the first node, the cache memory caching administrative data pertaining to files on the storage device, wherein the second node notifies the first node of changes to the cached administrative data by publishing a change notification that indicates the changes affecting the cached administrative data, and wherein if the change notification indicates that changes that affect the cached administrative data have been made, then a file data requested by the first node is obtained from the storage device.

34. The digital data processing system of claim 33, wherein the second node notifies the first node via a file notification system that runs on the second node and sends the change notification to the first node, wherein the first node is registered with the second node for receiving the change notification, wherein the change notification includes changes to data and the administrative data stored in the cache memory.

35. A method of sharing storage access in a digital data processing system having a first node and a second node coupled for communication and a storage device coupled for communication with at least the first node, the method comprising:

caching in a cache memory coupled to and associated with the first node, administrative data pertaining to files on the storage device;

communicating, from the second node, a change notification to the first node, wherein the change notification indicates changes affecting the cached administrative data, and wherein the change notification is published by the second node to communicate the change notification to the first node;

receiving, by the first node, the communicated change notification;

determining, by the first node, whether to apply the cached administrative data for accessing a file;

accessing, by the first node, the file from the storage device if the first node determines not to apply the cached administrative data for accessing the file.

36. The method of claim 35, wherein a file notification system runs on the second node and sends the change notification to the first node, wherein the first node is registered with the second node for receiving the change notification, wherein the change notification includes changes to file data and the administrative data stored in the cache memory.

37. A method for accessing information stored on a storage device, wherein the storage device is shared by a plurality of first nodes and a second node, wherein the plurality of first nodes and the second node communicate over a communications pathway, wherein the second node is coupled to the storage device for determining meta data for accessing file data in the storage device, and wherein the first nodes are configured for accessing file data from the storage device using the meta data, the method comprising:

caching meta data for a file accessed by a first node in a cache memory of the first node;

providing a file application on the first node configured to get requested file data utilizing the cached meta data;

storing file change notifications at the first node indicating changes that may affect the cached data, and wherein the file change notifications are communicated to the first node by publishing the file change notifications via the second node; and

determining, based on the file change notifications, whether the file application may utilize the cached meta data for a requested file or whether the file application accesses the requested file from the storage device, wherein if the file change notifications indicate that changes that affect the cached meta data have been made, then the requested file is obtained from the storage device coupled to the second node.

38. The method of claim 37, wherein a file notification system runs on the second node and sends a file change notification to one first node, wherein the one first node is registered with the second node for receiving the file change notification, wherein the file change notification includes changes to file data and the meta data.