Data storage management for network interconnected processors

• US 5,537,585 A
• Filed: 02/25/1994
• Issued: 07/16/1996
• Est. Priority Date: 02/25/1994
• Status: Expired due to Term

• Current Assignee: Clouding Corporation (Marathon Patent Group, Inc.)
• Original Assignee: Avail Systems Corporation
• Inventors: Kirchner, Anton H., Winter, Richard A., Blickenstaff, Ronald L., Trede, Brian E., Dodd, Paul D., Montez, Jennifer K., Brant, Catherine I.
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): Lintz, Paul R.
• Application Number:
  US08/201,658
• Time in Patent Office: 872 Days
• Field of Search: 395/600, 395/800, 395/440, 395/444
• US Class Current: 1/1
• CPC Class Codes: G06F 12/08 in hierarchically structured memory systems, e.g. virtual memory systems
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup
• CPC Class Codes: Y10S 707/99956 File allocation

Abstract

The data storage system is connected to a local area network and includes a storage server that on a demand basis and/or on a periodically scheduled basis audits the activity on each volume of each data storage device that is connected to the network. Low priority data files are migrated via the network and the storage server to backend data storage media, and the directory resident in the data storage device is updated with a placeholder entry to indicate that this data file has been migrated to backend storage. When the processor requests this data file, the placeholder entry enables the storage server to recall the requested data file to the data storage device from which it originated.

Claims

1. A data storage management system for a data network which functions to interconnect a plurality of file servers, each of which stores data files, comprising:

secondary storage means for storing data files migrated from said file servers;

storage server means connected to said network for automatically managing transfer of data files, independent of said file servers, between said plurality of file servers and said secondary storage means;

means for collecting a plurality of data files, that are transmitted to said secondary storage means, into a transfer unit.

2. The system of claim 1 wherein said secondary storage means comprises:

a multi-layer hierarchical memory, wherein said layers in said hierarchical memory comprise media of differing characteristics.

3. The system of claim 2 wherein said hierarchical memory comprises a plurality of layers, which include memory devices comprised of media of at least one of the class of:

magnetic media, optical media, and solid-state media.

4. The system of claim 2 wherein each layer of said hierarchical memory is connected to said storage server means.

5. The system of claim 2 wherein at least one of said layers comprises:

a plurality of data storage elements for storing data files migrated from said file servers;

at least one data storage element drive means for reading/writing data on a data storage element mounted in said data storage element drive means; and

automated data storage element management means for robotically mounting a selected one of said plurality of data storage elements in said data storage element drive means.

6. The system of claim 1 wherein said storage server means comprises:

directory means for storing data indicative of a physical data storage location that identifies a locus in said secondary storage means of each of said data files migrated to said secondary storage means.

7. The system of claim 1 wherein said secondary storage means comprises a multi-layer hierarchical memory, wherein said layers in said hierarchical memory comprise media of differing characteristics, said storage server means comprises:

means for storing said transfer unit on a first layer of said hierarchy;

means for automatically relocating data files that are stored on a layer of said hierarchy to an other layer of said hierarchy as a function of predetermined criteria; and

wherein said automatically relocating means transmits said transfer unit to said other layer in said hierarchy.

8. The system of claim 6 wherein said directory means comprises:

transfer unit directory means for storing data indicative of a correspondence between a data file and a transfer unit in which said data file is located; and

media object directory means for storing data indicative of a correspondence between a transfer unit and a media on which said transfer unit is located.

9. The system of claim 8 wherein said storage server updates said media object directory means when a transfer unit is relocated from said media on which said transfer unit is located to another media in said hierarchy.

10. The system of claim 7 further comprising:

backup means for writing each said transfer unit written on said first layer of said hierarchy on to a mountable data storage element.

11. The system of claim 10 wherein said backup means writes a partially filled transfer unit on to said mountable data storage element upon expiration of a predetermined time interval absent sufficient data files to fill a transfer unit.

12. The system of claim 11 wherein said backup means overwrites a partially filled transfer unit on said mountable data storage element with a filled transfer unit that includes all data files written in said partially filled transfer unit.

13. The system of claim 10 wherein said storage server means is responsive to loss of a requested data file in an assigned transfer unit for retrieving said requested data file from said backup means by reading at least a portion of said assigned transfer unit from said mountable data storage element to recover said lost data files.

14. The system of claim 10 wherein said storage server means further comprises:

means, responsive to lack of further available data storage space on a lowest layer of said hierarchy, for deleting at least one selected transfer unit from said lowest layer.

15. The system of claim 14 wherein said storage server further comprises:

means, responsive to addition of a layer of media to said hierarchy lower than said lowest layer, for transferring said selected deleted transfer unit from said backup means to said added layer of media by reading said selected deleted transfer unit from said mountable data storage element.

16. The system of claim 14 wherein said storage server further comprises:

means, responsive to addition of data storage capacity to said lowest layer, for transferring said selected deleted transfer unit from said backup means to said added data storage capacity by reading said selected deleted transfer unit from said mountable data storage element.

17. The system of claim 10 wherein said storage server means further comprises:

means, responsive to removal of available data storage space on a selected layer of said hierarchy, for deleting all transfer units stored on said removed available data storage space from said selected layer.

18. The system of claim 17 wherein said storage server further comprises:

means, responsive to available data storage capacity in a layer lower in said hierarchy than said selected layer, for transferring said deleted transfer units from said backup means to said available data storage capacity by reading said deleted transfer units from said mountable data storage element.

19. The system of claim 10 wherein at least one of said layers comprises:

shelf layer comprising at least one mountable data storage element that contains data files migrated from said file servers and whose locations on said data storage element are maintained in said directory means.

20. The system of claim 19 wherein said shelf layer mountable media comprises said backup mountable media.

21. The system of claim 19 wherein one of said layers comprises:

retirement layer comprising at least one mountable data storage element that contains data files migrated from said file servers and whose directory entries are purged from said transfer unit directory means.

22. The system of claim 19 wherein said storage server means is responsive to loss of a mountable data storage element in said shelf layer for retrieving said mountable data storage element from said backup means by reading all transfer units that were stored on said lost mountable data storage element from said mountable backup data storage element.

23. The system of claim 19 wherein said storage server means is responsive to loss of a media in one of said layers for retrieving said lost media from said backup means by reading all transfer units that were stored on said lost media from said mountable backup data storage element.

24. The system of claim 7 wherein said storage server means further comprises:

means for determining a size of said transfer unit as a function of said media on said next lower layer.

25. The system of claim 1 wherein said storage server means comprises:

means, located in each of said file servers, for intercepting a call at a selected file server to data files that have been migrated to said secondary storage means; and

means responsive to said intercepting means for recalling said migrated data files from said secondary storage means to said selected file server.

26. The system of claim 1 wherein said network includes a directory for identifying a storage location of data files on said file servers, said storage server means comprises:

means for writing in said network directory at a directory location for each migrated data file, data indicating that said data file has been migrated to said secondary storage means.

27. The system of claim 26 wherein said storage server means comprises:

means, located in each of said file servers, for intercepting a call at a selected file server to data files that have been stored in said file server; and

means, responsive to said data written in said network directory indicating that said requested data file has been migrated to said secondary storage means, for recalling said requested data file from said secondary storage means to said file server.

28. The system of claim 27 wherein said storage server means comprises:

directory means for storing data indicative of a physical data storage location that identifies a locus in said secondary storage means of each of said data files migrated to said secondary storage means; and

wherein said recalling means comprises;

means for reading said data stored in said directory means to identify a physical data storage location that identifies a locus in said secondary storage means of said requested migrated data file, and

means for transmitting said requested migrated data file from said locus in said secondary storage means to said selected file server.

29. The system of claim 26 wherein said storage server means comprises:

directory means for storing data indicative of a physical data storage location that identifies a locus in said secondary storage means of each of said data files migrated to said secondary storage means; and

wherein said data written by said writing means in said network directory at a directory location for a selected migrated data file identifies a data storage location in said directory means that stores data indicative of a physical data storage location that identifies a locus in said secondary storage means of said selected migrated data file.

30. The system of claim 26 wherein said data written by said writing means in said network directory at a directory location for a selected migrated data file is stored as part of the data file attributes.

31. The system of claim 1 wherein said network includes a directory for identifying a storage location of data files on said file servers, said storage server means includes data file migration means comprising:

means for looking at said network directory to obtain data relating to data files stored on a selected file server, which data indicates various characteristics of said data files.

32. The system of claim 31 wherein said data file migration means further comprises:

means for ordering data files stored on said selected file server into a priority ordering by selected characteristics of said data files.

33. The system of claim 32 wherein each said file server contains a plurality of volumes of data storage, said data file migration means further comprises:

means for reviewing each volume of said at least one file server to identify lowest priority data files stored thereon.

34. The system of claim 33 wherein said data file migration means further comprises:

means for transmitting at least one of said identified lowest priority data files to said secondary storage means.

35. The system of claim 34 wherein said storage server means further comprises:

means for activating said data file migration means for successive lowest priority data files until available memory in a volume of said selected file server is at least as great as a predefined threshold.

36. The system of claim 34 wherein said storage server means further comprises:

means for scheduling activation of said data file migration means on a temporal basis.

37. The system of claim 34 wherein said storage server means further comprises:

means for activating said data file migration means as a function of volume space utilization.

38. The system of claim 34 wherein said storage server means further comprises:

means for activating said data file migration means as a function of activity on said data network.

39. The system of claim 35 wherein said data file migration means further comprises:

means for copying at least one of said priority ordered data files from said selected file server to said secondary storage means; and

means responsive to a subsequent determination of insufficient available data storage space on said selected file server for utilizing data storage space occupied by said copied at least one said priority ordered data files as available data storage space.

40. The system of claim 1 wherein said secondary storage means comprises a multi-layer hierarchical memory, said storage server means comprises:

means for collecting a plurality of data files, that are transmitted to said secondary storage means, into a transfer unit;

means for storing said transfer unit on a first layer of said hierarchy;

wherein said secondary storage means further comprises;

means for storing a duplicate copy of said transfer unit stored on said first layer of said hierarchy.

41. The system of claim 1 wherein said secondary storage means comprises a multi-layer hierarchical memory with a lowest layer of said hierarchy being divided into a plurality of media units for storing data files relocated from hierarchically senior layers, said secondary storage means further comprises:

means, responsive to a one of said plurality of media units exhausting available data storage space thereon, for storing a duplicate copy of said media unit on a media replacement unit element.

42. The system of claim 41 wherein said storage server means is responsive to loss of a media unit for retrieving said lost media unit from said media replacement unit element by reading all data files that were stored on said lost media unit from said media replacement unit element.

43. The system of claim 41 wherein said media replacement unit element comprises a mountable data storage element.

44. The system of claim 1 wherein said secondary storage means comprises a multi-layer hierarchical memory, wherein said layers in said hierarchical memory comprise media of differing characteristics, said storage server means comprises:

means for collecting a plurality of data files, that are transmitted to said secondary storage means, into a transfer unit;

means for storing said transfer unit on a first layer of said hierarchy;

means for automatically relocating said transfer units that are stored on said first layer of said hierarchy to an other layer of said hierarchy as a function of predetermined criteria; and

wherein said automatically relocating means transmits a plurality of said transfer units assembled into a media unit to said other layer in said hierarchy for storage thereon.

45. A method for storing data files in a data storage management system for a data network which functions to interconnect a plurality of file servers, each of which stores data files, which data storage management system comprises a secondary storage element and a storage server element connected to said network, comprising the steps of:

storing data files migrated from said file servers on said secondary storage element;

automatically managing transfer of data files, independent of said file servers, between said plurality of file servers and said secondary storage element; and

collecting a plurality of data files, that are transmitted to said secondary storage element, into a transfer unit.

46. The method of claim 45 wherein said secondary storage means comprises a multi-layer hierarchical memory, wherein said layers in said hierarchical memory comprise media of differing characteristics which include memory devices comprised of media of at least one of the class of:

magnetic media, optical media, and solid-state media.

47. The method of claim 46 wherein said secondary storage means comprises a multi-layer hierarchical memory and wherein at least one of said layers comprises a plurality of data storage elements for storing data files migrated from said file servers and at least one data storage element drive for reading/writing data on a data storage element mounted in said data storage element drive, said method further comprises the step of:

robotically mounting a selected one of said plurality of data storage elements in said data storage element drive means.

48. The method of claim 47 further comprising the step of:

storing in a directory data indicative of a physical data storage location that identifies a locus in said secondary storage element of each of said data files migrated to said secondary storage element.

49. The method of claim 45 wherein said secondary storage element comprises a multi-layer hierarchical memory, wherein said layers in said hierarchical memory comprise media of differing characteristics, said method further comprises:

storing said transfer unit on a first layer of said hierarchy;

automatically relocating data files that are stored on a layer of said hierarchy to an other layer of said hierarchy as a function of predetermined criteria; and

transmitting said transfer unit to said other layer in said hierarchy.

50. The method of claim 48 wherein said method further comprises the step of:

storing in a transfer unit section of said directory data indicative of a correspondence between a data file and a transfer unit in which said data file is located; and

storing in a media object section of said directory data indicative of a correspondence between a transfer unit and a media on which said transfer unit is located.

51. The method of claim 50 wherein said storage server element updates said media object directory when a transfer unit is relocated from said media on which said transfer unit is located to another media in said hierarchy.

52. The method of claim 49 further comprising the step of:

writing each said transfer unit written on said first layer of said hierarchy on to a mountable backup data storage element.

53. The method of claim 52 wherein said step of writing writes a partially filled transfer unit on to said mountable backup data storage element upon expiration of a predetermined time interval absent sufficient data files to fill a transfer unit.

54. The method of claim 53 wherein said step of writing overwrites a partially filled transfer unit on said mountable backup data storage element with a filled transfer unit that includes all data files written in said partially filled transfer unit.

55. The method of claim 54 further comprising the step of:

retrieving, in response to loss of a requested data file in an assigned transfer unit, said requested data file from said mountable backup data storage element by reading at least a portion of said assigned transfer unit from said mountable backup data storage element to recover said lost data file.

56. The method of claim 52 further comprising the step of:

deleting, in response to lack of further available data storage space on a lowest layer of said hierarchy, for deleting at least one selected transfer unit from said lowest layer.

57. The method of claim 56 further comprising the step of:

transferring, in response to addition of a layer of media to said hierarchy lower than said lowest layer, said selected deleted transfer unit from said backup mountable data storage element to said added layer of media by reading said selected deleted transfer unit from said mountable backup data storage element.

58. The method of claim 56 further comprising the step of:

transferring, in response to addition of data storage capacity to said lowest layer, said selected deleted transfer unit from said backup mountable data storage element to said added data storage capacity by reading said selected deleted transfer unit from said mountable backup data storage element.

59. The method of claim 56 further comprising the step of:

deleting, in response to removal of available data storage space on a selected layer of said hierarchy, all transfer units stored on said removed available data storage space from said selected layer.

60. The method of claim 59 further comprising the step of:

transferring, in response to available data storage capacity in a layer lower in said hierarchy than said selected layer, said deleted transfer units from said backup mountable data storage element to said available data storage capacity by reading said deleted transfer units from said mountable backup data storage element.

61. The method of claim 52 wherein at least one of said layers comprises a shelf layer comprising at least one mountable data storage element that contains data files migrated from said file servers and whose locations on said data storage element are maintained in said directory, wherein said shelf layer mountable media comprises said backup mountable media.

62. The method of claim 61 wherein one of said layers comprises a retirement layer comprising at least one mountable data storage element that contains data files migrated from said file servers and whose directory entries are purged from said transfer unit directory.

63. The method of claim 61 further comprising the step of:

retrieving, in response to loss of a mountable data storage element in said shelf layer, said mountable data storage element from said backup mountable data storage element by reading all transfer units that were stored on said lost mountable data storage element from said mountable backup data storage element.

64. The method of claim 61 further comprising the step of:

reading, in response to loss of a media in one of said layers for retrieving said lost media from said backup means, all transfer units that were stored on said lost media from said mountable backup data storage element.

65. The method of claim 49 further comprising the step of:

determining a size of said transfer unit as a function of said media on said next lower layer.

66. The method of claim 45 further comprising the steps of:

intercepting a call at a selected file server to data files that have been migrated to said secondary storage element; and

recalling said migrated data files from said secondary storage element to said selected file server.

67. The method of claim 45 wherein said network includes a directory for identifying a storage location of data files on said file servers, said method further comprises the step of:

writing in said network directory at a directory location for each migrated data file, data indicating that said data file has been migrated to said secondary storage element.

68. The method of claim 67 further comprising the steps of:

intercepting a call at a selected file server to data files that have been stored in said file server; and

recalling, in response to said data written in said network directory indicating that said requested data file has been migrated to said secondary storage means, said requested data file from said secondary storage element to said file server.

69. The method of claim 68 further comprising the steps of:

storing data in a directory, said data indicative of a physical data storage location that identifies a locus in said secondary storage element of each of said data files migrated to said secondary storage element; and

wherein said step of recalling comprises;

reading said data stored in said directory to identify a physical data storage location that identifies a locus in said secondary storage element of said requested migrated data file, and

transmitting said requested migrated data file from said locus in said secondary storage element to said selected file server.

70. The method of claim 67 further comprising the step of:

storing data in a directory, said data indicative of a physical data storage location that identifies a locus in said secondary storage element of each of said data files migrated to said secondary storage element; and

wherein said data written by said step of writing in said network directory at a directory location for a selected migrated data file identifies a data storage location in said directory that stores data indicative of a physical data storage location that identifies a locus in said secondary storage element of said selected migrated data file.

71. The method of claim 67 wherein said data written by said step of writing in said network directory at a directory location for a selected migrated data file is stored as part of the data file attributes.

72. The method of claim 45 wherein said network includes a directory for identifying a storage location of data files on said file servers, said method includes a data file migration step comprising:

looking at said network directory to obtain data relating to data files stored on a selected file server, which data indicates various characteristics of said data files.

73. The method of claim 72 wherein said data file migration step further comprises:

ordering data files stored on said selected file server into a priority ordering by selected characteristics of said data files.

74. The method of claim 73 wherein each said file server contains a plurality of volumes of data storage, said data file migration step further comprises:

reviewing each volume of said at least one file server to identify lowest priority data files stored thereon.

75. The method of claim 74 wherein said data file migration step further comprises:

transmitting at least one of said identified lowest priority data files to said secondary storage element.

76. The method of claim 75 further comprising the step of:

activating said step of data file migration for successive lowest priority data files until available memory in a volume of said selected file server is at least as great as a predefined threshold.

77. The method of claim 75 further comprising the step of:

scheduling activation of said step of data file migration on a temporal basis.

78. The method of claim 75 further comprising the step of:

activating said step of data file migration as a function of volume space utilization.

79. The method of claim 75 further comprising the step of:

activating said step of data file migration as a function of activity on said data network.

80. The method of claim 74 wherein said step of data file migration further comprises:

copying at least one of said priority ordered data files from said selected file server to said secondary storage means; and

utilizing, in response to a subsequent determination of insufficient available data storage space on said selected file server, data storage space occupied by said copied at least one said priority ordered data files as available data storage space.

81. The method of claim 45 wherein said secondary storage element comprises a multi-layer hierarchical memory, said method comprises:

collecting a plurality of data files, that are transmitted to said secondary storage element, into a transfer unit;

storing said transfer unit on a first layer of said hierarchy;

storing a duplicate copy of said transfer unit stored on said first layer of said hierarchy.

82. The method of claim 45 wherein said secondary storage element comprises a multi-layer hierarchical memory with a lowest layer of said hierarchy being divided into a plurality of media units for storing data files relocated from hierarchically senior layers, said method further comprises:

storing, in response to a one of said plurality of media units exhausting available data storage space thereon, a duplicate copy of said media unit on a media replacement unit element.

83. The method of claim 82 further comprising the step of:

reading, in response to loss of a media unit for retrieving said lost media unit from said media replacement unit element, all data files that were stored on said lost media unit from said media replacement unit element.

84. The method of claim 82 wherein said media replacement unit element comprises a mountable data storage element.

85. The method of claim 45 wherein said secondary storage element comprises a multi-layer hierarchical memory, wherein said layers in said hierarchical memory comprise media of differing characteristics, said method comprises:

collecting a plurality of data files, that are transmitted to said secondary storage element, into a transfer unit;

storing said transfer unit on a first layer of said hierarchy;

automatically relocating said transfer units that are stored on said first layer of said hierarchy to an other layer of said hierarchy as a function of predetermined criteria and wherein said plurality of said transfer units are assembled into a media unit as transmitted prior to transmission to said other layer in said hierarchy for storage thereon.