Data storage system

• US 20060200700A1
• Filed: 08/18/2004
• Published: 09/07/2006
• Est. Priority Date: 08/18/2003
• Status: Active Grant

• Current Assignee: Computer Associates Think Inc. (Broadcom, Inc.)
• Original Assignee: Computer Associates Think Inc. (Broadcom, Inc.)
• Inventors: Malcolm, Peter Bryan
• Granted Patent: US 8,099,394 B2
• Time in Patent Office: Days
• US Class Current: 714/38
• CPC Class Codes: G06F 16/125 characterised by the use of retention policies retention policies for HSM systems G06F16/185

Abstract

A system for efficiently storing application data is disclosed which allows a practically infinite number of physically separate storage devices to be used as archival storage means by one or more application programs. Application data is organized among the devices such that files with a similar expiry date (that is the date beyond which the files are no longer required to be archived) are grouped together on the same physical device, such that after the expiry date the device used for such files may be erased and re-used. Optionally, files with particular content-types may be grouped on the same physical device, or among physical devices of the same type.

Claims

1. A method of operating a data processing system, the system comprising one or more application programs requiring persistent data storage for data files of application data, a plurality of storage devices each accessible via a computer network to one or more computers executing said application programs, and a broker program,

wherein the method comprises receiving, by means of said broker program, a request for storage of a data file of application data, and

selecting for said data file which of said storage devices will be used to store said data file in accordance with the characteristics of the application data to be stored and the state of said storage devices, wherein said characteristics of the data to be stored include an expiry date, beyond which the application data is no longer required and may be deleted.

2. A method according to claim 1, comprising monitoring, by means of the broker program, the remaining storage space available on each of said storage devices, to distinguish between in-use storage devices which have had data files written to them and empty storage devices which have not.

3. A method according to claim 2, wherein said selecting step comprises selecting in-use storage devices in preference to empty storage devices.

4. A method according to claim 2, comprising:

monitoring how much data is waiting to be written to each storage device, to detect an overload condition in the process of writing the data; and

selecting, if an overload condition is detected for a storage device selected for storage, a different storage device for storage.

5. A method according to claim 4, wherein said selecting step comprises selecting in-use storage devices in preference to empty storage devices.

6. A method according to claim 1, comprising:

storing, for each storage device, the latest expiry date of data files stored on that device, or of data files that are to be stored; and

permitting application data to be stored on a storage device if its expiry date is within a predetermined range of said latest expiry date;

such that application data with similar expiry dates can be stored together and when such similar expiry dates have passed the storage device can be erased and re-used.

7. A method according to claim 6, comprising selecting another storage device for storage, if the expiry date of said application data is outside of the predetermined range of said latest expiry date.

8. A method according to claim 1, comprising storing for each of said storage devices a target expiry date, and selecting which of said storage devices to use in dependence on a comparison of said expiry date and said target expiry date.

9. A method according to claim 8, comprising preventing application data from being stored on a storage device, if the target expiry date for that storage device is earlier than said expiry date.

10. A method according to claim 8, comprising preventing application data from being stored on a storage device if the target expiry date for that storage device is earlier than said expiry date by more than a predetermined margin.

11. A method according to claim 1, comprising, after said data file has been written to said storage device, preventing said file from being modified or deleted until said expiry date has passed.

12. A method according to claim 6, comprising, after said latest expiry date has passed, erasing the contents of said storage device.

13. A method according to claim 1, wherein said characteristics of the data to be stored include a classification of the content of said application data.

14. A method according to claim 13, comprising:

storing for each of said storage devices, a target content type;

comparing said classification of the content of said application data and said target content type; and

preventing said application data from being stored in a storage device if the target content type for that device and the classification do not match.

15. A method according to claim 1, wherein said characteristics of the data to be stored include the application program which requires its storage.

16. A method according to claim 1, wherein said characteristics of the data to be stored include the size of the application data.

17. A method according to claim 1, comprising writing said application data to and/or reading said application data from said storage devices directly by means of said application programs via said computer network.

18. A method according to claim 1, comprising notifying said application program of the storage device used to store said data file as determined by said broker program, such that said application program can store means to identify the device.

19. A method according to claim 18, wherein said data files are retrieved from said storage device by said application program directly via said computer network and without reference to said broker program.

20. A method according to claim 1, comprising determining, by means of the broker program, the directory location for storage of said data file on said storage devices.

21. A method according to claim 20, comprising creating, by means of said broker program, the directory entry for said data file in said directory location in anticipation of data being written to said file by the application program requesting storage.

22. A method according to claim 20, comprising notifying, by means of the broker program, said directory location of said data file to said application program for storage by said application program.

23. A method according to claim 20, wherein the directory entry for said data file in said directory location is created by said application program prior to data being written by it.

24. A method according to claim 1, wherein the state of said storage devices includes the current availability of such devices for data to be written thereto.

25. A method according to claim 1, wherein the state of said storage devices includes the amount of free space available in said storage devices.

26. A method according to claim 1, wherein the state of said storage devices includes the rate at which data is being read from and/or written to such devices.

27. A method according to claim 1, comprising monitoring the status of said storage devices, detecting when new storage devices have been added, and making these available for storage.

28. A method of operating a data processing system to store data, comprising:

receiving a request from an application program, among one or more application programs, for storage of a data file of application data;

determining one or more characteristics of said application data, including an expiry date beyond which the application data is no longer required;

monitoring the status of storage devices in a plurality of storage devices; and

selecting a storage device from said plurality of storage devices to store a data file of application data in accordance with the characteristics of the data to be stored and the state of said storage devices;

wherein said data file is stored on the selected storage device.

29. A system for storing data comprising a broker program for receiving requests for storage of a data file of application data from one or more application programs requiring persistent data storage, and for selecting for said data file which of a plurality of storage devices, accessible to said one or more application programs and said broker program via a computer network, will be used to store said data files;

wherein said Broker program selects said storage device in dependence on the characteristics of the data to be stored and the state of said storage devices, wherein said characteristics of the data to be stored include an expiry date, beyond which the application data is no longer be required and may be deleted;

such that said application data is stored in the allocated storage device as a data file.

30. A system according to claim 29, comprising a look-up table accessible by the broker program, wherein the look-up table stores the remaining storage space available on each of said storage devices such that in-use storage devices which have had data files written to them and empty storage devices which have not are distinguished from each other.

31. A system according to claim 30, wherein said broker program is operable to select in-use storage devices in preference to empty storage devices.

32. A system according to claim 30, wherein the look-up table stores how much data is waiting to be written to each storage device, such that an overload condition in the process of writing the data to a storage device can be detected;

wherein the broker program is operable to select, if an overload condition is detected for a storage device selected for storage, a different storage device for storage.

33. A system according to claim 32, wherein said broker program is operable to select in-use storage devices in preference to empty storage devices.

34. A system according to claim 29, comprising a look-up table accessible by said broker program, wherein said look-up table stores, for each storage device, the latest expiry date of data files stored on that device, or of data files that are to be stored, and

wherein the broker program is operable to permit application data to be stored on a storage device if its expiry date is within a predetermined range of said latest expiry date, such that application data with similar expiry dates can be stored together and when such similar expiry dates have passed the storage device can be erased and re-used.

35. A system according to claim 34, wherein said broker program is operable to select another storage device for storage, if the expiry date of said application data is outside of the pre-determined range of said latest expiry date.

36. A system according to claim 29, comprising a look-up table accessible by said broker program, wherein said look-up table stores, for each storage device a target expiry date, and said broker program is operable to select which of said storage devices to use in dependence on a comparison on said expiry date and said target expiry date.

37. A system according to claim 36, wherein said broker program is operable to prevent application data from being stored on a storage device, if the target expiry date for that storage device is earlier than said expiry date.

38. A system according to claim 36, wherein said broker program is operable to prevent application data from being stored on a storage device if the target expiry date for that storage device is earlier than said expiry date by more than a predetermined margin.

39. A system according to claim 29, wherein said broker program is operable to prevent, after said data file has been written to said storage device, said file from being modified or deleted until said expiry date has passed.

40. A system according to claim 34, wherein said broker program is operable to erase, after said latest expiry date has passed, the contents of said storage device.

41. A system according to claim 29, wherein said characteristics of the data to be stored include a classification of the content of said application data.

42. A system according to claim 41, comprising a look-up table accessible by said broker program, wherein said look-up table stores a target content type;

and wherein said broker program is operable to compare said classification of the content of said application data and said target content type, and prevent said application data from being stored in a storage device if the target content type for that device and the classification do not match.

43. A system according to claim 29, wherein said characteristics of the data to be stored include the application program which requires its storage.

44. A system according to claim 29, wherein said characteristics of the data to be stored include the size of the application data.

45. A system according to claim 29, wherein said broker program is operable to notify said application program of the storage device used to store said data file as determined by said broker program, such that said application program can store means to identify the device.

46. A system according to claim 29, wherein the broker program is operable to determine the directory location for storage of said data file on said storage devices.

47. A system according to claim 46, wherein the broker program is operable to notify said directory location of said data file to said application program for storage by said application program.

48. A system according to claim 29, wherein the state of said storage devices includes the current availability of such devices for data to be written thereto.

49. A system according to claim 29, wherein the state of said storage devices includes the amount of free space available in said storage devices.

50. A system according to claim 29, wherein the state of said storage devices includes the rate at which data is being read from and/or written to such devices.

51. A system according to claim 29, wherein the broker program is operable to monitor the status of said storage devices, detect when new storage devices have been added, and make these available for storage.

52. A system for storing data, comprising:

one or more application programs requiring persistent data storage for data files of application data;

a plurality of storage devices each accessible via a computer network to one or more computers executing said application programs; and

a broker program for receiving a request from an application program for storage of a data file of application data; and

for selecting for said data file which of said storage devices will be used to store said data file in accordance with the characteristics of the application data to be stored and the state of the storage devices, the characteristics of the application data including an expiry date beyond which the application data is no longer required.

53. A system according to claim 52, comprising an Application Program Interface running on the one or more computers to pass commands to and from the broker program and the application program.

54. A computer program product for controlling a computer in a data storage system, said computer being operable to receive requests from one or more application programs, running on one or more computers, and requiring persistent data storage for data files of application data, and operable to monitor a plurality of storage devices,

said computer program product comprising a recording medium readable by said computer, having program code stored thereon which when executed on said computer configures said computer to perform the steps of:

receive a request for storage of a data file of application data from an application program, and

select for said data file which of said storage devices will be used to store said data file in accordance with the characteristics of the application data to be stored and the state of said storage devices, wherein said characteristics of the data to be stored include an expiry date, beyond which the application data is no longer required and may be deleted.

55. A computer program product according to claim 54, which when executed on said computer configures said computer to monitor the remaining storage space available on each of said storage devices to distinguish between in-use storage devices which have had data files written to them, and empty storage devices which have not.

56. A computer program product according to claim 55, wherein said selecting step comprises selecting in-use storage devices in preference to empty storage devices.

57. A computer program product according to claim 55, which when executed on said computer configures said computer to monitor how much data is waiting to be written to each storage device, to detect an overload condition in the process of writing the data;

and

select, if an overload condition is detected for a storage device selected for storage, a different storage device for storage.

58. A computer program product according to claim 57, wherein said selecting step comprises selecting in-use devices in preference to empty devices.

59. A computer program product according to claim 54, which when executed on said computer configures said computer to:

store, for each storage device, the latest expiry date of data files stored on that device, or of data files that are to be stored; and

permit application data to be stored on a storage device if its expiry date is within a predetermined range of said latest expiry date;

such that application data with similar expiry dates can be stored together and when such similar expiry dates have passed the storage device can be erased and re-used.

60. A computer program product according to claim 59, which when executed on said computer configures said computer to:

select another storage device for storage, if the expiry date of said application data is outside of the pre-determined range of said latest expiry date.

61. A computer program product according to claim 58, comprising storing for each of said storage devices a target expiry date, and selecting which of said storage devices to use in dependence on a comparison on said expiry date and said target expiry date.

62. A computer program product according to claim 61, which when executed on said computer configures said computer to prevent application data from being stored on a storage device, if the target expiry date for that storage device is earlier than said expiry date.

63. A computer program product according to claim 61, which when executed on said computer configures said computer to prevent application data from being stored on a storage device if the target expiry date for that storage device is earlier than said expiry date by more than a predetermined margin.

64. A computer program product according to claim 57, which when executed on said computer configures said computer to prevent, after said data file has been written to said storage device, said file from being modified or deleted until said expiry date has passed.

65. A computer program product according to claim 59, which when executed on said computer configures said computer to erase the contents of said storage device, after said latest expiry date has passed.

66. A computer program product according to claim 54, wherein said characteristics of the data to be stored include a classification of the content of said application data.

67. A computer program product according to claim 66, which when executed on said computer configures said computer to:

store for each of said storage devices, a target content type;

compare said classification of the content of said application data and said target content type; and

prevent said application data from being stored in a storage device if the target content type for that device and the classification do not match.

68. A computer program product according to claim 54, wherein said characteristics of the data to be stored include the application program which requires its storage.

69. A computer program product according to claim 54, wherein said characteristics of the data to be stored include the size of the application data.

70. A computer program product according to claim 54, which when executed on said computer configures said computer to notify said application program of the storage device used to store said data file as determined by said broker program, such that said application program can store means to identify the device.

71. A computer program product according to claim 54, which when executed on said computer configures said computer to determine the directory location for storage of said data file on said storage devices.

72. A computer program product according to claim 71, which when executed on said computer configures said computer to notify said directory location of said data file to said application program for storage by said application program.

73. A computer program product according to claim 54, wherein the state of said storage devices includes the current availability of such devices for data to be written thereto.

74. A computer program product according to claim 54, wherein the state of said storage devices includes the amount of free space available in said storage devices.

75. A computer program product according to claim 54, wherein the state of said storage devices includes the rate at which data is being read from and/or written to such devices.

76. A computer program product according to claim 54, which when executed on said computer configures said computer to monitor the status of said storage devices, detecting when new storage devices have been added, and making these available for storage.

77. A computer program product for controlling a computer in a data storage system, said computer program product comprising a recording medium readable by said computer, having program code stored thereon which when executed on said computer configures said computer to perform the steps of:

receiving a request from an application program, among one or more application programs, for storage of a data file of application data;

determining one or more characteristics of said application data;

monitoring ;

he status of storage devices in a plurality of storage devices;

selecting a storage device from said plurality of storage devices to store a data file of application data in accordance with the characteristics of the data to be stored and the state of said storage devices, the characteristics of the application data including an expiry date beyond which the application data is no longer required;

wherein said data file is stored on the selected storage device.

78. A method of storing data substantially as described herein and with reference to the drawings.

79. A system for storing data substantially as described herein and with reference to the drawings.

80. A computer program product for controlling a computer, in a system for storing data, substantially as described herein and with reference to the drawings.