Method and apparatus for an improved file repository

• US 7,761,678 B1
• Filed: 09/29/2004
• Issued: 07/20/2010
• Est. Priority Date: 09/29/2004
• Status: Expired due to Fees

• Current Assignee: RPX Corporation
• Original Assignee: VeriSign, Inc.
• Inventors: Fleming, Jonah Kaj, Bodnar, Eric, Tarantino, Mark, Sheth, Devdutt, Bodmer, Brian
• Primary Examiner(s): Bragdon, Reginald G
• Assistant Examiner(s): Ruiz, Aracelis
• Application Number:
  US10/954,380
• Time in Patent Office: 2,120 Days
• Field of Search: 711/165
• US Class Current: 711/165
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

A method and apparatus for of storing data comprising monitoring a plurality of storage units within a mass storage area and detecting when a storage unit within the mass storage area is overloaded. The method further comprising randomly distributing the data on the overloaded storage unit to the other storage units within the mass storage area.

Claims

1. A method of storing data comprising:

monitoring a plurality of storage units individually addressable for mass data storage as a mass storage area;

detecting when a storage unit in the mass storage area is overloaded;

distributing the data on the overloaded storage unit according to a random spray pattern to other storage units in the mass storage area;

detecting when a new storage unit addressable for mass data storage is added to the plurality of storage units, wherein the new storage unit is not addressable in the mass storage area before being added to the plurality of storage units; and

selecting a data randomly from the plurality of storage units to be stored in the new storage unit when the addition of the new storage unit is detected.

2. The method of claim 1, further comprising:

identifying a storage unit in the mass storage area which is active upon receiving a request for storage of data;

returning a path to the selected storage unit to the requester.

3. The method of claim 2, further comprising:

storing the data on the storage unit;

storing a pointer to the data in a database, including a fully qualified path to the data.

4. The method of claim 2, further comprising:

evaluating a data priority for the data; and

selecting the storage unit based on current storage unit load.

5. The method of claim 4, further comprising:

selecting the storage unit based on storage unit speed.

6. The method of claim 4, wherein data priority comprises frequency of access to the data.

7. The method of claim 4, wherein data priority comprises a subscription level of the user.

8. The method of claim 1, further comprising:

monitoring the mass storage area to determine when a storage unit is at capacity; and

marking the storage unit in the mass storage area as over capacity.

9. The method of claim 8, wherein the capacity is set at a level below 100%, to enable replacement of images with better quality images.

10. The method of claim 1, further comprising:

monitoring access frequency of files in the mass storage area; and

prioritizing removal of the files based on the access frequency.

11. The method of claim 1, further comprising upon receiving data for storage:

determining whether the data is a version of stored data prior to storing the data; and

determining if the new data is better quality than the stored data.

12. The method of claim 11, further comprising one or more of the following:

replacing the stored data with the better quality new data, flagging the stored data indicating there is a better quality new data available, and storing the new better quality data and discarding the stored data.

13. The method of claim 1, further comprising:

real-time monitoring of use of the data on the storage units to prioritize removal of data based on usage.

14. The method of claim 1, further comprising:

assigning a priority to data; and

degrading the priority over time by aging the data.

15. The method of claim 14, wherein the priority is used to determine priority for removal to a different storage unit.

16. The method of claim 1, further comprising:

detecting imminent storage unit failure of a failing storage unit in the mass storage area; and

marking the failing storage unit as disabled; and

randomly distributing the data from the failing storage unit to the other storage units in the mass storage area.

17. The method of claim 1, further comprising:

allowing an administrator to mark a storage unit for retirement; and

randomly distributing the data from the unit to other storage units in the mass storage area.

18. A system comprising:

a storage unit monitor to monitor a plurality of storage units individually addressable for mass data storage in a mass storage area, the storage unit monitor to detect when a new storage unit addressable for mass data storage is added to the plurality of storage units, wherein the new storage unit is not addressable in the mass storage area before being added to the plurality of storage units;

a usage evaluator to detect that a storage unit is running hot; and

a dynamic real locator to redistribute at least part of the data from the storage unit running hot on the mass storage area when the new storage unit is detected, the data being redistributed according to a random spray pattern to relieve the hot storage unit, the dynamic real locator to select a data randomly from the plurality of storage units to be stored in the new storage unit.

19. The system of claim 18, further comprising:

a storage unit selector to identify an available volume in the mass storage area upon receiving a request for storage of data; and

a user interface to return a path to the selected storage unit to the requester.

20. The system of claim 19, further comprising:

a data storage logic to store the data on the selected storage unit; and

a media database to store a pointer to the data on the selected storage unit, including a fully qualified path to the data.

21. The system of claim 19, further comprising:

activity level logic to determine an expected data priority for the data to be stored;

and the storage unit selector to select the storage unit based on current storage unit load.

22. The system of claim 21, further comprising:

the storage unit selector to select the storage unit based on storage unit speed.

23. The system of claim 21, wherein data priority comprises frequency of access to the data.

24. The system of claim 21, wherein data priority comprises a subscription level of the user.

25. The system of claim 18, further comprising:

the storage unit monitor to monitor the mass storage area to determine when a storage unit is at capacity; and

a volume database to indicate that the storage unit in the mass storage area as over capacity.

26. The system of claim 25, wherein the capacity is set at a level below 100%, to enable replacement of images with better quality images.

27. The system of claim 18, further comprising:

the storage unit monitor to monitor access frequency to storage units in the mass storage area; and

prioritization logic to prioritize redistribution of data based on the access frequency.

28. The system of claim 27, further comprising:

a real-time monitor to monitor individual files for frequency of access; and

the prioritization logic to prioritize removal of high activity files from hot storage units.

29. The system of claim 18, further comprising:

the prioritization logic to assign priority to data and degrade the priority over time by aging the data.

30. The system of claim 29, wherein the priority is used to determine priority for removal to a different storage unit.

31. The system of claim 18, further comprising upon receiving data for storage:

a duplicate detector to determine whether the data is a version of stored data prior to storing the data; and

a version control to determine if the new data is better quality than the stored data.

32. The system of claim 31, further comprising:

the version control further to perform one or more of the following;

replace the stored data with the better quality new data, flag the stored data indicating there is a better quality new data available, and store the new better quality data and discarding the stored data.

33. The system of claim 18, further comprising:

the storage unit monitor to detect imminent storage unit failure of a failing storage unit in the mass storage area; and

a volume database to include a marking of the failing storage unit as disabled;

and a dynamic real locator to randomly distribute the data from the failing storage unit to the other storage units in the mass storage area.

34. The system of claim 18, further comprising:

a storage unit monitor to permit an administrator to mark a storage unit for retirement; and

a dynamic real locator to randomly distribute the data from the retired storage unit to other storage units in the mass storage area.

35. The system of claim 18, further comprising:

a media database to store pointers to data on the selected storage units, including a fully qualified path to the data; and

a backup logic to determine when one or more storage units in the mass storage area should be backed up, the backup logic using the media database to select the data for backup.

36. The system of claim 35, further comprising:

a restore logic to restore data from the backup, the restore logic to restore data to its original location.

37. The system of claim 36, wherein the restore logic is further to synchronize the backup pointer information with a current version of the media database, to restore data to its last location.