Method and Apparatus for an Improved File Repository

• US 20100218040A1
• Filed: 05/04/2010
• Published: 08/26/2010
• Est. Priority Date: 09/29/2004
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: VeriSign, Inc.
• Inventors: Sheth, Devdutt, Tarantino, Mark, Bodmer, Brian, Bodnar, Eric, Fleming, Jonah Kaj
• Granted Patent: US 8,082,412 B2
• Time in Patent Office: Days
• US Class Current: 714/7
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

A method and apparatus for of storing data comprising monitoring a plurality of storage units within a mass storage area and detecting when a storage unit within the mass storage area is overloaded. The method further comprising randomly distributing the data on the overloaded storage unit to the other storage units within the mass storage area.

Claims

1. A method of storing data comprising:

monitoring a plurality of storage units individually addressable for mass data storage as a mass storage area;

detecting when a storage unit in the mass storage area is overloaded;

distributing the data on the overloaded storage unit to other storage units in the mass storage area;

detecting when a new storage unit addressable for mass data storage is added to the plurality of storage units; and

selecting a data randomly from the plurality of storage units to be stored in the new storage unit when the addition of the new storage unit is detected.

2. The method of claim 1, wherein distributing data on the overloaded storage unit is done according to a random spray pattern.

3. The method of claim 1, wherein the new storage unit is not addressable in the mass storage area before being added to the plurality of the storage units.

4. The method of claim 1, further comprising:

monitoring the mass storage area to determine when a storage unit is at capacity; and

marking the storage unit in the mass storage area.

5. The method of claim 4, wherein the capacity is set at a level below 100%.

6. The method of claim 1, further comprising upon receiving data for storage:

determining whether the data is a version of stored data prior to storing the data; and

determining if the new data is better quality than the stored data.

7. The method of claim 6, further comprising one or more of the following:

replacing the stored data with the better quality new data, flagging the stored data indicating there is a better quality new data available, and storing the new better quality data and discarding the stored data.

8. The method of claim 1, further comprising:

assigning a priority to data; and

degrading the priority over time by aging the data.

9. The method of claim 1, further comprising:

detecting imminent storage unit failure of a failing storage unit in the mass storage area;

marking the failing storage unit; and

distributing the data from the failing storage unit to the other storage units in the mass storage area.

10. The method of claim 1, further comprising:

allowing an administrator to mark a storage unit for retirement; and

distributing the data from the unit to other storage units in the mass storage area.

11. A system comprising:

a storage unit monitor to monitor a plurality of storage units individually addressable for mass data storage in a mass storage area, the storage unit monitor to detect when a new storage unit addressable for mass data storage is added to the plurality of storage units;

a usage evaluator to detect that a storage unit is running hot; and

a dynamic reallocator to redistribute at least part of the data from the storage unit running hot on the mass storage area,

wherein the dynamic reallocator, in response to the detection of a new storage unit, to select a data randomly from the plurality of storage units to be stored in the new storage unit.

12. The system of claim 11, wherein the dynamic reallocator distributes data from the storage unit running hot according to a random spray pattern.

13. The system of claim 11, wherein the new storage unit is not addressable in the mass storage area before being added to the plurality of the storage units.

14. The system of claim 11, further comprising:

activity level logic to determine an expected data priority for the data to be stored; and

a storage unit selector to select the storage unit based on current storage unit load.

15. The system of claim 11, further comprising:

the storage unit monitor to monitor the mass storage area to determine when a storage unit is at capacity; and

a volume database to indicate that the storage unit in the mass storage area as at capacity.

16. The system of claim 15, wherein the capacity is set at a level below 100%.

17. The system of claim 11, further comprising:

the storage unit monitor to monitor access frequency to storage units in the mass storage area; and

prioritization logic to prioritize redistribution of data based on the access frequency.

18. The system of claim 17, further comprising:

a real-time monitor to monitor individual files for frequency of access; and

the prioritization logic to prioritize removal of high activity files from hot storage units.

19. The system of claim 11, further comprising upon receiving data for storage:

a duplicate detector to determine whether the data is a version of stored data prior to storing the data; and

a version control to determine if the new data is better quality than the stored data.

20. The system of claim 19, further comprising:

the version control further to perform one or more of the following;

replace the stored data with the better quality new data, flag the stored data indicating there is a better quality new data available, and store the new better quality data and discarding the stored data.

21. The system of claim 11, further comprising:

the storage unit monitor to detect imminent storage unit failure of a failing storage unit in the mass storage area; and

a volume database to include a marking of the failing storage unit as disabled; and

a dynamic reallocator to distribute the data from the failing storage unit to the other storage units in the mass storage area.

22. The system of claim 11, further comprising:

a storage unit monitor to permit an administrator to mark a storage unit for retirement; and

a dynamic reallocator to distribute the data from the retired storage unit to other storage units in the mass storage area.

23. The system of claim 11, further comprising:

a media database to store pointers to data on the selected storage units, including a fully qualified path to the data; and

a backup logic to determine when one or more storage units in the mass storage area should be backed up, the backup logic using the media database to select the data for backup.

27. The file repository system of claim 22, further comprising:

a restore logic to monitor the mass storage media to determine when a restoration of a storage drive or data is required, the restore logic to perform a full restore of all data of a failing storage drive and a partial restore for data that is determined to be corrupted;

wherein the full restore to restore all data stored on the failing storage drive to a replacement storage drive, to randomly distribute all data stored on the failing storage drive to remaining storage drives of the mass storage media, or a combination thereof;

wherein the partial restore to restore corrupted data; and

wherein the restore logic to update the file location and date in the database for data that has been restored.

24. A file repository system comprising:

a mass storage media, data being stored in the mass storage media;

a media database coupled with the mass storage media, comprising;

a file location to locate the data in the mass storage media of the file repository, the file location including a fully qualified path to the data;

a date indicating when the data was stored on the mass storage media; and

a backup logic, the backup logic to query the media database to determine whether to backup the data from the mass storage media based on the file location, the date, or a combination thereof, and to update a file location and a date in the media database for the date when the data is backed up.

25. The file repository system of claim 24, wherein the media database is used by a backup logic to identify data to be backed up without scanning the mass storage area.

26. The file repository system of claim 24, wherein the media database system is coupled to a storage unit monitor to monitor a plurality of storage units in the mass storage array, the media database used to relocate data from an overloaded storage unit to one or more other storage units.