Method and apparatus for an improved file repository

• US 8,082,412 B2
• Filed: 05/04/2010
• Issued: 12/20/2011
• Est. Priority Date: 09/29/2004
• Status: Expired due to Fees

• Current Assignee: RPX Corporation
• Original Assignee: VeriSign, Inc.
• Inventors: Bodmer, Brian, Bodnar, Eric, Tarantino, Mark, Fleming, Jonah Kaj, Sheth, Devdutt
• Primary Examiner(s): Bragdon, Reginald
• Assistant Examiner(s): Ruiz, Aracelis
• Application Number:
  US12/773,111
• Publication Number:
  US 20100218040A1
• Time in Patent Office: 595 Days
• Field of Search: None
• US Class Current: 711/165
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

A method and apparatus for of storing data comprising monitoring a plurality of storage units within a mass storage area and detecting when a storage unit within the mass storage area is overloaded. The method further comprising randomly distributing the data on the overloaded storage unit to the other storage units within the mass storage area.

Claims

1. A method of storing data comprising:

monitoring a plurality of storage units individually addressable for mass data storage as a mass storage area;

detecting when a storage unit in the mass storage area is overloaded; and

distributing the data on the overloaded storage unit according to a random spray pattern to other storage units in the mass storage area.

2. The method of claim 1, further comprising:

detecting when a new storage unit addressable for mass data storage is added to the plurality of storage units; and

selecting data from the plurality of storage units to be stored in the new storage unit when the addition of the new storage unit is detected.

3. The method of claim 2, wherein selecting data from the plurality of storage units to be stored in the new storage unit when the addition of the new storage unit is detected is done randomly.

4. The method of claim 1, further comprising:

monitoring the mass storage area to determine when a storage unit is at capacity; and

marking the storage unit in the mass storage area.

5. The method of claim 4, wherein the capacity is set at a level below 100%.

6. The method of claim 1, further comprising upon receiving new data for storage:

determining whether the new data is a version of stored data prior to storing the data; and

determining if the new data is better quality than the stored data.

7. The method of claim 6, further comprising one or more of the following:

replacing the stored data with the new data if the new data is better quality,

flagging the stored data indicating there is a better quality new data available, or

storing the new better quality data and discarding the stored data.

8. The method of claim 1, further comprising:

assigning a priority to data; and

degrading the priority over time by aging the data.

9. The method of claim 1, further comprising:

detecting imminent storage unit failure of a failing storage unit in the mass storage area;

distributing the data from the failing storage unit to the other storage units in the mass storage area.

10. The method of claim 1, further comprising:

marking a storage unit for retirement in response to a command from an administrator; and

distributing the data from the marked unit to other storage units in the mass storage area.

11. A system comprising:

a non-transitory computer-readable medium housing a volume database for tracking a plurality of storage units containing stored data and individually addressable for mass data storage in a mass storage area;

a storage unit monitor to monitor the storage units, the storage unit monitor detecting when a new storage unit addressable for mass data storage is added to the plurality of storage units and updating the volume database with information about the new storage unit;

a usage evaluator to detect that a storage unit is running hot; and

a dynamic reallocator to redistribute at least part of the data from the storage unit running hot to other storage units in the mass storage area, and, in response to the detection of a new storage unit, to randomly select data from the plurality of storage units to be stored in the new storage unit.

12. The system of claim 11, wherein the dynamic reallocator distributes data from the storage unit running hot according to a random spray pattern.

13. The system of claim 11, wherein the new storage unit comprises a storage unit not addressable in the mass storage area before being added to the plurality of the storage units.

14. The system of claim 11, further comprising:

activity level logic to determine an expected data priority for the data to be stored; and

a storage unit selector to select the storage unit based on current storage unit load.

15. The system of claim 11, wherein the storage unit monitor comprises a storage unit monitor to:

monitor the mass storage area to determine when a storage unit is at capacity; and

to update the volume database based on the capacity information of the storage unit in the mass storage area.

16. The system of claim 15, wherein the capacity is set at a level below 100%.

17. The system of claim 11, wherein:

the storage unit monitor comprises a storage unit monitor to monitor access frequency to storage units in the mass storage area; and

the system comprises prioritization logic to prioritize redistribution of data based on the access frequency.

18. The system of claim 17, further comprising:

a real-time monitor to monitor individual files for frequency of access; and

the prioritization logic comprises prioritization logic to prioritize removal of high activity files from hot storage units.

19. The system of claim 11, further comprising:

a duplicate detector to, upon receipt of new data, determine whether the new data is a version of the stored data; and

a version control unit to determine if the new data is better quality than the stored data.

20. The system of claim 19, wherein the version control unit performs one or more of the following:

replace the stored data with the better quality new data, flag the stored data indicating there is a better quality new data available, or store the new better quality data and discard the stored data.

21. The system of claim 11, wherein:

the storage unit monitor comprises storage unit monitor to detect imminent storage unit failure of a failing storage unit in the mass storage area and to mark the failing storage unit as disabled in the volume database; and

the dynamic reallocator comprises a dynamic reallocator to distribute the data from the failing storage unit to the other storage units in the mass storage area.

22. The system of claim 11, wherein:

the storage unit monitor comprises a storage unit monitor to permit an administrator to mark a storage unit for retirement; and

the dynamic reallocator comprises a dynamic reallocator to distribute the data from the retired storage unit to other storage units in the mass storage area.

23. The system of claim 11, further comprising:

a media database to store pointers to the randomly selected data stored in the new storage units, including a fully qualified path to the data; and

a backup logic unit to determine when one or more storage units in the mass storage area should be backed up, the backup logic unit using the media database to select the storage units for backup.

24. A file repository system comprising:

mass storage media storing data; and

a media database coupled with the mass storage media, comprising;

a file location to locate the data in the mass storage media of the file repository, the file location including a fully qualified path to the data;

a date indicating when the data was stored on the mass storage media; and

a backup logic, the backup logic to query the media database to determine whether to backup the data from the mass storage media based on the file location, the date, or a combination thereof, and to update a file location and a date in the media database for the date when the data is backed up.

25. The file repository system of claim 24, wherein the backup logic uses the media database to identify data to be backed up, without scanning the mass storage area.

26. The file repository system of claim 24, wherein the storage unit monitor uses the media database:

to monitor a plurality of storage units in the mass storage array, and

to relocate data from an overloaded storage unit to one or more other storage units.

27. The file repository system of claim 24, further comprising a restore logic to monitor the mass storage media, the restore logic:

performing a full restore of all data of a storage drive when the storage drive is determined to be failing,

wherein the full restore includes at least one of;

copying all data stored on the failing storage drive to a replacement storage drive, or randomly distributing all data stored on the failing storage drive to other storage drives of the mass storage media;

performing a partial restore of a storage drive when data on the storage drive is determined to be corrupted; and

updating the file location and date in the database for data that has been restored.

28. A system comprising:

a non-transitory computer-readable medium housing a volume database for tracking a plurality of storage units;

a storage unit monitor to monitor the plurality of storage units individually addressable for mass data storage in a mass storage area, the storage unit monitor detecting when a new storage unit addressable for mass data storage is added to the plurality of storage units and updating the volume database with information about the new storage unit;

a usage evaluator to detect that a storage unit is running hot; and

a dynamic reallocator to redistribute at least part of the data from the storage unit running hot on the mass storage area, the dynamic reallocator, in response to the detection of a new storage unit, selecting a data from the plurality of storage units to be stored in the new storage unit, the new storage unit being not addressable in the mass storage area before being added to the plurality of the storage units.