DIRECTED PLACEMENT OF DATA IN A REDUNDANT DATA STORAGE SYSTEM

• US 20090259665A1
• Filed: 04/09/2008
• Published: 10/15/2009
• Est. Priority Date: 04/09/2008
• Status: Active Grant

• Current Assignee: Harmonic Incorporated
• Original Assignee: Harmonic Incorporated
• Inventors: Howe, John, Mitaru, Alexander, Morris, Charles, Wanigasekara-Mohotti, Don Harschadath
• Granted Patent: US 8,103,628 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/184 implemented as replicated file system

Abstract

A data processing apparatus, comprising a metadata store storing information about files that are stored in a distributed data storage system, and comprising a class database; one or more processing units; logic configured for receiving and storing in the class database a definition of a class of data storage servers comprising one or more subclasses each comprising one or more server selection criteria; associating the class with one or more directories of the data storage system; in response to a data client storing a data file in a directory, binding the class to the data file, determining and storing a set of identifiers of one or more data storage servers in the system that match the server selection criteria, and providing the set of identifiers to the data client.

Claims

1. A data processing apparatus, comprising:

a metadata store storing information about files that are stored in a distributed data storage system, and comprising a class database;

one or more processing units;

logic coupled to the processing unit(s) and to the metadata store and comprising stored sequences of instructions which, when executed by the processing unit(s), cause;

receiving and storing in the class database a definition of a class of data storage servers comprising one or more subclasses each comprising one or more server selection criteria;

associating the class with one or more directories of the data storage system;

in response to a data client storing a data file in a directory, binding the class to the data file, determining and storing a set of identifiers of one or more data storage servers in the system that match the server selection criteria, and providing the set of identifiers to the data client.

2. The apparatus of claim 1, further comprising sequences of instructions which, when executed by the processing unit(s), cause receiving a representation count in the definition of the one or more subclasses, determining a replication factor for the data file based on a sum of the representation counts, and storing a plurality of replicas of the data file in the data storage servers according to the replication factor.

3. The apparatus of claim 1, further comprising sequences of instructions which, when executed by the processing unit(s), cause receiving a representation count of zero in the definition of the one or more subclasses, determining a replication factor for the data file based on a replication factor property of the data file that is inherited from a directory in which the data file was created, and storing a plurality of replicas of the data file in the data storage servers according to the replication factor.

4. The apparatus of claim 1, further comprising sequences of instructions which, when executed by the processing unit(s), cause determining and storing the set of identifiers based further on one or more best availability criteria that are applied only to the data storage servers in the system that match the server selection criteria.

5. The apparatus of claim 1, wherein the server selection criteria comprise processor speed, owner and location.

6. The apparatus of claim 1, further comprising sequences of instructions which, when executed by the processing unit(s), cause automatically storing the data file only on one or more first data storage servers that are identified in the set of identifiers.

7. The apparatus of claim 6, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

receiving an updated definition of one or more of the subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the updated server selection criteria;

causing automatically storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers.

8. The apparatus of claim 1, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

receiving a representation count in the definition of the one or more subclasses, determining a first replication factor for the data file based on a sum of the representation counts, and storing a plurality of replicas of the data file in the data storage servers according to the first replication factor;

receiving an updated definition of one or more of the subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the updated server selection criteria, and determining a second replication factor for the data file based on a sum of the representation counts in the updated definition;

storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers, in a plurality of replicas according to the second replication factor.

9. The apparatus of claim 1, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

receiving an updated file attribute for the data file that comprises an updated class identifier;

determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the server selection criteria of the one or more subclasses of the class of the updated class identifier;

storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers.

10. The apparatus of claim 9, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

determining a replication factor for the data file based on a sum of representation counts of the one or more subclasses of the updated class identifier;

storing the data file only on the second data storage servers that are identified in the updated set of identifiers, in a plurality of replicas according to the replication factor.

12. The storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause receiving a representation count in the definition of the one or more subclasses, determining a replication factor for the data file based on a sum of the representation counts, and storing a plurality of replicas of the data file in the data storage servers according to the replication factor.

13. The storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause receiving a representation count of zero in the definition of the one or more subclasses, determining a replication factor for the data file based on a replication factor property of the data file that is inherited from a directory in which the data file was created, and storing a plurality of replicas of the data file in the data storage servers according to the replication factor.

14. The storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause determining and storing the set of identifiers based further on one or more best availability criteria that are applied only to the data storage servers in the system that match the server selection criteria.

15. The storage medium of claim 10, wherein the server selection criteria comprise processor speed, owner and location.

16. The storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause automatically storing the data file only on one or more first data storage servers that are identified in the set of identifiers.

17. The storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

receiving an updated definition of one or more of the subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the updated server selection criteria;

causing automatically storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers.

18. The storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

receiving a representation count in the definition of the one or more subclasses, determining a first replication factor for the data file based on a sum of the representation counts, and storing a plurality of replicas of the data file in the data storage servers according to the first replication factor;

receiving an updated definition of one or more of the subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the updated server selection criteria, and determining a second replication factor for the data file based on a sum of the representation counts in the updated definition;

storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers, in a plurality of replicas according to the second replication factor.

19. The storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

receiving an updated file attribute for the data file that comprises an updated class identifier;

determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the server selection criteria of the one or more subclasses of the class of the updated class identifier;

storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers.

20. The storage medium of claim 19, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

determining a replication factor for the data file based on a sum of representation counts of the one or more subclasses of the updated class identifier;

storing the data file only on the second data storage servers that are identified in the updated set of identifiers, in a plurality of replicas according to the replication factor.

11. A computer-readable data storage medium comprising stored sequences of instructions which, when executed by one or more processing unit(s), cause:

receiving and storing, in a class database in a metadata store storing information about files that are stored in a distributed data storage system, a definition of a class of data storage servers comprising one or more subclasses each comprising one or more server selection criteria;

associating the class with one or more directories of the data storage system;

in response to a data client storing a data file in a directory, binding the class to the data file, determining and storing a set of identifiers of one or more data storage servers in the system that match the server selection criteria, and providing the set of identifiers to the data client.

21. A computer-implemented method, comprising:

receiving and storing, in a class database in a metadata store storing information about files that are stored in a distributed data storage system, a definition of a class of data storage servers comprising one or more subclasses each comprising one or more server selection criteria;

associating the class with one or more directories of the data storage system;

in response to a data client storing a data file in a directory, binding the class to the data file, determining and storing a set of identifiers of one or more data storage servers in the system that match the server selection criteria, and providing the set of identifiers to the data client.

22. The method of claim 21, further comprising receiving a representation count in the definition of the one or more subclasses, determining a replication factor for the data file based on a sum of the representation counts, and storing a plurality of replicas of the data file in the data storage servers according to the replication factor.

23. The method of claim 21, further comprising receiving a representation count of zero in the definition of the one or more subclasses, determining a replication factor for the data file based on a replication factor property of the data file that is inherited from a directory in which the data file was created, and storing a plurality of replicas of the data file in the data storage servers according to the replication factor.

24. The method of claim 21, further comprising determining and storing the set of identifiers based further on one or more best availability criteria that are applied only to the data storage servers in the system that match the server selection criteria.

25. The method of claim 21, further comprising automatically storing the data file only on one or more first data storage servers that are identified in the set of identifiers.

26. The method of claim 25, further comprising:

receiving an updated definition of one or more of the subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the updated server selection criteria;

causing automatically storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers.

27. The method of claim 21, further comprising:

receiving a representation count in the definition of the one or more subclasses, determining a first replication factor for the data file based on a sum of the representation counts, and storing a plurality of replicas of the data file in the data storage servers according to the first replication factor;

receiving an updated definition of one or more of the subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the updated server selection criteria, and determining a second replication factor for the data file based on a sum of the representation counts in the updated definition;

storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers, in a plurality of replicas according to the second replication factor.

28. The method of claim 21, further comprising:

receiving an updated file attribute for the data file that comprises an updated class identifier;

determining and storing an updated set of identifiers of one or more different data storage servers in the system that match the server selection criteria of the one or more subclasses of the class of the updated class identifier;

storing the data file only on one or more second data storage servers that are identified in the updated set of identifiers.

29. The method of claim 28, further comprising:

determining a replication factor for the data file based on a sum of representation counts of the one or more subclasses of the updated class identifier;

storing the data file only on the second data storage servers that are identified in the updated set of identifiers, in a plurality of replicas according to the replication factor.