HETEROGENEOUS DISTRIBUTED FILE SYSTEM USING DIFFERENT TYPES OF STORAGE MEDIUMS

• US 20150347451A1
• Filed: 08/30/2014
• Published: 12/03/2015
• Est. Priority Date: 06/03/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: LEE, Jaehwan, KI, Yang Seok
• Granted Patent: US 9,773,014 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

According to one general aspect, a system may include a plurality of data nodes. Each data node may include either or both of a first storage medium and a second storage medium that is slower than the first storage medium. Each data node may be configured to store a piece data in either the first storage medium or the second storage medium. The system may be configured to store a plurality of copies of an active piece of data within two or more data nodes. A fast copy of the active piece of data may be stored by a first storage medium of a first data node. One or more slow copies of the active piece of data may be stored by respective second storage mediums of one or more respective other data nodes.

Claims

1. A system comprising:

a plurality of data nodes, each data node comprising either or both of a first storage medium and a second storage medium that is slower than the first storage medium, and each data node configured to store a piece data in either the first storage medium or the second storage medium; and

wherein the system is configured to;

store a plurality of copies of an active piece of data within two or more data nodes,

wherein a fast copy of the active piece of data is stored by a first storage medium of a first data node, and

wherein one or more slow copies of the active piece of data are stored by respective second storage mediums of one or more respective other data nodes.

2. The system of claim 1 wherein the system is configured to:

receive a write request to write the piece of data to the system,

in response to the write request, select the first data node to store the fast copy of the piece of data,

fulfill the write request by writing the piece of data to the first storage medium of the first data node, and

after the write request has been fulfilled, for each redundant copy of the piece of data, write the respective redundant piece of data to a second storage medium of a respective other data node.

3. The system of claim 1 wherein the active piece of data is associated with an expiration time, and,

wherein when the expiration time has passed, the fast copy of the active piece of data is moved from the first storage medium of the first data node to the second storage medium of the first data node.

4. The system of claim 1 wherein the system is configured to perform background load balancing by, when a data node is not active, move a fast copy of the active piece of data from a first data node to a first storage medium of a second data node.

5. The system of claim 1 comprising a job tracker computing device;

and

wherein the job tracker computing device is configured to;

receive a computing job request, wherein the computing job is associated with an associated piece of data,

determine if the associated piece of data is stored within a first storage medium of a data node, and

if so, assign the computing job to the data node that includes the first storage medium that includes the associated piece of data.

6. The system of claim 1 wherein the plurality of data nodes is grouped into a plurality of data groups, each data group comprising at least one data node;

wherein the system comprises a computing device that is local to a data group; and

wherein the computing device is configured to;

receive a computing job request, wherein the computing job is associated with an associated piece of data,

determine if the associated piece of data is stored within a first storage medium of the data group,

if so, assign the computing job to a data node that includes the first storage medium,

if not, determine if the associated piece of data is stored within a second storage medium of the data group,

if so, assign the computing job to a data node that includes the second storage medium, and

if not, assign the computing job to a data node included by another data group.

7. The system of claim 1 wherein the system comprises a name node configured to:

maintain a directory tree of pieces of data stored by the plurality of data nodes, wherein the directory tree includes, for each piece of data, an indication as to which data node, if any, stores the respective piece of data in the data node'"'"'s first storage medium; and

receive a notification when a copy of a piece of data is moved from a first storage medium of a data node to a second storage medium of a data node.

8. The system of claim 1 wherein the system comprises a name node configured to:

maintain a directory tree of pieces of data stored by the plurality of data nodes, wherein the directory tree includes, for each piece of data, an indication as to which data node, if any, stores a fast copy of a respective piece of data,

receive a data access request from a computing device to access a requested piece of data, and

transmit a message to the computing device indicating that piece of data is to be accessed via the data node that stores the fast copy of the respective piece of data; and

wherein the system is configured to;

service, by the data node, the data access, and

for each slow copy of the piece of data stored within the system, replicate the data access to each of the slow copies of the piece of dat.

9. The system of claim 1 wherein the first storage medium comprises a non-mechanical storage medium, and

wherein the second storage medium comprises an electro-mechanical storage medium.

10. A method of storing data in a plurality of data nodes, wherein each data node comprising either or both of a first storage medium and a second storage medium that is slower than the first storage medium, and each data node configured to store a piece data in either the first storage medium or the second storage;

the method comprising;

storing a fast copy of an active piece of data within a first storage medium of a first data node; and

storing one or more slow copies of the active piece of data within respective second storage mediums of one or more respective other data nodes.

11. The method of claim 10, the method further comprising:

receiving a write request to write the piece of data to the plurality of data nodes, in response to the write request, selecting the first data node to store the fast copy of the piece of data,

fulfilling the write request by writing the piece of data to the first storage medium of the first data node, and

after the write request has been fulfilled, for each redundant copy of the piece of data, writing the respective redundant piece of data to a second storage medium of a respective other data node.

12. The method of claim 10, wherein the active piece of data is associated with an expiration time;

and

wherein the method further comprises, when the expiration time has passed, moving the fast copy of the active piece of data from the first storage medium of the first data node to the second storage medium of the first data node.

13. The method of claim 10, the method further comprising performing background load balancing between the plurality of data nodes.

14. The method of claim 10, further comprising”

receiving a computing job request, wherein the computing job is associated with the piece of data,

assigning the computing job to the data node that includes the fast copy of the piece of data.

15. The method of claim 10, wherein the plurality of data nodes is grouped into a plurality of data groups, each data group comprising at least one data node;

and

the method further comprising;

receiving a computing job request, wherein the computing job is associated with an associated piece of data,

determining if the associated piece of data is stored within a first storage medium of the data group,

if so, assigning the computing job to a data node that includes the first storage medium,

if not, determining if the associated piece of data is stored within a second storage medium of the data group,

if so, assigning the computing job to a data node that includes the second storage medium, and

if not, assigning the computing job to a data node included by another data group.

16. The method of claim 10, the method further comprising:

maintaining a directory tree of pieces of data stored by the plurality of data nodes, wherein the directory tree includes, for each piece of data, an indication as to which data node, if any, stores the respective piece of data in the data node'"'"'s first storage medium.

17. A method comprising:

arranging a heterogeneous distributed file system into multiple tiers of data storage, wherein the multiple tiers include a fast tier comprising a first type of storage medium, and a slower tier comprising a second type of storage medium;

preferably storing active data within the fast tier; and

preferably storing inactive data within the slow tier.

18. The method of claim 17, further comprising:

evicting data from the fast tier into the slow tier based upon a pseudo-first-in-first-out scheme.

19. The method of claim 17, further comprising:

maintaining redundant copies of a piece of data by,

first writing a fast copy of the piece of data to the fast tier, and

after the fast copy has been written, writing one or more slow copies of the piece of data to the second tier.

20. The method of claim 17, wherein the heterogeneous distributed file system is configured to store data on a plurality of data nodes;

and

the method further comprising;

receiving a request to schedule a computing job, wherein the computing job is associated with a piece of data, and

preferably scheduling the computing job to a data node that stores the piece of data within the fast tier.