Distributed network data storage system and method

• US 7,574,527 B1
• Filed: 04/23/2007
• Issued: 08/11/2009
• Est. Priority Date: 04/09/2001
• Status: Expired due to Fees

• Current Assignee: Virtuozzo International GmbH
• Original Assignee: Swsoft Holdings, Ltd.
• Inventors: Pudgorodsky, Yuri, Tormasov, Alexander, Beloussov, Serguei, Protassov, Stanislav
• Primary Examiner(s): Follansbee; John
• Assistant Examiner(s): Tran; Nghi V
• Application Number:
  US11/738,732
• Time in Patent Office: 841 Days
• Field of Search: 709/223, 709231-232, 709/243
• US Class Current: 709/243
• CPC Class Codes: G06F 16/1837 Management specially adapted to peer-to-peer storage networks topology management mechanisms of peer-to-peer networks H04L67/1042
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/10015 Access to distributed or replicated servers, e.g. using brokers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1021 based on client or server locations
• CPC Class Codes: H04L 67/1029 using data related to the state of servers by a load balancer
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A system for distributed file storage includes a plurality of servers providing, to a plurality of clients, file access services for accessing files stored on the plurality of servers. A list of neighbor servers is maintained by each server. The neighbor servers are a subset of the plurality of servers. At least one server of the plurality of servers is switched into a neighbor group of servers based on network distance. Each file is stored in the form of a plurality of N pieces on N servers, the pieces being generated from the file. The list is used to obtain information for reconstructing files stored on the neighbor servers, such that any K out of the N pieces can be used to reconstruct any file.

Claims

1. A system for distributed file storage comprising:

a plurality of host servers; and

a list of neighbor servers maintained by each server, wherein the neighbor servers are a subset of the plurality of hosting servers, and wherein at least one server of the plurality of hosting servers is switched into a neighbor group of servers based on network distance,

wherein each file is stored in the form of a plurality of N pieces on N servers, the pieces being generated form the file, and

wherein the list is used to obtain information for reconstructing files stored on the neighbor servers, such that any K out of the N pieces can be used to reconstruct any file, such that K<

N, and the reconstruction of the file is guaranteed with at least K pieces and is impossible with fewer than K pieces;

wherein a server belonging to more than one group acts as a boundary server,

wherein boundary servers are used to transfer pieces of the file to servers of groups other than a group to which a client has connected and

wherein, when a new server is connected to one of the groups, the new server uses information of the boundary servers to connect to a group so as to have optimal server-to-server response time to its neighbors within its group.

2. The system of claim 1, wherein the files are stored using a peer-to-peer network.

3. The system of claim 1, wherein the servers are functionally equivalent.

4. The system of claim 1, wherein any number of pieces less than K cannot be used to obtain any meaningful information about contents of the file.

5. The system of claim 1, wherein at least one of the servers is a logical server that includes a plurality of physical servers connected via a network.

6. The system of claim 1, wherein each server belongs to a group defined by its corresponding list,

wherein a server that belongs to multiple groups maintains corresponding multiple lists for each group to which it belongs, and

wherein, upon receiving a request for a file, the server distributes the request to each of the multiple groups.

7. The system of claim 1, wherein none of the pieces is unique.

8. The system of claim 1, wherein the file pieces into which a particular file is transformed are of equal size.

9. The system of claim 1, wherein the servers are organized into a plurality of groups such that server-to-server response time between any two servers of the same group does not exceed a predetermined limit.

10. The system of claim 1, wherein the list is a dynamic list.

11. A system for distributed file storage comprising:

on a plurality of hosting servers, providing, to a plurality of clients, file access services for accessing files stored on the plurality of hosting servers; and

on each server, maintaining a list of neighbor servers, wherein the neighbor servers are a subset of the plurality of servers;

switching at least one server of the plurality of hosting servers into a neighbor group of servers based on network distance,

storing each file in the form of a plurality of N pieces on N servers, the pieces being generated form the file, and

wherein the list is used to obtain information for reconstructing files stored on the neighbor servers, such that any K out of the N pieces can be used to reconstruct any file, such that K<

N, and the reconstruction of the file is guaranteed with at least K pieces and is impossible with fewer than K pieces;

wherein a server belonging to more than one group acts as a boundary server,

wherein boundary servers are used to transfer pieces of the file to servers of groups other than a group to which a client has connected and

wherein, when a new server is connected to one of the groups, the new server uses information of the boundary servers to connect to a group so as to have optimal server-to-server response time to its neighbors within its group.

12. The method of claim 11, wherein the files are stored using a peer-to-peer network.

13. The method of claim 11, wherein the servers are functionally equivalent.

14. The method of claim 11, wherein any number of pieces less than K cannot be used to obtain any meaningful information about contents of the file.

15. The method of claim 11, wherein at least one of the servers is a logical server that includes a plurality of physical servers connected via a network.

16. The method of claim 11, wherein each server belongs to a group defined by its corresponding list,

wherein a server that belongs to multiple groups maintains corresponding multiple lists for each group to which it belongs, and

wherein, upon receiving a request for a file, the server distributes the request to each of the multiple groups.

17. The method of claim 11, wherein none of the pieces is unique.

18. The method of claim 11, wherein the file pieces into which a particular file is transformed are of equal size.

19. The method of claim 11, wherein the servers are organized into a plurality of groups such that server-to-server response time between any two servers of the same group does not exceed a predetermined limit.

20. The method of claim 11, wherein the list is a dynamic list.

21. A system for distributed filed storage comprising:

a plurality of hosting servers providing, to a plurality of clients, file access services for accessing files stored on the plurality of hosting servers;

a file stored in the form of a plurality of N pieces on N of the servers, the pieces being generated from the file, such that K<

N; and

a dynamic list of neighbor servers maintained by each server, wherein the neighbor servers are a subset of the plurality of hosting servers, and wherein at least one server of the plurality of servers is switched into a neighbor group of servers based on minimizing network distance,

wherein the list is used to obtain information for reconstructing files stored on the neighbor servers, and the reconstruction of the file is guaranteed with at least K pieces and is impossible with fewer than K pieces;

wherein a server belonging to more than one group acts as a boundary server,

wherein boundary servers are used to transfer pieces of the file to servers of groups other than a group to which a client has connected and

wherein, when a new server is connected to one of the groups, the new server uses information of the boundary servers to connect to a group so as to have optimal server-to-server response time to its neighbors within its group.

22. A system for distributed file storage comprising:

a plurality of hosting servers providing, to a plurality of clients, file access services for accessing files stored on the plurality of hosting servers;

wherein the hosting servers are organized into a plurality of groups so as to minimize network distance between hosting servers of the same group;

wherein a server belonging to more than one group acts as a boundary server,

wherein, when a new server is connected to one of the groups, the new server uses information of the boundary servers to connect to a group so as to have optimal network distance to its neighbors within its group,

wherein, when one server is disconnected from the system, the remaining servers reconfigure their groups so as to have optimal server-to-server response times to their neighbors within their groups; and

a dynamic list of neighbor servers of each server'"'"'s group maintained by each server, wherein the neighbor servers are a subset of the plurality of servers,

wherein each file is stored on the servers in the form of a plurality of N pieces generated from the file, and the reconstruction of the file is guaranteed with at least K pieces and is impossible with fewer than K pieces, such that K<

N,

wherein any client can use any of the plurality of servers to access any of its files stored on the system, and

wherein boundary servers are used to transfer pieces of the file to servers of groups other than a group to which a client has connected.