Distributed data storage system, data distribution method, and apparatus and program to be used for the same

• US 8,239,621 B2
• Filed: 02/19/2008
• Issued: 08/07/2012
• Est. Priority Date: 02/20/2007
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Yamato, Junichi
• Primary Examiner(s): Peugh, Brian
• Assistant Examiner(s): SIMONETTI, NICHOLAS J
• Application Number:
  US12/033,426
• Publication Number:
  US 20080201336A1
• Time in Patent Office: 1,631 Days
• Field of Search: 707736-740, 711/114, 711/127, 711/153, 711/157, 711170-173
• US Class Current: 711/114
• CPC Class Codes: G06F 16/10 File systems; File servers

Abstract

There is provided a distributed data storage system for splitting content-data into a plurality of split data and storing a plurality of copy data corresponding to at least one of the plurality of split data in a plurality of memory devices. The system comprises a copying management information storing unit and a access location determining unit. The copying management information storing unit stores copying management information indicating a storage location of the copy data corresponding to the split data. The access location determining unit determines at least one of the plurality of memory devices storing the copy data corresponding to the split data as an access location for accessing split data.

Claims

1. A distributed data storage system for splitting content-data into a plurality of split data and storing a plurality of copy data corresponding to at least one of the plurality of split data in a plurality of memory devices, the system comprising:

a copy number planner configured to determine a number of copies for each of the split data based on a system situation including at least one of a demand for the content-data, a predicted demand for the content-data, and an available number of the split data;

a copying management information storing unit for storing copying management information indicating a storage location of the copy data corresponding to the split data;

an access location determining unit for determining at least one of the plurality of memory devices storing the copy data corresponding to the split data as an access location for accessing split data; and

a split data manager for dividing content-data into a plurality of split data and determining a plurality of memory devices as storage locations of a plurality of copy data corresponding to at least one of the plurality of split data,

wherein the access location is determined so that the copy data and the corresponding split data are stored in different memory devices, and so as to reduce correlation between the memory devices;

the copying management information storage unit stores copying management information indicating the storage locations of the plurality of copy data determined by the split data manager;

the split data manager determines assignment locations of copy data so as to uniformize the share numbers of combinations of memory devices, the share number being the number of copy data corresponding to same split data shared by two memory devices;

when determining an assignment location of copy data while the assignment location of copy data corresponding to the same split data is already determined, the split data manager determines the memory device showing the smallest share number when combined with the memory device already determined as assignment location of the copy data out of the memory devices except the memory device already determined as assignment location; and

the split data manager determines the assignment locations of copy data using a table that shows a two-dimensional arrangement n×

n memory devices.

2. The distributed data storage system according to claim 1, wherein:

the access location determining unit determines, for a host terminal reading at least part of content-data, at least one of the plurality of memory devices storing the copy data corresponding to the split data forming the at least part of content-data as an access location for accessing split data, and notifies it of the host terminal.

3. The distributed data storage system according to claim 1, wherein:

the access location determining unit determines, at a host terminal reading at least part of content-data, at least one of the plurality of memory devices storing the copy data corresponding to the split data forming the at least part of content-data as an access location for accessing split data.

4. The distributed data storage system according to claim 1, wherein:

the access location determining unit randomly determines at least one of the plurality of memory devices storing the plurality of copy data corresponding to the split data based on a random number.

5. The distributed data storage system according to claim 1, wherein:

the access location determining unit determines a memory device having the lowest load in a plurality of memory devices storing a plurality of copy data corresponding to at least one of the plurality of split data as an access location.

6. The distributed data storage system according to claim 1, wherein:

the split data manager determines storage locations of copy data so as to make memory devices uniformly store the plurality of copy data corresponding to the split data.

7. The distributed data storage system according to claim 1, wherein:

the split data manager determines assignment locations of copy data so as to make memory devices store a plurality of copy data corresponding to a predetermined number of split data that are continuous in content-data in a distributed manner.

8. The distributed data storage system according to claim 1, wherein:

when a memory device is disabled, the split data manager deletes information indicating a disabled memory device as storage location from the copying management information storage unit.

9. The distributed data storage system according to claim 1, further comprising:

a copying processor for executing a copying process of copying data from a memory device to a memory device and a deletion process of deleting copy data from a memory device according to an instruction from the split data manager, wherein;

the split data manager determines a new assignment location for the copy data assigned to a disabled memory device as assignment location, determines a memory device as copy originating memory device out of the memory devices storing copy data corresponding to the split data same as the former copy data, and causes the copying processor to execute the copying process of copying the copy data from the originating memory device.

10. The distributed data storage system according to claim 1, further comprising:

a copying processor for executing a copying process of copying data from a memory device to a memory device and a deletion process of deleting copy data from a memory device according to an instruction from the split data manager, wherein;

the split data manager determines a new assignment location for copy data at a predetermined timing according to the copying management information and causes the copying processor to execute the copying process of copying data to the new assignment location and the deletion process of deleting copy data from the former assignment location.

11. The distributed data storage system according to claim 1, further comprising:

a copy number planner for determining the number of copies that is the number of copy data to be stored in memory devices for split data forming content-data, wherein;

the split data manager determines the assignment locations of copy data corresponding to the split data according to the number of copies of the split data as determined by the copy number planner.

12. The distributed data storage system according to claim 11, wherein:

the copy number planner determines the number of copies of each split data according to the position of the split data in content-data.

13. The distributed data storage system according to claim 12, wherein:

when the split data including data at the leading position of content-data is given and content-data are provided with chapters, the copy number planner makes the number of copies of split data including data at the leading position of each chapter larger than the number of copies of any other split data of content-data.

14. The distributed data storage system according to claim 11, wherein:

the copy number planner determines the number of copies of split data according to the access requests of content-data.

15. The distributed data storage system according to claim 14, wherein:

the copy number planner determines the number of copies of split data according to;

(a) the predicted access requests or the access requests of content-data, (b) the processing capacity of each memory device and (c) the safety coefficient defined by the system.

16. The distributed data storage system according to claim 11, further comprising:

a copying processor for executing a copying process of copying data from a memory device to a memory device and a deletion process of deleting copy data from a memory device according to an instruction from the split data manager, wherein;

the split data manager determines a new assignment location of copy data corresponding to split data or an assignment location for deleting copy data corresponding to the split data so as to make the number of copies of the split data as determined by the copy number planner and the number of copy data corresponding to the split data and stored in memory devices agree with each other and causes the copying processor to execute the copying process of copying the copy data or deletion process of deleting the copy data.

17. A split data management apparatus used for a distributed data storage system for splitting content-data into a plurality of split data and storing a plurality of copy data corresponding to at least one of the plurality of split data in a plurality of memory devices, the apparatus being adapted to notify a host terminal reading at least part of content-data of the access location for accessing split data forming content-data, the apparatus comprising:

a copy number planner configured to determine a number of copies for each of the split data based on a system situation including at least one of a demand for the content-data, a predicted demand for the content-data, and an available number of the split data;

an access location determining unit for determining at least one memory device as an access location for accessing the split data out of the memory devices storing copy data corresponding to the split data according to copying management information indicating the assignment locations of copy data corresponding to the split data included in content-data read range of the host terminal; and

a split data manager for dividing content-data into a plurality of split data and determining a plurality of memory devices as storage locations of a plurality of copy data corresponding to at least one of the plurality of split data,

wherein the access location is determined so that the copy data and the corresponding split data are stored in different memory devices, and so as to reduce correlation between the memory devices;

the copying management information storage unit stores copying management information indicating the storage locations of the plurality of copy data determined by the split data manager;

the split data manager determines assignment locations of copy data so as to uniformize the share numbers of combinations of memory devices, the share number being the number of copy data corresponding to same split data shared by two memory devices;

when determining an assignment location of copy data while the assignment location of copy data corresponding to the same split data is already determined, the split data manager determines the memory device showing the smallest share number when combined with the memory device already determined as assignment location of the copy data out of the memory devices except the memory device already determined as assignment location; and

the split data manager determines the assignment locations of copy data using a table that shows a two-dimensional arrangement n×

n memory devices.

18. A host terminal used for a distributed data storage system for splitting content-data into a plurality of split data and storing a plurality of copy data corresponding to at least one of the plurality of split data in a plurality of memory devices, the host terminal being adapted to access the plurality of memory devices storing split data forming content-data in order to read at least part of content-data, the host terminal comprising:

a copy number planner configured to determine a number of copies for each of the split data based on a system situation including at least one of a demand for the content-data, a predicted demand for the content-data, and an available number of the split data;

an access location determining unit for determining at least one of the plurality of memory devices as an access location for accessing the split data out of the memory devices storing copy data corresponding to the split data according to copying management information indicating the assignment locations of copy data corresponding to the split data included in content-data read range thereof; and

a split data manager for dividing content-data into a plurality of split data and determining a plurality of memory devices as storage locations of a plurality of copy data corresponding to at least one of the plurality of split data,

wherein the access location is determined so that the copy data and the corresponding split data are stored in different memory devices, and so as to reduce correlation between the memory devices;

the copying management information storage unit stores copying management information indicating the storage locations of the plurality of copy data determined by the split data manager;

the split data manager determines assignment locations of copy data so as to uniformize the share numbers of combinations of memory devices, the share number being the number of copy data corresponding to same split data shared by two memory devices;

when determining an assignment location of copy data while the assignment location of copy data corresponding to the same split data is already determined, the split data manager determines the memory device showing the smallest share number when combined with the memory device already determined as assignment location of the copy data out of the memory devices except the memory device already determined as assignment location; and

the split data manager determines the assignment locations of copy data using a table that shows a two-dimensional arrangement n×

n memory devices.

19. A data distribution method for a distributed data storage system for splitting content-data into a plurality of split data and storing a plurality of copy data corresponding to at least one of the plurality of split data in a plurality of memory devices, the method comprising:

a step in which a copy number planner determines a number of copies for each of the split data based on a system situation including at least one of a demand for the content-data, a predicted demand for the content-data, and an available number of the split data;

a step in which a copying management information storing unit stores copying management information indicating the storage locations of copy data corresponding to split data;

a step in which an access location determining unit determines at least one of the plurality of memory devices storing the copy data corresponding to split data as a location for accessing the split data according to the copying management information;

a step in which a split data manager divides content-data into a plurality of split data and determines a plurality of memory devices as storage locations of a plurality of copy data corresponding to at least one of the plurality of split data; and

a step in which the copying management information storage unit stores copying management information indicating the storage locations of the plurality of copy data determined by the split data manager;

a step in which the split data manager determines assignment locations of copy data so as to uniformize the share numbers of combinations of memory devices, the share number being the number of copy data corresponding to same split data shared by two memory devices;

a step in which, when determining an assignment location of copy data while the assignment location of copy data corresponding to the same split data is already determined, the split data manager determines the memory device showing the smallest share number when combined with the memory device already determined as assignment location of the copy data out of the memory devices except the memory device already determined as assignment location,

wherein the access location is determined so that the copy data and the corresponding split data are stored in different memory devices, and so as to reduce correlation between the memory devices; and

a step in which the split data manager determines the assignment locations of copy data using a table that shows a two-dimensional arrangement n×

n memory devices.

20. The data distribution method according to claim 19, further comprising:

a step of in which the access location determining unit determines, for a host terminal reading at least part of content-data, at least one of the plurality of memory devices storing the copy data corresponding to the split data forming the at least part of content-data as an access location for accessing split data, and notifies it of the host terminal.

21. The data distribution method according to claim 19, further comprising:

a step in which the access location determining unit determines, at a host terminal reading at least part of content-data, at least one of the plurality of memory devices storing the copy data corresponding to the split data forming the at least part of content-data as an access location for accessing split data.

22. The data distribution method according to claim 19, further comprising:

a step in which the access location determining unit randomly determines at least one of the plurality of memory devices storing the plurality of copy data corresponding to the split data based on a random number.

23. The data distribution method according to claim 19, further comprising:

a step in which the access location determining unit determines a memory device having the lowest load in a plurality of memory devices storing a plurality of copy data corresponding to at least one of the plurality of split data as an access location.

24. The data distribution method according to claim 19, further comprising:

a step in which the split data manager determines storage locations of copy data so as to make memory devices uniformly store the plurality of copy data corresponding to the split data.

25. The data distribution method according to claim 19, further comprising:

a step in which the split data manager determines assignment locations of copy data so as to make memory devices store a plurality of copy data corresponding to a predetermined number of split data that are continuous in content-data in a distributed manner.

26. The data distribution method according to claim 19, further comprising:

a step in which, when a memory device is disabled, the split data manager deletes information indicating a disabled memory device as storage location from the copying management information storage unit.

27. The data distribution method according to claim 19, further comprising:

a step in which a copying processor executes a copying process of copying data from a memory device to a memory device and a deletion process of deleting copy data from a memory device according to an instruction from the split data manager; and

a step in which the split data manager determines a new assignment location for the copy data assigned to a disabled memory device as assignment location, determines a memory device as copy originating memory device out of the memory devices storing copy data corresponding to the split data same as the former copy data, and causes the copying processor to execute the copying process of copying the copy data from the originating memory device.

28. The data distribution method according to claim 19, further comprising:

a step in which a copying processor executes a copying process of copying data from a memory device to a memory device and a deletion process of deleting copy data from a memory device according to an instruction from the split data manager; and

a step in which the split data manager determines a new assignment location for copy data at a predetermined timing according to the copying management information and causes the copying processor to execute the copying process of copying data to the new assignment location and the deletion process of deleting copy data from the former assignment location.

29. The data distribution method according to claim 19, further comprising:

a step in which a copy number planner determines the number of copies that is the number of copy data to be stored in memory devices for split data forming content-data; and

a step in which the split data manager determines the assignment locations of copy data corresponding to the split data according to the number of copies of the split data as determined by the copy number planner.

30. The data distribution method according to claim 29, further comprising:

a step in which the copy number planner determines the number of copies of each split data according to the position of the split data in content-data.

31. The data distribution method according to claim 30, further comprising:

a step in which, when the split data including data at the leading position of content-data is given and content-data are provided with chapters, the copy number planner makes the number of copies of split data including data at the leading position of each chapter larger than the number of copies of any other split data of content-data.

32. The data distribution method according to claim 29, further comprising:

a step in which the copy number planner determines the number of copies of split data according to the access requests of content-data.

33. The data distribution method according to claim 32, further comprising:

a step in which the copy number planner determines the number of copies of split data according to;

(a) the predicted access requests or the access requests of content-data, (b) the processing capacity of each memory device and (c) the safety coefficient defined by the system.

34. The data distribution method according to claim 29, further comprising:

a step in which a copying processor executes a copying process of copying data from a memory device to a memory device and a deletion process of deleting copy data from a memory device according to an instruction from the split data manager; and

a step in which the split data manager determines a new assignment location of copy data corresponding to split data or an assignment location for deleting copy data corresponding to the split data so as to make the number of copies of the split data as determined by the copy number planner and the number of copy data corresponding to the split data and stored in memory devices agree with each other and causes the copying processor to execute the copying process of copying the copy data or deletion process of deleting the copy data.

35. A computer program, encoded in a non-transitory computer-readable medium, for a split data management apparatus in a distributed data storage system for splitting content-data into a plurality of split data and storing a plurality of copy data corresponding to at least split data in a plurality of memory devices, the apparatus notifying an access location for accessing split data of content-data to a host terminal reading at least part of content-data, the program causing a computer to execute a process of:

determining a number of copies for each of the split data based on a system situation including at least one of a demand for the content-data, a predicted demand for the content-data, and an available number of the split data;

determining at least one of the plurality of memory devices storing the copy data corresponding to the split data as an access location for accessing split data according to copying management information indicating the location of the copy data corresponding to the split data included in content-data reading range of the host terminal;

dividing content-data into a plurality of split data and determining a plurality of memory devices as storage locations of a plurality of copy data corresponding to at least one of the plurality of split data;

storing copying management information indicating the storage locations of the plurality of copy data;

determining assignment locations of copy data so as to uniformize the share numbers of combinations of memory devices, the share number being the number of copy data corresponding to same split data shared by two memory devices;

when determining an assignment location of copy data while the assignment location of copy data corresponding to the same split data is already determined, determining the memory device showing the smallest share number when combined with the memory device already determined as assignment location of the copy data out of the memory devices except the memory device already determined as assignment location,

wherein the access location is determined so that the copy data and the corresponding split data are stored in different memory devices, and so as to reduce correlation between the memory devices; and

determining the assignment locations of copy data using a table that shows a two-dimensional arrangement n×

n memory devices.

36. A computer program, encoded in a non-transitory computer-readable medium, for a host terminal in a distributed data storage system for splitting content-data into a plurality of split data and storing a plurality of copy data corresponding to at least one of the plurality of split data in a plurality of memory devices, the host terminal being configured to access the plurality of memory devices storing split data forming content-data in order to read at least part of content-data, the program causing a computer to execute a process of:

determining a number of copies for each of the split data based on a system situation including at least one of a demand for the content-data, a predicted demand for the content-data, and an available number of the split data;

determining at least one of the plurality of memory devices as an access location for accessing the split data out of the memory devices storing copy data corresponding to the split data according to copying management information indicating an assignment location of copy data corresponding to the split data included in content-data read range thereof and

dividing content-data into a plurality of split data and determining a plurality of memory devices as storage locations of a plurality of copy data corresponding to at least one of the plurality of split data;

storing copying management information indicating the storage locations of the plurality of copy data;

determining assignment locations of copy data so as to uniformize the share numbers of combinations of memory devices, the share number being the number of copy data corresponding to same split data shared by two memory devices;

when determining an assignment location of copy data while the assignment location of copy data corresponding to the same split data is already determined, determining the memory device showing the smallest share number when combined with the memory device already determined as assignment location of the copy data out of the memory devices except the memory device already determined as assignment location,

wherein the access location is determined so that the copy data and the corresponding split data are stored in different memory devices, and so as to reduce correlation between the memory devices; and

determining the assignment locations of copy data using a table that shows a two-dimensional arrangement n×

n memory devices.