System for load balancing by replicating portion of file while being read by first stream onto second device and reading portion with stream capable of accessing

• US 5,802,301 A
• Filed: 05/11/1994
• Issued: 09/01/1998
• Est. Priority Date: 05/11/1994
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dan, Asit, Yu, Philip Shi-lung, Sitaram, Dinkar, Kienzle, Martin Gerhard
• Primary Examiner(s): Lee, Thomas C.
• Assistant Examiner(s): Chen, Anderson I.
• Application Number:
  US08/241,086
• Time in Patent Office: 1,574 Days
• Field of Search: 395/182.04, 395/441, 395/485, 395/488, 395/600, 364/222.81, 364/238.3, 364/245.2, 364/246.3, 364/248.1, 364/283.1, 364/260, 364/281
• US Class Current: 709/223
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04N 21/2182 involving memory arrays, e.g. RAID disk arrays RAID arrays per se G06F3/0689; use of parity to protect data in RAID systems G06F11/1008

Abstract

A method and system for improving load balancing in a file server is disclosed. The method includes determining the existence of an overload condition on a storage device, the storage device having a plurality of retrieval streams accessing at least one file thereon; selecting a first retrieval stream reading a file; replicating a portion of said file being read by said first retrieval stream onto a second storage device; and reading the replicated portion of said file on said second storage device with a retrieval stream capable of accessing the replicated portion of said file. The method and system enables the dynamic replication of data objects to respond to fluctuating user demand. The method and system is particularly useful in file servers such as multimedia servers delivering continuously in real time large multimedia files such as movies.

Claims

1. A method for improving load balancing in a file server comprising:

determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

selecting a first retrieval stream reading a file;

replicating a portion of said file while being read by said first retrieval stream onto a second storage device; and

reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file.

2. The method of claim 1 wherein determining the existence of an overload condition on the first storage device includes detecting a threshold level on said first storage device.

3. The method of claim 1 wherein said first retrieval stream is selectively chosen from a plurality of retrieval streams reading said file.

4. The method of claim 1 wherein said second storage device is chosen from a plurality of available storage devices, said second storage device having the largest available amount of storage space.

5. The method of claim 1 wherein replicating a portion of said file being read by said first retrieval stream onto said second storage device includes initially creating a switching point entry on a file information table so that said portion of said file can be indexed for ease of switching between said second storage device and another storage device.

6. The method of claim 1 wherein said file comprises a plurality of blocks.

7. The method of claim 6 wherein said file comprises a plurality of segments, each of said plurality of segments comprising a plurality of blocks.

8. The method of claim 1 wherein said file comprises a plurality of segments and said portion of the file being replicated comprises one or more segments.

9. The method of claim 1 where in said file includes at least one primary segment.

10. The method of claim 9 wherein said file includes a plurality of primary segments.

11. The method of claim 10 wherein said primary segments are permanent segments.

12. The method of claim 1 wherein said file server comprises a multimedia file server.

13. The method of claim 1 wherein said storage device comprises a disk.

14. The method of claim 1 wherein said file server comprises a video server, and wherein said file comprises a video file.

15. A method for improving load balancing in a file server comprising:

determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

selecting a first retrieval stream reading a file;

replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said replicating a portion of said file being read by said first retrieval stream onto a second storage device comprises;

reading said file using said first retrieval stream from said first storage device; and

writing said portion of said file using a retrieval stream capable of accessing said replicated portion of said file onto said second storage device.

19. The method of claim 15 further comprising converting one of said at least one primary segment to a secondary segment.

16. A method for improving load balancing in a file server comprising:

determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

selecting a first retrieval stream reading a file;

replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said file includes at least one permanent, primary segment; and

wherein said replicated portion of said file comprises at least one secondary segment.

17. The method of claim 16 wherein said secondary segment is a temporary segment.

18. The method of claim 17 wherein said secondary segment may be deleted .

20. The method of claim 18 further comprising converting one of said at least one secondary segment to a primary segment.

21. A method for improving load balancing in a file server comprising:

determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

selecting a first retrieval stream reading a file;

replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said file comprises a plurality of blocks; and

wherein replicating said portion of said file being read by said first retrieval stream onto said second storage device comprises;

sequentially reading said plurality of blocks from said first retrieval stream; and

writing one or more of said plurality of blocks onto said second storage device.

22. A method for improving load balancing in a file server comprising:

determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

selecting a first retrieval stream reading a file;

replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said file comprises a plurality of segments, each of said plurality of segments comprising a plurality of blocks; and

wherein replicating said portion of said file being read by said first retrieval stream onto said second storage device comprises;

sequentially reading at least one of said segments from said first retrieval stream; and

writing at least one of said segments on to said second storage device.

23. A method for improved load balancing in a file server comprising:

determining an excess number of permanently stored primary copies of a segment of a file;

converting one of said permanently stored primary copies of said segment, not being accessed, to a temporary secondary copy of said segment, wherein said secondary copy of said segment is capable of being overwritten; and

overwriting said secondary copy of said segment for improving the load balancing in said file server.

24. A method for improved load balancing in a file server comprising:

determining an excess number of permanently stored primary segments of a file;

estimating a required number of primary segments based on load history;

converting one of said permanently stored primary segments, not being accessed, to a temporary secondary segment wherein said secondary segment is capable of being overwritten; and

overwriting said secondary for improving the load balancing in said file server.

25. The method claim 24 further comprising converting all permanently stored primary segments greater than said required number of primary segments.

26. A method for improved load balancing in a file server comprising:

determining a load on a temporarily stored secondary copy of a segment of a file;

determining if said load on said secondary copy of said segment exceeds a threshold value, and

converting said temporarily stored secondary copy of said segment to a permanently stored primary copy of said segment of said file, wherein said primary copy of said segment is prevented from being overwritten.

27. A method for improved load balancing in a file server comprising:

determining a load on a temporarily stored secondary segment of a file including determining a variable load history for said secondary segment;

determining if said load on said secondary segment exceeds a threshold value; and

converting said temporarily stored secondary segment to a permanently stored primary segment of said file, wherein said primary segment is prevented from being overwritten.

28. The method of claim 27 further comprising analyzing said variable load history for determining whether said threshold value is exceeded.

29. A system for improving load balancing in a file server comprising:

means for determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

means for selecting a first retrieval stream reading a file;

means for replicating a portion of said file while being read by said first retrieval stream onto a second storage device; and

means for reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file.

30. The system of claim 29 wherein said determining means determines the existence of an overload condition on the first storage device by detecting a threshold level on said storage device.

31. The system of claim 29 wherein said first retrieval stream is selectively chosen from a plurality of retrieval streams reading said file.

32. The system of claim 29 wherein said second storage device is chosen from a plurality of available storage devices, said second storage device having the largest available amount of storage space.

33. The system of claim 29 wherein said means for replicating a portion of said file being read by said first retrieval stream onto said second storage device includes means for initially creating a switching point entry on a file information table so that said portion of said file can be indexed for ease of switching between said second storage device and another storage device.

34. The system of claim 29 wherein said file comprises a plurality of blocks.

35. The system of claim 34 wherein each of said file comprises a plurality of segments, each of said plurality of segments comprising a plurality of blocks.

36. The system of claim 29 wherein said file comprises a plurality of segments and said portion of the file being replicated comprises one or more segments.

37. The system of claim 29 wherein said file includes at least one primary segment.

38. The system of claim 37 wherein said file includes a plurality of primary segments.

39. The system of claim 38 wherein said primary segments are permanent segments.

40. The system of claim 29 wherein said file server comprises a multimedia file server.

41. The system of claim 29 wherein said storage device comprises a disk.

42. The system of claim 29 wherein said file server comprises a video server, and wherein said file comprises a video file.

43. A system for improving load balancing in a file server comprising:

means for determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

means for selecting a first retrieval stream reading a file;

means for replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

means for reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said means for replicating a portion of said file being read by said first retrieval stream onto a second storage device comprises;

means for reading said file using said first retrieval stream from said first storage device; and

means for writing said portion of said file using a retrieval stream capable of accessing said replicated portion of said file onto said second storage device.

44. A system for improving load balancing in a file server comprising:

means for determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

means for selecting a first retrieval stream reading a file;

means for replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

means for reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said file includes at least one permanent, primary segment; and

wherein said replicated portion of said file comprises at least one secondary segment.

45. The system of claim 44 wherein said secondary segment is a temporary segment.

46. The system of claim 45 wherein said secondary segment may be deleted.

47. The system of claim 46 further comprising means for converting one of said at least one primary segment to a secondary segment.

48. The system of claim 46 further comprising means for converting one of said at least one secondary segment to a primary segment.

49. A system for improving load balancing in a file server comprising:

means for determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

means for selecting a first retrieval stream reading a file;

means for replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

means for reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said file comprises a plurality of blocks; and

wherein said means for replicating said portion of said file being read by said first retrieval stream onto said second storage device comprises;

means for sequentially reading said plurality of blocks from said first retrieval stream; and

means for writing one or more of said plurality of blocks onto said second storage device.

50. A system for improving load balancing in a file server comprising:

means for determining the existence of an overload condition on a first storage device, said first storage device having a plurality of retrieval streams accessing at least one file thereon;

means for selecting a first retrieval stream reading a file;

means for replicating a portion of said file being read by said first retrieval stream onto a second storage device; and

means for reading said replicated portion of said file on said second storage device with a retrieval stream capable of accessing said replicated portion of said file;

wherein said file comprises a plurality of segments, each of said plurality of segments comprising a plurality of blocks; and

wherein said means for replicating said portion of said file being read by said first retrieval stream onto said second storage device comprises;

means for sequentially reading at least one of said segments from said first retrieval stream; and

means for writing at least one of said segments on to said second storage device.

51. A system for improved load balancing in a file server comprising:

means for determining an excess number of permanently stored primary copies of a segment of a file;

means for converting one of said permanently stored primary copies of said segment, not being accessed, to a temporary secondary copy of said segment, wherein said secondary copy of said segment is capable of being overwritten; and

means for overwriting said secondary copy of said segment for improving the load balancing in said file server.

52. A system for improved load balancing in a file serve comprising:

means for determining an excess number of permanently store primary segment of a file;

means for estimating a required number of primary segments based on load history;

means for converting one of said permanently stored primary segments, not being accessed, to a temporary secondary segment, wherein said secondary segment is capable of being overwritten; and

means for overwriting said secondary segment for improving the load balancing in said server.

53. The system claim 52 further comprising means for converting all permanently stored primary segments greater than said required number of primary segments.

54. A system for improved load balancing in a file server comprising:

means for determining a load on a temporarily stored secondary copy of a segment of a file;

means for determining if said load on said secondary copy of said segment exceeds a threshold value; and

means for converting said temporarily stored secondary copy of said segment to a permanently stored primary copy of said segment of said file, wherein said primary copy of said segment is prevented from being overwritten.

55. A system for improved load balancing in a file server comprising:

means for determining a load on a temporarily stored secondary segment of a file including means for determining a variable load history for said secondary segment;

means for determining if said load on said secondary segment exceeds a threshold value; and

means for converting said temporarily stored secondary segment to a permanently stored primary segment of said file, wherein said primary segment is prevented from being overwritten.

56. The method of claim 55 further comprising means for analyzing said variable load history for determining if said threshold value is exceeded.