Methods and apparatus for managing the storage of content in a file system

• US 7,805,470 B2
• Filed: 06/23/2005
• Issued: 09/28/2010
• Est. Priority Date: 06/23/2005
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Todd, Stephen, Armangau, Philippe
• Primary Examiner(s): To; Baoquoc N
• Application Number:
  US11/165,104
• Publication Number:
  US 20060294164A1
• Time in Patent Office: 1,923 Days
• Field of Search: 707/3, 707/10, 707/205
• US Class Current: 707/822
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31

Abstract

One embodiment of the invention is directed to the containerization of content units that are accessed using an identifier. Applicants have appreciated that file systems sometimes impose limits on the number of files that may be stored therein that are too restrictive. Thus, in one embodiment, multiple content units may be stored in a single file, called a container file, in a file system. Each content unit may have an identifier associated with it. When an accessing entity requests access to a previously-stored content unit and provides the identifier for the content unit, the identifier may be used to locate the container file in which the content unit is stored.

Claims

1. A method of managing content units in a file system installed on a computer system comprising at least one input, and at least one controller coupled to the at least one input and configured to store content units in the file system, the method comprising acts of:

receiving via the at least one input, a first request to store a first content unit, the first content unit having a first content address that is computed, at least in part, by hashing the content of the first content unit;

in response to the first request, storing the first content unit in a file in the file system;

receiving via the at least one input, a second request to store a second content unit, the second content unit having a second address that is computed, at least in part, by hashing the content of the second content unit;

in response to the second request, storing the second content unit in the same file in which the first content unit is stored and in a manner that enables the first and second content units to be separately accessible;

receiving a third request to store a third content unit, the third content unit having a same content as the first content unit;

determining that the first and third content units have the same content; and

in response to determining that the first and third content units have the same content, declining to store the third content unit.

2. The method of claim 1, wherein the file system includes a time-based directory structure having a plurality of directories, wherein each of the plurality of directories corresponds to a period of time, and wherein the file is stored in one of the plurality of directories that corresponds to a period of time during which the first and second request requests were received.

3. The method of claim 1, further comprising acts of:

receiving a fourth request to access the third content unit; and

in response to the fourth request, returning the first content unit.

4. The method of claim 1, wherein the file is a first file, and wherein the method further comprises an act of:

storing information in a second file in the file system indicating the location of the first content unit within the first file; and

storing information in the second file indicating the location of the second content unit within the first file.

5. The method of claim 1, wherein the file in which the first content unit and the second content unit are stored is not a folder or directory in the file system.

6. The method of claim 1, wherein the file system imposes a limit on a number of files that may be stored on the file system, and wherein storing the second content unit in the same file in which the first content unit is stored enables the number of content units stored in the file system to exceed the limit.

7. The method of claim 1, wherein the first request is a request to replicate the first content unit from a storage system that stores the first content unit.

8. The method of claim 2, further comprising an act of:

creating an entry for the first content unit in an index file, the entry identifying the file that stores the first content unit and wherein the index file is stored in one of the plurality of directories that corresponds to an index time for the first content unit.

9. The method of claim 8, wherein the first request is a request to replicate the first content unit from a storage system that stores the first content unit and wherein the index time is a time related to the time of storage of the first content unit on the storage system.

10. The method of claim 8, wherein the index time is a time related to the time of receipt of the first request.

11. The method of claim 8, wherein the index time is a time derived from the content address of the first content unit.

12. The method of claim 11, wherein the time derived from the content address of the first content unit is derived from a portion of the content address that is based on the content of the first content unit.

13. The method of claim 2, where the first request is received from a storage system that stores the first content unit, and wherein the first content address includes a timestamp that indicates a time related to the time of storage of the first content unit on the storage system.

14. The method of claim 13, wherein the file corresponds to a period of time that includes the times at which the first and second requests were received.

15. The method of claim 13, wherein the file corresponds to a period of time that is subsumed by the period time of the one of the plurality of directories in which the file is stored.

16. The method of claim 13, further comprising acts of:

in response to the first request, determining one the plurality of directories that corresponds to a period of time that includes the time indicated in the timestamp; and

storing, in the one the plurality of directories that corresponds to a period of time that includes the time indicated in the timestamp, an indication of the file that stores first the content unit.

17. The method of claim 16, further comprising an act of:

in response to the first request, associating an identifier with the first content unit.

18. The method of claim 17, where the identifier is the first content address.

19. The method of claim 17, further comprising an act of:

returning the identifier as acknowledgement of receipt of the first request.

20. The method of claim 17, further comprising acts of:

receiving a fourth request to access the first content unit, the fourth request identifying the first content unit using the identifier;

locating the indication of the file that stores first the content unit using the identifier;

locating the file that stores the first content unit based on the indication;

locating the first content unit in the file; and

returning the first content unit in response to the fourth request.

21. At least one computer readable medium encoded with instructions that, when executed on a computer system, perform a method of managing content units in a file system, the method comprising acts of:

receiving a first request to store a first content unit, the first content unit having a first content address that is computed, at least in part, by hashing the content of the first content unit;

in response to the first request, storing the first content unit in a file in the file system;

receiving a second request to store a second content unit, the second content unit having a second address that is computed, at least in part, by hashing the content of the second content unit; and

in response to the second request, storing the second content unit in the same file in which the first content unit is stored and in a manner that enables the first and second content units to be separately accessible;

receiving a third request to store a third content unit, the third content unit having a same content as the first content unit;

determining that the first and third content units have the same content; and

in response to determining that the first and third content units have the same content, declining to store the third content unit.

22. The at least one computer readable medium of claim 21, wherein the first request is a request to replicate the first content unit from a storage system that stores the first content unit.

23. The at least one computer readable medium of claim 21, wherein the method further comprising acts of:

receiving a fourth request to access the third content unit; and

in response to the fourth request, returning the first content unit.

24. The at least one computer readable medium of claim 21, wherein the file is a first file, and wherein the method further comprises an act of:

storing information in a second file in the file system indicating the location of the first content unit within the first file; and

storing information in the second file indicating the location of the second content unit within the first file.

25. The at least one computer readable medium of claim 21, wherein the file system includes a time-based directory structure having a plurality of directories, wherein each of the plurality of directories corresponds to a period of time, and wherein the file is stored in one of the plurality of directories that corresponds to a period of time during which the first and second request requests were received.

26. The at least one computer readable medium of claim 25, wherein the method further comprises an act of:

creating an entry for the first content unit in an index file, the entry identifying the file that stores the first content unit and wherein the index file is stored in one of the plurality of directories that corresponds to an index time for the first content unit.

27. The at least one computer readable medium of claim 26, wherein the first request is a request to replicate the first content unit from a storage system that stores the first content unit and wherein the index time is a time related to the time of storage of the first content unit on the storage system.

28. The at least one computer readable medium of claim 26, wherein the index time is a time related to the time of receipt of the first request.

29. The at least one computer readable medium of claim 26, wherein the index time is a time derived from the content address of the first content unit.

30. The at least one computer readable medium of claim 29, wherein the time derived from the content address of the first content unit is derived from a portion of the content address that is based on the content of the first content unit.

31. The at least one computer readable medium of claim 25, where the first request is received from a storage system that stores the first content unit, and wherein the first content address includes a timestamp that indicates a time related to the time of storage of the first content unit on the storage system.

32. The at least one computer readable medium of claim 31, wherein the file corresponds to a period of time that includes the times at which the first and second requests were received.

33. The at least one computer readable medium of claim 31, wherein the file corresponds to a period of time that is subsumed by the period time of the one of the plurality of directories in which the file is stored.

34. The at least one computer readable medium of claim 31, wherein the method further comprises acts of:

in response to the first request, determining one the plurality of directories that corresponds to a period of time that includes the time indicated in the timestamp; and

storing, in the one the plurality of directories that corresponds to a period of time that includes the time indicated in the timestamp, an indication of the file that stores first the content unit.

35. The at least one computer readable medium of claim 34, wherein the method further comprises an act of:

in response to the first request, associating an identifier with the first content unit.

36. The at least one computer readable medium of claim 35, where the identifier is the first content address.

37. The at least one computer readable medium of claim 35, wherein the method further comprises an act of:

returning the identifier as acknowledgement of receipt of the first request.

38. The at least one computer readable medium of claim 35, wherein the method further comprises acts of:

receiving a fourth request to access the first content unit, the fourth request identifying the first content unit using the identifier;

locating the indication of the file that stores first the content unit using the identifier;

locating the file that stores the first content unit based on the indication;

locating the first content unit in the file; and

returning the first content unit in response to the fourth request.

39. A computer system that manages content units in a file system comprising:

at least one input; and

at least one controller coupled to the at least one input that;

receives, via the input, a first request to store a first content unit, the first content unit having a first content address that is computed, at least in part, by hashing the content of the first content unit;

in response to the first request, stores the first content unit in a file in the file system;

receives, via the input, a second request to store a second content unit, the second content unit having a second address that is computed, at least in part, by hashing the content of the second content unit;

in response to the second request, stores the second content unit in the same file in which the first content unit is stored and in a manner that enables the first and second content units to be separately accessible;

p2 receives a third request to store a third content unit, the third content unit having a same content as the first content unit;

determines that the first and third content units have the same content; and

in response to determining that the first and third content units have the same content, declines to store the third content unit.

40. The computer of claim 39, wherein the first request is a request to replicate the first content unit from a storage system that stores the first content unit.

41. The computer of claim 39, wherein the at least one controller:

receives, via the input, a fourth request to access the third content unit; and

in response to the fourth request, returns the first content unit.

42. The computer of claim 39, wherein the at least one controller further comprises means for:

receiving, via the input, a first request to store a first content unit, the first content unit having a first content address that is computed based, at least in part, on the content of the first content unit;

in response to the first request, storing the first content unit in a file in the file system;

receiving, via the input, a second request to store the second content unit, the second content unit having a second address that is computed based, at least in part, on the content of the second content unit; and

in response to the second request, storing the second content unit in the file.

43. The computer of claim 39, wherein the file is a first file, and wherein the at least one controller:

stores information in a second file in the file system indicating the location of the first content unit within the first file; and

stores information in the second file indicating the location of the second content unit within the first file.

44. The computer of claim 39, wherein the file system includes a time-based directory structure having a plurality of directories, wherein each of the plurality of directories corresponds to a period of time, and wherein the file is stored in one of the plurality of directories that corresponds to a period of time during which the first and second request requests were received.

50. The computer of claim 44, where the first request is received from a storage system that stores the first content unit, and wherein the first content address includes a timestamp that indicates a time related to the time of storage of the first content unit on the storage system.

51. The computer of claim 50, wherein the at least one controller:

in response to the first request, determines one the plurality of directories that corresponds to a period of time that includes the time indicated in the timestamp; and

stores, in the one the plurality of directories that corresponds to a period of time that includes the time indicated in the timestamp, an indication of the file that stores first the content unit.

52. The computer of claim 50, wherein the file corresponds to a period of time that includes the times at which the first and second requests were received.

53. The computer of claim 50, wherein the file corresponds to a period of time that is subsumed by the period time of the one of the plurality of directories in which the file is stored.

54. The computer of claim 51, wherein the at least one controller:

in response to the first request, associates an identifier with the first content unit.

55. The computer of claim 54, where the identifier is the first content address.

56. The computer of claim 54, wherein the at least one controller:

returns the identifier as acknowledgement of receipt of the first request.

57. The computer of claim 54, wherein the at least one controller:

receives, via the input, a fourth request to access the first content unit, the fourth request identifying the first content unit using the identifier;

locates the indication of the file that stores first the content unit using the identifier;

locates the file that stores the first content unit based on the indication;

locates the first content unit in the file; and

returns the first content unit in response to the fourth request.

45. The computer of clam 44, wherein the at least one controller:

creates an entry for the first content unit in an index file, the entry identifying the file that stores the first content unit and wherein the index file is stored in one of the plurality of directories that corresponds to an index time for the first content unit.

46. The computer of claim 45, wherein the first request is a request to replicate the first content unit from a storage system that stores the first content unit and wherein the index time is a time related to the time of storage of the first content unit on the storage system.

47. The computer of claim 45, wherein the index time is a time related to the time of receipt of the first request.

48. The computer of claim 45, wherein the index time is a time derived from the content address of the first content unit.

49. The computer of claim 48, wherein the time derived from the content address of the first content unit is derived from a portion of the content address that is based on the content of the first content unit.