Method and system for managing distributed content and related metadata

• US 7,272,613 B2
• Filed: 10/26/2001
• Issued: 09/18/2007
• Est. Priority Date: 10/26/2000
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Chan, Desmond Cho-Hung, Orzen, Matthew, Huang, Tsan-Fung, Mills, George Harlow, Flood, James C. Jr., Isaacson, Trygve, Sim, Siew Young, Chai, Wencheng
• Primary Examiner(s): PARDO, THUY N
• Application Number:
  US09/984,024
• Publication Number:
  US 20020133491A1
• Time in Patent Office: 2,153 Days
• Field of Search: 707/10, 707/2, 707/8, 707/205, 707/206, 707/102, 707/204, 707/203, 707/101, 707/103.X, 707/104.1, 711133-136, 711159-160, 711/122, 714/6, 709/217, 709/220, 709/223
• US Class Current: 709/223
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1014 based on the content of a request
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 69/14 Multichannel or multilink protocols
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

The invention provides a method and system for creating an innovative file system that separates its directory presentation from its data store. The method and system include processing, division, distribution, managing, synchronizing, and reassembling of file system objects that does not delay the presentation of the content to the user, but also uses a reduced amount of storage space. The invention includes the ability to manage and control the integrity of the files distributed across the network, and the ability to serve and reconstruct files in real time using a Virtual File Control System.

Claims

1. A method comprising:

creating a new file system object;

decomposing the new file system object into a plurality of segments;

generating a virtual storage capacity by storing individual segments of the plurality of segments on a plurality of servers in a network, wherein each server in the network stores a segment and is configured to have a virtual appearance that all of the segments are stored locally at the server;

generating object metadata, wherein the object metadata is used to reconstruct the new file system object from the plurality of segments; and

distributing the object metadata according to a set of distribution criteria that includes one or more of using specified regions in a network and size of the new file system object, wherein the distributing continues until the object metadata reaches a plurality of nodes in the network.

2. The method of claim 1, further comprising:

updating an existing file system object; and

parsing the existing file system object to update existing object metadata corresponding to the existing file system object.

3. The method of claim 1, wherein the set of distribution criteria further comprises distributing using specific nodes from the plurality of nodes in the network.

4. The method of claim 1, wherein the object metadata comprises one or more of the following:

attributes of the file system object;

information on how the file system object is to be segmented; and

locations of the plurality of segments of the file system object.

5. The method of claim 1, further comprising:

requesting access to the file system object;

searching for the plurality of segments that make up the file system object;

retrieving the plurality of segments; and

reconstructing the file system object from the plurality of segments.

6. The method of claim 5, further comprising:

checking a table for file system objects with an expired expiration time;

deleting the file system objects with an expired expiration time;

determining if pruning is required based on overall storage availability and volume storage reservation information;

deleting the file system objects based on usage statistics; and

updating deleted object metadata associated with the deleted file system objects.

7. A machine-readable storage medium having stored thereon data representing sets of instructions which, when executed by a machine, cause the machine to:

create a new file system object;

decompose the new file system object into a plurality of segments;

generate a virtual storage capacity by storing individual segments of the plurality of segments on a plurality of servers in a network, wherein each server in the network stores a segment and is configured to have a virtual appearance that all of the segments are stored locally at the server;

generate object metadata, wherein the object metadata is used to reconstruct the new file system object from the plurality of segments; and

distribute the object metadata according to a set of distribution criteria that includes using specified regions in a network or size of the new file system object, wherein the distributing continues until the object metadata reaches a plurality of nodes in the network.

8. The machine-readable medium of claim 7, further cause the machine to:

request access to the file system object;

search for the plurality of segments that make up the file system object;

retrieve the plurality of segments; and

reconstruct the file system object from the plurality of segments.

9. The machine-readable medium of claim 8, further cause the machine to:

check a table for file system objects with an expired expiration time;

delete the file system objects with an expired expiration time;

determine if pruning is required based on overall storage availability and volume storage reservation information;

delete the file system objects based on usage statistics; and

update deleted object metadata associated with the deleted file system objects.

10. A system comprising:

a distribution center; and

a content management server (CMS) coupled with the distribution center, the CMS to

create a new file system object,

decompose the new file system object into a plurality of segments,

generate a virtual storage capacity by storing individual segments of the plurality of segments on a plurality of servers in a network,

wherein each server in the network stores a segment and is configured to have a virtual appearance that all of the segments are stored locally at the server,

generate object metadata, wherein the object metadata is used to reconstruct the new file system object from the plurality of segments, and

distribute the object metadata according to a set of distribution rules that include using specified regions in a network or size of the new file system object, wherein the CMS continues to distribute until the object metadata reaches a plurality of nodes in the network.

11. The system of claim 10, wherein the CMS is further to update an existing file system object;

and

parse the existing file system object to update existing object metadata corresponding to the existing file system object.

12. The system of claim 10, wherein the distribution center comprises an application server cluster, wherein the application server cluster includes:

a plurality of application servers;

a load balancing system, wherein the load balancing system distributes traffic among the plurality of application servers; and

a virtual file cluster server (VFCS) coupled with the plurality of application servers via the load balancing system, wherein the VFCS is to

request access to the file system object,

search for the plurality of segments that make up the file system object,

retrieve the plurality of segments, and

reconstruct the file system object from the plurality of segments.

13. The system of claim 12, wherein the distribution center further comprises a control unit coupled with the application server cluster, wherein the control unit includes:

a service management system;

a resource management system; and

a storage management system, wherein the storage management system is to

check a table for file system objects with expired expiration times,

delete the file system objects with expired expiration times,

determine if pruning is required based on overall storage availability and volume storage reservation information,

delete the file system objects based on usage statistics, and

update deleted object metadata associated with the deleted file system objects.