Content storage and replication in a managed internet content storage environment

• US 20020147774A1
• Filed: 04/01/2002
• Published: 10/10/2002
• Est. Priority Date: 04/02/2001
• Status: Active Grant

• Current Assignee: Akamai Technologies, Inc.
• Original Assignee: Akamai Technologies, Inc.
• Inventors: Nicolaou, Cosmos, Lisiecki, Philip A., Rose, Kyle R.
• Granted Patent: US 7,340,505 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1012 based on compliance of requirements or conditions with available server resources
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/2895 Intermediate processing functionally located close to the data provider application, e.g. reverse proxies
• CPC Class Codes: H04L 67/563 Data redirection of data network streams

Abstract

A method for content storage on behalf of participating content providers begins by having a given content provider identify content for storage. The content provider then uploads the content to a given storage site selected from a set of storage sites. Following upload, the content is replicated from the given storage site to at least one other storage site in the set. Upon request from a given entity, a given storage site from which the given entity may retrieve the content is then identified. The content is then downloaded from the identified given storage site to the given entity. In an illustrative embodiment, the given entity is an edge server of a content delivery network (CDN).

Claims

1. A managed storage site operative at a given location in a network, comprising:

a network filesystem;

a set of client servers;

a set of one or more file servers that export the network filesystem to the set of client servers;

a content upload process operative on at least one of the client servers for receiving a content upload from a participating content provider;

a content replication process operative on at least one of the client servers for managing (a) storage of the content uploaded from the participating content provider, and (b) replication of the content to at least one commonly-managed replica site; and

a content download process operative on at least one of the client servers for managing delivery of the content to a requesting entity.

2. The managed storage site as described in claim 1 further including:

at least one first switch to which each of the client servers is connectable, and to which each of the file servers is connectable, the first switch isolating the file servers from access except through a given one of the client servers.

3. The managed storage site as described in claim 1 further including a second switch to which each of the client servers is connectable to the publicly-routable Internet.

4. The managed storage site as described in claim 1 further including a traffic management mechanism for selecting a given managed storage site to receive a content upload or to provide a content download as a function of network traffic conditions.

5. The managed storage site as described in claim 1 wherein at least a given subset of the client servers execute the content replication process.

6. The managed storage site as described in claim 1 wherein the content replication process includes a leader election mechanism to select a given content replication process to manage content uploads and downloads from the managed storage site.

7. The managed storage site as described in claim 1 wherein the leader election mechanism elects at most one content replication process to manage content uploads and downloads from the managed storage site.

8. The managed storage site as described in claim 1 further including a local action log having one or more entries, wherein each entry in the local action log is associated with an upload event at the managed storage site.

9. The managed storage site as described in claim 8 further including a remote action log having one or more entries, wherein each entry in the remote action log is associated with an upload event at the commonly-managed replica site.

10. The managed storage site as described in claim 9 further including code executing in a client server for reconciling the network filesystem of the managed storage site and the network filesystem at the commonly-managed replica site.

13. In the managed storage site as described in claim 11, the improvement further comprising:

at least one first switch connectable between each of the set of file servers and each of the set of client servers, wherein the first switch isolates the network filesystem from the content upload.

11. In a storage site comprising a network filesystem and a set of file servers that export the network file system, the improvement comprising:

a set of client servers;

at least one content upload process operative on at least one of the client servers for receiving a content upload from a participating content provider;

a set of content replication processes operative on the client servers; and

a leader election routine for selecting a given one of the content replication processes to manage (a) storage of the content uploaded from the participating content provider, and (b) replication of the content to at least one commonly-managed replica site.

12. In the managed storage site as described in claim 11, the improvement further comprising:

at least one content download process operative on at least one of the client servers for managing delivery of the content to a requesting entity.

14. In the managed storage site as described in claim 11, the improvement further comprising:

a switch connectable between each of the set of client servers and the Internet.

16. In the storage site as described in claim 15 wherein the content upload process is server having a file transfer mechanism.

17. In the storage site as described in claim 15 wherein the content download process in an HTTP server.

18. In the storage site as described in claim 15 wherein the requesting entity is a content delivery network (CDN) server.

19. In the storage site as described in claim 15 further including:

a local action log having one or more entries, wherein each entry in the local action log is associated with an upload event at the managed storage site;

a remote action log having one or more entries, wherein each entry in the remote action log is associated with an upload event at the commonly-managed replica site; and

code executing in a client server for using the local action log and the remote action log to reconcile the network filesystem of the managed storage site with the network filesystem at the commonly-managed replica site.

15. In a storage site including a network filesystem and a set of file servers that export the network filesystem to a set of client servers, a managed content storage system, comprising:

at least one content upload process operative on at least one of the client servers for receiving a content upload from a participating content provider;

a set of content replication processes operative on the client servers;

a leader election routine for selecting a given one of the content replication processes to manage (a) storage of the content uploaded from the participating content provider, and (b) replication of the content to at least one commonly-managed replica site; and

at least one content download process operative on at least one of the client servers for managing delivery of the content to a requesting entity.

20. A managed storage service, comprising:

a set of storage sites located at first and second given locations connectable over the public Internet, wherein each storage site has associated therewith a network filesystem and a set of one or more file servers that export the network file system, each storage site comprising;

a set of client servers;

a content upload process operative on at least one of the client servers for receiving content uploads to the storage site from participating content providers;

a content replication process operative on at least one of the client servers for managing (a) storage the content uploads in the network filesystem, and (b) replication of the content uploads to at least one other of the storage sites; and

a content download process operative on at least one of the client servers for managing download of stored content to a requesting entity;

wherein the storage sites operate autonomously with respect to each other such that content uploads to and content downloads from a given storage site take place irrespective of whether the network filesystems in the storage sites are synchronized.