Scalable, high performance and highly available distributed storage system for Internet content

• US 7,624,169 B2
• Filed: 11/10/2006
• Issued: 11/24/2009
• Est. Priority Date: 04/02/2001
• Status: Active Grant

• Current Assignee: Akamai Technologies, Inc.
• Original Assignee: Akamai Technologies, Inc.
• Inventors: Nicolaou, Cosmos, Lisiecki, Philip A., Rose, Kyle R.
• Primary Examiner(s): ISMAIL, SHAWKI SAIF
• Application Number:
  US11/595,230
• Publication Number:
  US 20070055765A1
• Time in Patent Office: 1,110 Days
• Field of Search: 709/203, 709217-218, 709223-224, 709/238, 709/248
• US Class Current: 709/223
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1012 based on compliance of requirements or conditions with available server resources
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/2895 Intermediate processing functionally located close to the data provider application, e.g. reverse proxies
• CPC Class Codes: H04L 67/563 Data redirection of data network streams

Abstract

A method for content storage on behalf of participating content providers begins by having a given content provider identify content for storage. The content provider then uploads the content to a given storage site selected from a set of storage sites. Following upload, the content is replicated from the given storage site to at least one other storage site in the set. Upon request from a given entity, a given storage site from which the given entity may retrieve the content is then identified. The content is then downloaded from the identified given storage site to the given entity. In an illustrative embodiment, the given entity is an edge server of a content delivery network (CDN).

Claims

1. A method of content storage on behalf of participating content providers, comprising:

maintaining a set of storage sites managed by a content delivery network (CDN) service provider distinct from the participating content providers, wherein the storage sites of the set of storage sites are located in different Internet-accessible locations and operate asynchronously and autonomously from one another, wherein each of the given storage sites comprises a network filesystem, a set of one or more client servers, a set of one or more file servers that export the network filesystem to the set of client servers, a content upload process operative on at least one of the client servers, a content replication process operative on at least one of the client servers, and a content download process operative on at least one of the client servers, the service provider also operating a traffic manager that estimates relative connectivity to the storage sites from a set of proxy points, wherein each proxy point is determined by directing a trace route from given storage sites and determining a given point in the Internet where the trace routes intersect, and wherein the relative connectivity is determined by probing each of the proxy points;

for each piece of content identified for upload by a participating content provider, selecting a given upload storage site from the set of storage sites by having the traffic manager resolve a first DNS query without reference to an identifier for the piece of content;

at the given upload storage site identified by resolving the first DNS query, using a content upload process to receive an upload of the piece of content identified by the participating content provider, and using a replication process to store the piece of content received by the content upload process and to initiate replication of the piece of content to at least one other storage site in the set of storage sites;

for each piece of content identified for download by a given server entity, the given server entity being a CDN edge server, selecting a given download storage site from the set of storage sites by having the traffic manager resolve a second DNS query without reference to an identifier for the piece of content, wherein the second DNS query differs from the first DNS query;

at the given download storage site identified by resolving the second DNS query, using a download process to attempt to download the piece of content to the given server entity;

for a given piece of content identified for download by a given server entity that is not then available for download from the given download storage site selected from the set of storage sites, redirecting a request for the given piece of content to one or more of the storage sites, wherein during the redirecting step a redirecting threshold is maintained such that, when the redirecting threshold is met, the given piece of content is then dynamically received from a participating content provider origin server;

maintaining an activity log at each storage site; and

at a second storage site, replaying an activity log recorded at a first storage site to synchronize content across the first and second storage sites.

2. The method as described in claim 1 wherein the edge server identifies the piece of content for download upon the occurrence of a cache miss at the edge server.

3. The method as described in claim 1 wherein the edge server communicates with the given download server storage site over a secure connection.

4. The method as described in claim 1 wherein the piece of content is one of:

Web content, a media file, and a software file.

5. The method as described in claim 1 wherein the content upload process is a file transfer mechanism.

6. The method as described in claim 1 further including a step of presenting the participating content provider with a top-level directory unique to the participating content provider for the upload.