Scalable, high performance and highly available distributed storage system for Internet content

• US 7,472,178 B2
• Filed: 04/01/2002
• Issued: 12/30/2008
• Est. Priority Date: 04/02/2001
• Status: Expired due to Term

• Current Assignee: Akamai Technologies, Inc.
• Original Assignee: Akamai Technologies, Inc.
• Inventors: Nicolaou, Cosmos, Lisiecki, Philip A., Rose, Kyle R.
• Primary Examiner(s): Najjar; Saleh
• Assistant Examiner(s): Ismail; Shawki S
• Application Number:
  US10/113,183
• Publication Number:
  US 20020143888A1
• Time in Patent Office: 2,465 Days
• Field of Search: 709217-218, 709/219, 709/204, 709248-249, 709/203, 709223-224, 709/238, 707/204
• US Class Current: 709/223
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1012 based on compliance of requirements or conditions with available server resources
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/2895 Intermediate processing functionally located close to the data provider application, e.g. reverse proxies
• CPC Class Codes: H04L 67/563 Data redirection of data network streams

Abstract

A method for content storage on behalf of participating content providers begins by having a given content provider identify content for storage. The content provider then uploads the content to a given storage site selected from a set of storage sites. Following upload, the content is replicated from the given storage site to at least one other storage site in the set. Upon request from a given entity, a given storage site from which the given entity may retrieve the content is then identified. The content is then downloaded from the identified given storage site to the given entity. In an illustrative embodiment, the given entity is an edge server of a content delivery network (CDN).

Claims

1. A method of content storage and delivery, operative in a content delivery network (CDN) deployed, operated and managed by a service provider on behalf of participating content providers, where the participating content providers are distinct from the service provider, comprising:

(a) deploying a content delivery network that comprises (i) a plurality of CDN content servers that are organized into sets, (ii) one or more CDN storage sites that are distinct from the plurality of content servers; and

(iii) one or more CDN-managed domain name servers each of which is authoritative to resolve predetermined CDN-specific hostnames, where each such hostname has a first portion uniquely associated with a given participating content provider and a second portion associated with the service provider;

(b) for a first participating content provider;

(i) establishing a first content storage directory at a storage site;

(ii) providing the first participating content provider with a first CDN-specific hostname for use in association with the first content storage directory, the first CDN-specific hostname having a first portion uniquely associated with the first participating content provider, and the second portion;

(iii) receiving, via upload, and storing, in the first content storage directory at the storage site, first content, where the first content is one of;

a web object, a media file, or a software download that the first participating content provider desires to be stored and delivered over the CDN;

(c) for a second participating content provider distinct from the first content provider;

(i) establishing a second content storage directory at a storage site;

(ii) providing the second participating content provider with a second CDN-specific hostname for use in association with the second content storage directory, the second CDN-specific hostname having a first portion uniquely associated with the second participating content provider, and the second portion;

(iii) receiving, via upload, and storing, in the second content storage directory at the storage site, second content, where the second content is one of;

web object, a media file, or a software download that the second participating content provider desires to be stored and delivered over the CDN;

(d) upon receipt by a CDN-managed domain name server of a DNS query associated with the first CDN-specific hostname;

(i) identifying one or more CDN content servers;

(ii) receiving a request for the first content at one of the identified content servers;

(iii) if the first content is available to be served, serving the first content; and

(iv) if the first content is not available to be served, obtaining a copy of the first content from the first content storage directory as needed to respond to the request for the first content;

(e) upon receipt by a CDN-managed domain name server of a DNS query associated with the second CDN-specific hostname;

(i) identifying one or more CDN content servers;

(ii) receiving a request for the second content at one of the identified CDN content servers;

(iii) if the second content is available to be served, serving the second content; and

(iv) if the second content is not available to be served, obtaining a copy of the second content from the second content storage directory as needed to respond to the request for the second content;

where the second portion is common to each of the first and second CDN-specific hostnames, and where the first CDN-specific hostname also has associated therewith a value that identifies the first content storage directory from which the copy of the first content can be obtained, and where the second CDN-specific hostname also has associated therewith a value that identifies the second content storage directory from which the copy of the second content can be obtained;

wherein the one or more CDN storage sites include first and second CDN storage sites that are each located in different Internet-accessible locations and operate asynchronously and autonomously from one another.

2. The method as described in claim 1 wherein a CDN content server is directed to a CDN storage site upon the occurrence of a cache miss at the CDN content server.

3. The method as described in claim 2 wherein the CDN content server communicates with the CDN storage site over a secure connection.

4. The method as described in claim 1 further including the step of replicating the first content from a first CDN storage site to at least one other CDN storage site.

5. The method of claim 1 further including:

maintaining an activity log at each CDN storage site, and

at a first CDN storage site, replaying an activity log recorded at a second CDN storage site to synchronize content across the first and second CDN storage sites.

6. The method as described in claim 1 wherein the upload is an FTP upload.

7. The method as described in claim 1 wherein the upload is performed via FTP and the request for the first or second content is received via HTTP.

8. The method as described in claim 1 wherein the first and second content storage directories are located on a single CDN storage site.

9. The method as described in claim 1 wherein the first and second content storage directories are located on respective first and second CDN storage sites.

10. The method as described in claim 1 further including storing multiple content files of a participating content provider over multiple storage areas within a storage site.