SYSTEMS AND METHODS FOR STORING DIGITAL CONTENT

• US 20130138649A1
• Filed: 07/01/2011
• Published: 05/30/2013
• Est. Priority Date: 07/02/2010
• Status: Active Grant

• Current Assignee: MetaCDN Pty Ltd
• Original Assignee: MetaCDN Pty Ltd
• Inventors: Broberg, James Andrew
• Granted Patent: US 9,727,579 B2
• Time in Patent Office: Days
• US Class Current: 707/736
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 65/103 in the network
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1012 based on compliance of requirements or conditions with available server resources
• CPC Class Codes: H04L 67/1021 based on client or server locations
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1025 Dynamic adaptation of the criteria on which the server selection is based
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/289 Intermediate processing functionally located close to the data consumer application, e.g. in same machine, in same home or in same sub-network
• CPC Class Codes: H04L 67/567 Integrating service provisioning from a plurality of service providers
• CPC Class Codes: H04L 67/5682 Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 67/61 taking into account QoS or priority requirements
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests

Abstract

An overlay content distribution network provides a consistent interface to enable the storage and retrieval of digital content at multiple storage providers (210). An allocator component (140) instantiates and destroys location and/or provider specific agent processes (240) which transfer digital content to and from the storage providers (210). Encoding agent processes (250) are geographically located near the storage providers (210) at which the content is to be stored. Content may be stored in a manner which guarantees content availability, each storage provider (210) being periodically probed by a Quality of Service (QoS) probes distributed within the network.

Claims

1. A computer-implemented system for storing digital content at at least one remote storage location maintained by a storage provider, the system including

a gateway component to receive storage request data representing a request to store digital content, the storage request data including data identifying a source location of the digital content;

an allocator component configured to;

determine from a plurality of remote storage locations the at least one remote storage location at which the digital content is to be stored, wherein the plurality of remote storage locations includes a first location accessible using a first message format and a second location accessible using a second different message format;

send, to at least one agent process associated with the at least one remote storage location, transfer request data representing a request to retrieve the digital content from the source location, and store it in the at least one storage location.

2. A system as claimed in claim 1 wherein the first location and second location are accessible independently of each other and independently of the allocator component.

3. A system as claimed in claim 1 wherein the allocator component is further configured to:

receive at least one status message from each of the at least one agent processes, each status message containing;

data representing whether or not the digital content was successfully stored; and

if the digital content was successfully stored, stored content location data representing a reference to the stored digital content.

4. A system as claimed in claim 3 wherein the allocator component is further configured to store the stored content location data in a content reference database communicatively connected to the allocator component.

5. A system as claimed in claim 3 wherein the allocator component is further configured to create one or more additional agent processes upon receiving storage request data if the size of the digital content requested to be retrieved and stored by existing agent processes exceeds the size of the digital content successfully stored by the existing agent processes by an amount greater than a capacity threshold.

6. A system as claimed in claim 3 wherein the allocator component is further configured to create one or more additional agent processes upon receiving storage request data if all of the existing agent processes are retrieving digital content from a source location or storing it in at least one remote storage location.

7. A system as claimed in claim 5 wherein the allocator component is further configured to periodically terminate any idle agent processes.

8. A system as claimed in claim 1 wherein the storage request data includes strategy data representing a digital content deployment strategy.

9. A system as claimed in claim 8 wherein the strategy data represents one of:

deployment for maximum coverage and performance;

location-specific deployment;

quality-of-service deployment; and

cost-optimised deployment.

10. A system as claimed in claim 9, wherein where strategy data represents quality-of-service deployment and the storage request data also includes desired performance data representing desired content serving performance characteristics, the allocator component is further configured to query a storage location database storing details of the plurality of remote storage locations to identify at least one remote storage location capable of serving the digital content in a manner which meets or exceeds the desired content serving performance characteristics.

11. A system as claimed in claim 10, wherein the system further includes a manager component configured to:

periodically receive actual performance data representing measured content serving performance characteristics relating to at least one of the plurality of remote storage locations;

if the actual performance data represents measured content serving performance characteristics that do not meet or exceed the desired content serving performance characteristics, and a copy of the digital content is stored at the at the least remote storage location, send, to the at least one agent process associated with the at least one storage location, transfer request data representing a request to deactivate or delete the digital content from the at least one remote storage location; and

if the actual performance data represents measured content serving performance characteristics that meet or exceed the desired content serving performance characteristics, and a copy of the digital content is not stored at the at least remote storage location, send, to the at least one agent process associated with the at least one storage location, transfer request data representing a request to copy the digital content from a remote storage location from which it is available to the at least one storage location.

12. A system as claimed in claim 9, wherein where the strategy data represents cost-optimised deployment and the storage request data also includes desired budget data representing at least one constraint on the cost of storing and serving the digital content, the allocator component is further configured to:

retrieve, from a storage location database storing details of the plurality of remote storage locations, cost data representing the basis upon which the cost of storing and served the digital content at each of the remote storage locations may be calculated;

determine, from the storage request data and the cost data, selected storage locations at which the digital content may be stored and served at a cost within the constraint; and

send, to at least one agent process associated with each of the selected storage locations, transfer request data representing a request to retrieve the digital content from the source location, and store it in the at least one storage location.

13. A system as claimed in claim 12, wherein the allocator component is configured to determine the selected storage locations using an integer program.

14. A system as claimed in claim 1 wherein:

the gateway component is configured to receive storage request data including data representing one or more desired characteristics of multimedia digital content;

the allocator component is further configured to;

select or create at least one encoding process for execution on at least one remote processor, the at least one remote processor being selected based, at least in part, on its geographic location; and

send, to the least one remote encoding process, encoding request data representing a request to retrieve the multimedia digital content from the source location and encode it to generate target multimedia digital content conforming to the one or more desired characteristics of multimedia digital content.

15. A system as claimed in claim 14 wherein the allocator component is further configured to:

receive at least one encoding status message from each of the at least one remote encoding processes, each status message containing;

data representing whether or not the multimedia digital content was successfully encoded; and

if the multimedia digital content was successfully encoded, target content location data representing a reference to the target multimedia digital content,

upon receipt an encoding status message from a remote encoding process indicating successful encoding of the multimedia digital content, send, to at least one agent process, transfer request data representing a request to retrieve the target multimedia digital content using the target content location data, and store it in the at least one storage location.

16. A system as claimed in claim 14 wherein the allocator component is further configured to select or create the at least one encoding process based, at least in part, on its actual or desired geographic location relative to the geographic location of the at least one storage location.

17. A method for storing content at at least one storage location maintained by a storage provider, the method including the steps of:

receiving storage request data representing a request to store digital content, the storage request data including data identifying a source location of the digital content;

determining from a plurality of remote storage locations the at least one storage location at which the digital content is to be stored, wherein the plurality of remote storage locations includes a first location accessible using a first message format and a second location accessible using a second different message format; and

sending, to at least one agent process associated with the at least one storage location, transfer request data representing a request to retrieve the digital content from the source location, and store it in the at least one storage location.

18. A method as claimed in claim 17 wherein the first location and the second location are accessible independently of each other and without the use of the agent process.

19. A method as claimed in claim 17, further including the steps of:

receiving at least one status message from each of the at least one agent processes, each status message containing;

data representing whether or not the digital content was successfully stored; and

if the data was successfully stored, stored content location data representing a reference to the stored digital content.

20. A method as claimed in claim 17, further including the step of creating one or more additional agent processes upon receiving storage request data if the size of the digital content requested to be retrieved and stored by existing agent processes exceeds the size of the digital content successfully stored by the existing agent processes by an amount greater than a capacity threshold

21. A method as claimed in claim 20, further comprising the step of periodically destroying any idle agent processes.

22. A method as claimed in claim 17, further including the steps of:

receiving storage request data including data representing one or more desired characteristics of multimedia digital content;

selecting or creating at least one encoding process for execution on at least one remote processor, the at least one remote processor being selected or created based, at least in part, on its geographic location; and

sending, to the least one remote encoding process, encoding request data representing a request to retrieve the multimedia digital content from a source location and encode it to generate target multimedia digital content conforming to the one or more desired characteristics of multimedia digital content.

23. A method as claimed in claim 22, further comprising the steps of:

receiving at least one encoding status message from each of the at least one remote encoding processes, each status message containing;

data representing whether or not the multimedia digital content was successfully encoded; and

if the multimedia digital content was successfully encoded, target content location data representing a reference to the target multimedia digital content,

upon receipt of an encoding status message from a remote encoding process indicating successful encoding of the multimedia digital content, sending, to at least one agent process, transfer request data representing a request to retrieve the target multimedia digital content using the target content location data, and store it in the at least one storage location.