Systems and methods for storing digital content

• US 9,727,579 B2
• Filed: 07/01/2011
• Issued: 08/08/2017
• Est. Priority Date: 07/02/2010
• Status: Active Grant

• Current Assignee: MetaCDN Pty Ltd
• Original Assignee: MetaCDN Pty Ltd
• Inventors: Broberg, James Andrew
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Morris, John J
• Application Number:
  US13/807,279
• Publication Number:
  US 20130138649A1
• Time in Patent Office: 2,230 Days
• Field of Search: 707736, 709219, 709226
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 65/103 in the network
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1012 based on compliance of requirements or conditions with available server resources
• CPC Class Codes: H04L 67/1021 based on client or server locations
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/1025 Dynamic adaptation of the criteria on which the server selection is based
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/289 Intermediate processing functionally located close to the data consumer application, e.g. in same machine, in same home or in same sub-network
• CPC Class Codes: H04L 67/567 Integrating service provisioning from a plurality of service providers
• CPC Class Codes: H04L 67/5682 Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 67/61 taking into account QoS or priority requirements
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests

Abstract

An overlay content distribution network provides a consistent interface to enable the storage and retrieval of digital content at multiple storage providers (210). An allocator component (140) instantiates and destroys location and/or provider specific agent processes (240) which transfer digital content to and from the storage providers (210). Encoding agent processes (250) are geographically located near the storage providers (210) at which the content is to be stored. Content may be stored in a manner which guarantees content availability, each storage provider (210) being periodically probed by a Quality of Service (QoS) probes distributed within the network.

Claims

1. A computer-implemented system for storing digital content in a remote storage location, the system including:

a gateway module configured to receive storage request data representing a request to store digital content, the storage request data including data identifying a source location of the digital content;

an allocator module configured to;

determine from a plurality of remote storage locations the remote storage location at which the digital content is to be stored, wherein the plurality of remote storage locations includes a first location accessible using a first message format and a second location accessible using a second different message format, wherein at least one agent process communicates with the remote storage location;

receive at least one status message, from one or more of the at least one agent processes, containing data representing whether or not previous digital content has been successfully stored;

dynamically create one or more additional agent processes that communicate with the remote storage location,

wherein the one or more additional agent processes are overlay processes external to the storage location, and the one or more additional agent processes are configured to transfer the digital content,

wherein the creation of the one or more additional agent processes is in response to;

a determination, based on the at least one status message, that adding the digital content to a queue of the at least one agent process will cause the queue to exceed capacity threshold; and

/or

a determination that no agent processes that communicate with the remote storage location are available;

send, to the one or more additional agent processes that communicate with the remote storage location, transfer request data representing a request to retrieve the digital content from the source location, and store it in the remote storage location, wherein the one or more additional agent processes return a content location after storing the digital content, and the stored content location is for directly accessing the storage location without utilizing the one or more additional agent processes.

2. A system as claimed in claim 1 wherein the first location and second location are accessible independently of each other and independently of the allocator module.

3. A system as claimed in claim 1 wherein the at least one status message further contains:

data representing that the digital content was successfully stored; and

stored content location data representing a reference to the stored digital content.

4. A system as claimed in claim 3 wherein the allocator module is further configured to store the stored content location data in a content reference database communicatively connected to the allocator module.

5. A system as claimed in claim 1 wherein the allocator module is further configured to create the one or more additional agent processes upon the determination, based on the at least one status message, that adding the digital content to the queue of the at least one agent process will cause the queue to exceed the capacity threshold.

6. A system as claimed in claim 1 wherein the allocator module is further configured to create the one or more additional agent processes upon the determination that all agent processes associated with the at least one remote storage location are retrieving content;

storing content;

or are otherwise unavailable.

7. A system as claimed in claim 1 wherein the allocator module is further configured to periodically terminate any idle agent processes.

8. A system as claimed in claim 1 wherein the storage request data includes strategy data representing a digital content deployment strategy.

9. A system as claimed in claim 8 wherein the strategy data represents one of:

deployment for maximum coverage and performance;

location-specific deployment;

quality-of-service deployment; and

cost-optimized deployment.

10. A system as claimed in claim 9, wherein where strategy data represents quality-of-service deployment and the storage request data also includes desired performance data representing desired content serving performance characteristics, the allocator module is further configured to query a storage location database storing details of the plurality of remote storage locations to identify at least one remote storage location capable of serving the digital content in a manner which meets or exceeds the desired content serving performance characteristics.

11. A system as claimed in claim 10, wherein the system further includes a manager module configured to:

periodically receive actual performance data representing measured content serving performance characteristics relating to at least one of the plurality of remote storage locations;

if the actual performance data represents measured content serving performance characteristics that do not meet or exceed the desired content serving performance characteristics, and a copy of the digital content is stored at the at least remote storage location, send, to the at least one agent process associated with the at least one storage location, transfer request data representing a request to deactivate or delete the digital content from the at least one remote storage location; and

if the actual performance data represents measured content serving performance characteristics that meet or exceed the desired content serving performance characteristics, and a copy of the digital content is not stored at the at least remote storage location, send, to the at least one agent process associated with the at least one storage location, transfer request data representing a request to copy the digital content from a remote storage location from which it is available to the at least one storage location.

12. A system as claimed in claim 9, wherein where the strategy data represents cost-optimized deployment and the storage request data also includes desired budget data representing at least one constraint on the cost of storing and serving the digital content, the allocator module is further configured to:

retrieve, from a storage location database storing details of the plurality of remote storage locations, cost data representing the basis upon which the cost of storing and served the digital content at each of the remote storage locations may be calculated;

determine, from the storage request data and the cost data, selected storage locations at which the digital content may be stored and served at a cost within the constraint; and

send, to at least one agent process associated with each of the selected storage locations, transfer request data representing a request to retrieve the digital content from the source location, and store it in the at least one storage location.

13. A system as claimed in claim 12, wherein the allocator module is configured to determine the selected storage locations using an integer program.

14. A system as claimed in claim 1 wherein:

the storage request data further includes data representing one or more desired characteristics of multimedia digital content; and

the allocator module is further configured to;

select or create at least one remote encoding process for execution on at least one remote processor, the at least one remote processor being selected based, at least in part, on its geographic location; and

send, to the at least one remote encoding process, encoding request data representing a request to retrieve the multimedia digital content from the source location and encode it to generate target multimedia digital content conforming to the one or more desired characteristics of multimedia digital content.

15. A system as claimed in claim 14, wherein the allocator module is further configured to receive at least one encoding status message from each one or more of the at least one remote encoding processes, the at least one encoding status message containing:

data representing that the multimedia digital content was successfully encoded; and

target content location data representing a reference to the target multimedia digital content; and

wherein the transfer request data is sent by the allocator module upon receipt of the at least one encoding status message.

16. A system as claimed in claim 14 wherein the allocator module is further configured to select or create the at least one remote encoding process based, at least in part, on its actual or desired geographic location relative to the geographic location of the at least one storage location.

17. A method for storing content in a storage location, the method including:

receiving storage request data representing a request to store digital content, the storage request data including data identifying a source location of the digital content;

determining from a plurality of remote storage locations the storage location at which the digital content is to be stored, wherein the plurality of remote storage locations includes a first location accessible using a first message format and a second location accessible using a second different message format, wherein at least one agent process connects to the storage location;

receiving at least one status message, from one or more of the at least one agent processes, containing data representing whether or not previous digital content has been successfully stored;

dynamically creating one or more additional agent processes that connect to the storage location,

wherein the one or more additional agent processes are overlay processes external to the storage location, and the one or more additional agent processes are configured to transfer the digital content;

wherein the creation of the one or more additional agent processes is in response to;

a determination, based on the at least one status message, that adding the digital content to a queue of the at least one agent process will cause the queue to exceed capacity threshold; and

/or

a determination that no agent processes that communicates with the storage location are available; and

sending, to the one or more additional agent processes that connect to the storage location, transfer request data representing a request to retrieve the digital content from the source location, and store it in the storage location, wherein the one or more additional agent processes return a content location after storing the digital content, and the stored content location is for directly accessing the storage location without utilizing the one or more additional agent processes.

18. A method as claimed in claim 17 wherein the first location and the second location are accessible independently of each other and without the use of the at least one agent process.

19. A method as claimed in claim 17, wherein the at least one status message further contains:

data representing that the digital content was successfully stored; and

stored content location data representing a reference to the stored digital content.

20. A method as claimed in claim 17, further including creating the one or more additional agent processes upon the determination, based on the at least one status message, that adding the digital content to the queue of the at least one agent process will cause the queue to exceed the capacity threshold.

21. A method as claimed in claim 20, further comprising periodically destroying any idle agent processes.

22. A method as claimed in claim 17, wherein:

the storage request data further includes data representing one or more desired characteristics of multimedia digital content; and

the method further includes;

selecting or creating at least one remote encoding process for execution on at least one remote processor, the at least one remote processor being selected or created based, at least in part, on its geographic location; and

sending, to the at least one remote encoding process, encoding request data representing a request to retrieve the multimedia digital content from a source location and encode it to generate target multimedia digital content conforming to the one or more desired characteristics of multimedia digital content.

23. A method as claimed in claim 22, further comprising:

receiving at least one encoding status message from one or more of the at least one remote encoding processes, the at least one encoding status message containing;

data representing that the multimedia digital content was successfully encoded; and

target content location data representing a reference to the target multimedia digital content;

wherein the transfer request data is sent upon receipt of the at least one encoding status message.

24. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for storing content in a storage location, the operations comprising:

receiving storage request data representing a request to store digital content, the storage request data including data identifying a source location of the digital content;

determining from a plurality of remote storage locations the storage location at which the digital content is to be stored, wherein the plurality of remote storage locations includes a first location accessible using a first message format and a second location accessible using a second different message format, wherein at least one agent process connects to the storage location;

receiving at least one status message, from one or more of the at least one agent processes, containing data representing whether or not previous digital content has been successfully stored;

dynamically creating one or more additional agent processes that connect to the storage location,

wherein the one or more additional agent processes are overlay processes external to the storage location, and the one or more additional agent processes are configured to transfer the digital content;

wherein the creation of the one or more additional agent processes is in response to;

a determination, based on the at least one status message, that adding the digital content to a queue of the at least one agent process will cause the queue to exceed capacity threshold; and

/or

a determination that no agent processes that communicates with the storage location are available; and

sending, to the one or more additional agent processes that connect to the storage location, transfer request data representing a request to retrieve the digital content from the source location, and store it in the storage location, wherein the one or more additional agent processes return a content location after storing the digital content, and the stored content location is for directly accessing the storage location without utilizing the one or more additional agent processes.