Reading a file from a cloud storage solution

• US 8,260,913 B2
• Filed: 05/06/2010
• Issued: 09/04/2012
• Est. Priority Date: 05/06/2010
• Status: Active Grant

• Current Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Original Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Inventors: Knapp, Adam, Krouse, Brian, Miller, Joseph, Nilsen, Eric, Strange, Brent
• Primary Examiner(s): Won, Michael
• Application Number:
  US12/774,725
• Publication Number:
  US 20110276622A1
• Time in Patent Office: 852 Days
• Field of Search: 709/203, 709/212, 709/217, 709/231, 709/232, 709/236, 709/245, 709/202, 709/223, 709/230, 709/235, 709/238
• US Class Current: 709/224
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

Systems and methods of the present invention provide for a user requesting a file or a plurality of file chunks stored in a file storage system, and a customized metadata stored in data storage. One or more tracker servers may determine file locations in the file system, and the location of the customized metadata in the data storage. These locations may be returned to an application, possibly an API, which may determine one or more URLs for the file locations and perform an HTTP request to read the file or file chunks from the locations or URLs in the file system. The file or file chunks, along with the customized metadata, may then be returned and displayed to the user that requested them.

Claims

1. A system, comprising:

one or more server computers communicatively coupled to a network and running one or more job master worker modules configured to;

A) receive a request to read a file, and a customized metadata about said file, from said one or more server computers;

B) populate one or more system queues on said one or more server computers with one or more system tasks to read said file and said customized metadata from said one or more server computers;

C) generate at runtime one or more query worker modules configured to load balance said one or more system tasks, said one or more query worker modules comprising;

i) one or more process management validator worker modules configured to monitor said one or more system tasks in said one or more system queues and each of one or more servers for proper functionality;

ii) one or more checksum worker modules configured to determine an integrity of said file by;

a) performing a checksum on said file;

b) re-computing said checksum and comparing it with a stored checksum; and

c) responsive to a determination that said checksum and said stored checksum do not match, identifying one or more modifications or one or more errors introduced to said file during an attempted upload of said file to said one or more servers;

D) responsive to a completion of said one or more system tasks, delete each of a plurality of background processes.

2. The system of claim 1 wherein said file is downloaded or streamed from storage on an FTP site, an online file folder or a video-based social media website.

3. The system of claim 1 wherein said request is received via an application programming interface on said one or more server computers, said application programming interface being configured to:

i) manage communications between said application programming interface and a client computer communicatively coupled to said network;

ii) receive, from said client computer, a request to;

a) read or stream said file from said one or more server computers; and

b) read said customized metadata about said file from a database communicatively coupled to said network;

iii) communicate, to one or more tracker server modules, a plurality of data identifying said file and said customized metadata about said file;

iv) receive, from said one or more tracker server modules, one or more locations, on said one or more server computers, where said file is stored, said one or more locations having been calculated to maximize computing power for each of said one or more server computers;

v) execute a hypertext transfer protocol request, comprising a get command, to read or stream said file from said one or more locations; and

vi) transmit said file, for download or stream, from said one or more locations on said one or more tracker server modules.

4. The system of claim 3 wherein said application programming interface is further configured to:

i) receive, from said client computer, input from a user indicating a selection, via a user interface on said client computer, of said file and said customized metadata about said file;

ii) download, to said client computer, said file, or a stream of said file and said customized metadata.

5. The system of claim 4 wherein said user interface further comprises means to display said file and said customized metadata about said file.

6. The system of claim 3 wherein said application programming interface is further configured, responsive to a determination that said file has been separated and stored in one or more file pieces, to request one or more file piece locations, and receive from said tracker server module:

i) said one or more file piece locations; and

ii) said one or more file pieces.

7. The system of claim 3 further comprising said one or more tracker server modules configured to:

i) manage communications between said application programming interface and said one or more server computers;

ii) receive, from said application programming interface;

a) said request to read or stream said file from said one or more server computers;

b) said request to read said customized metadata about said file from said database; and

c) said plurality of data identifying said file and said customized metadata about said file;

iii) request and receive, from said database, said one or more locations, on said one or more server computers, where said file is stored;

iv) transmit, to said application programming interface, said one or more locations; and

v) initialize said plurality of query worker modules.

8. The system of claim 7 further comprising one or more storage node servers comprising:

i) a server cluster of one or more application servers comprising a distributed hypertext-transfer-protocol-backed data store configured to provide and control a redundancy, a replication and a file retention for said file; and

ii) a cloud storage file system which is;

a) accessed online;

b) configured to store said file;

c) accessible to verify one or more contents of said cloud storage file system;

d) configured to handle one or more request messages to store or serve a plurality of web content from said application programming interface; and

e) configured to act as a web server to store and serve said file and a plurality of other web content configured to handle hypertext transfer protocol request functionality.

9. The system of claim 8 wherein said one or more storage node servers are configured to run one or more parent processes and one or more child processes comprising:

i) one or more disk manager initializer worker modules configured to format one or more disks used by said one or more storage node servers;

ii) one or more disk manager worker modules configured to;

i) manage a health of said one or more disks and report storage statistics of said one or more disks; and

ii) facilitate a configuration, a remote monitoring and management of said server cluster;

iii) one or more file builder worker modules configured to handle one or more requests to build one or more files on said one or more server computers;

iv) one or more file count worker modules configured to count said one or more files stored on said one or more server computers;

v) one or more file delete worker modules configured to determine said one or more files flagged for deletion and delete said one or more files flagged for deletion;

vi) one or more input output statistic worker modules configured to report the performance or status of said one or more server computers in a regular interval; and

vii) one or more storage sync worker modules configured to search through said one or more server computers and validate that a plurality of data about each of said one or more files is located in said database.

10. The system of claim 7 wherein said database further comprises:

i) said one or more locations to store said file, said one or more locations having been calculated by said one or more server computers to maximize computing power for each of said one or more server computers;

ii) a plurality of location metadata for said one or more locations for said file;

iii) said customized metadata about said file;

iv) one or more location, configuration and status data for;

a) said application programming interface;

b) said one or more tracker servers;

c) one or more storage node servers; and

d) one or more database servers; and

v) one or more namespaces for said file, said one or more namespaces being linked to a plurality of information about said file used to more efficiently write to and read from said one or more server computers, said namespace comprising;

a) a key comprising a unique textual string used to identify said file;

b) a file class comprising said file which specifies a desired count or a desired state for file replication according to a replication policy;

c) a file domain comprising said file and said file class within a set of classes comprising a top-level separation of one or more files and comprising a set of classes that define said one or more files within said file domain; and

d) a file id comprising a unique internal numerical representation of said file.

11. The system of claim 3 wherein said one or more process management validator worker modules are further configured to:

i) manage communications from said application programming interface;

ii) ensure that all of said one or more query worker modules are running and functioning properly; and

iii) ensure that all of said one or more server computers are operational.

12. The system of claim 1 wherein:

i) said one or more job master worker modules comprise one or more event-based parent processes comprising a message bus configured to manage all communication from one or more applications on a client computer requesting operations to be performed;

ii) said one or more query worker modules comprise one or more child processes created, managed, handled, communicated with, stopped, reset or deleted by said one or more event-based parent processes, each of said one or more child processes comprising software or scripts executed on said one or more server computers and configured to accomplish one or more designated tasks requested by an application programming interface on said one or more server computers; and

iii) said one or more job master worker modules transmits one or more requests to one or more idle query workers for processing wherein, responsive to none of said one or more idle workers being available, said one or more requests are queued in said system queue for later processing by said one or more idle query workers.

13. The system of claim 1 further comprising one or more sync worker modules configured to:

i) monitor said one or more system tasks in said one or more system queues;

ii) load balance said one or more system tasks in said one or more system queues; and

iii) ensure that each of one or more files uploaded to said one or more server computers has a valid replication policy.

14. The system of claim 1 wherein one or more meta store worker modules configured to ensure that meta information is stored about each of one or more files stored on said one or more server computers.

15. The system of claim 1 further comprising one or more monitor worker modules configured to interface with said one or more process management validator worker modules to:

i) monitor a health and a status of said one or more server computers, said one or more query worker modules and a database communicatively coupled to said network;

ii) ensure that one or more storage node servers storing one or more files, including said file, are in contact with said network;

iii) verify that each copy of said one or more files is stored in one or more locations stored in said database;

iv) verify that said one or more files exist and are a correct length;

v) responsive to a determination that said one or more files are missing, indicated by no working paths to said one or more server computers within said network, attempt to find said one or more files on any of said one or more server computers;

vi) constantly contact said one or more server computers to determine which of said one or more server computers are available and online; and

vii) detect one or more changes to said one or more query worker modules.

16. The system of claim 1 further comprising one or more reaper and rebalance worker modules configured to:

i) ensure that a plurality of traffic in said one or more system queues does not meet a configurable threshold;

ii) calculate a number of bytes created by said plurality of traffic, and flowing through said network;

iii) responsive to a determination that said number of bytes is less than said configurable threshold, add one or more of said query worker modules to said one or more system queues;

iv) responsive to a determination that said number of bytes is greater than or equal to said configurable threshold, wait until said one or more system queues empty before adding one or more of said query workers to said one or more system queues;

v) ensure one or more files, from a non-functional server among said one or more server computers, are copied to one or more valid locations on said one or more servers to fulfill a replication policy for said one or more files;

vi) delete, from a database communicatively coupled to said network, one or more non-valid locations for said non-functional server; and

vii) schedule a replication of said one or more files from said non-functional server.

17. The system of claim 1 further comprising one or more delete worker modules configured to flag one or more files for deletion based on an archiving algorithm, said archiving algorithm being configured to process one or more delete queues among said one or more system queues to permanently remove one or more deleted files, one or more overwritten files or one or more temporary files from said one or more server computers.

18. The system of claim 1 further comprising one or more replication worker modules configured to:

i) automatically check said file against a replication policy;

ii) replicate one or more copies of said file on said one or more server computers according to said replication policy; and

iii) manage a drain and a rebalance of said one or more server computers.

19. The system of claim 18 wherein said replication policy:

i) comprises a desired state comprising a desired count of one or more replicated files for said file, said desired count comprising;

a) a property of a class to which said file belongs; and

b) a definition of how many times one or more files in said class need to be replicated within said one or more server computers to ensure a redundancy and to prevent a data loss within said one or more files;

ii) requires a predetermined number of valid copies of said one or more replicated files to exist prior to transmitting a confirmation of a successful write of said file;

iii) satisfies said desired state; and

iv) provides and controls a retention of said file.