Writing a file to a cloud storage solution

• US 8,234,372 B2
• Filed: 05/05/2010
• Issued: 07/31/2012
• Est. Priority Date: 05/05/2010
• Status: Active Grant

• Current Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Original Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Inventors: Knapp, Adam, Krouse, Brian, Miller, Joseph, Nilsen, Eric, Strange, Brent
• Primary Examiner(s): Salad, Abdullahi
• Assistant Examiner(s): Tran, Jimmy H
• Application Number:
  US12/774,723
• Publication Number:
  US 20110276656A1
• Time in Patent Office: 818 Days
• Field of Search: 709/219, 709/202, 709/203, 709/212, 709/217, 709/230, 709/238, 709/245
• US Class Current: 709/224
• CPC Class Codes: G06F 16/10 File systems; File servers

Abstract

Systems and methods of the present invention provide for a user uploading a file and a customized metadata about the file to a file storage system. This information may be forwarded to an application, possibly an API, which may divide the file into a plurality of file chunks if the file is greater than a predetermined file size. One or more tracker servers may check a database to locate optimal file locations for the file or file chunks to be stored within the file storage system. These optimal locations may be returned from the tracker servers to the application/API which may perform an HTTP request to write the file to the file storage system. The file write may be validated and finalized, and the successful write may be displayed to the user on a user interface on a client.

Claims

1. A system, comprising:

one or more server computers communicatively coupled to a network and running one or more job master worker modules configured to;

A) receive a request to store a file, and a customized metadata about said file, on said one or more server computers;

B) populate one or more system queues on said one or more server computers with one or more system tasks to write said file and said customized metadata to said one or more server computers;

C) generate at runtime one or more query worker modules configured to load balance said one or more system tasks, said one or more query worker modules comprising;

i) one or more process management validator worker modules configured to monitor said one or more system tasks in said one or more system queues and each of said one or more servers for proper functionality;

ii) one or more sync worker modules configured to remove one or more temporary files or one or more expired files created during an attempted upload of said file to said one or more server computers;

iii) one or more checksum worker modules configured to determine an integrity of said file by;

a) performing a checksum on said file;

b) re-computing said checksum and comparing it with a stored checksum; and

c) responsive to a determination that said checksum and said stored checksum do not match, identifying one or more modifications or one or more errors introduced to said file during said attempted upload of said file to said one or more servers;

iv) one or more restore worker modules configured to restore said file, responsive to said file having failed to be written to said one or more server computers during said attempted upload; and

v) one or more meta store worker modules configured to determine that said customized metadata and a metadata comprising one or more locations of said file on said one or more server computers were successfully written to a database on said one or more server computers; and

D) responsive to a completion of said one or more system tasks, delete each of said plurality of query worker modules.

2. The system of claim 1 wherein said file is uploaded to an FTP site, an online file folder or a video-based social media website.

3. The system of claim 1 wherein said request is received via an application programming interface on said one or more server computers, said application programming interface being configured to:

i) manage communications between said application programming interface and a client computer communicatively coupled to said network;

ii) receive, from said client computer, said request to store said file on said one or more server computers and to write said customized metadata about said file to said database;

iii) communicate, to one or more tracker server modules, a plurality of data identifying said file and said customized metadata about said file;

iv) receive, from said one or more tracker server modules, one or more locations, on said one or more server computers, to store said file, said one or more locations having been calculated to maximize computing power for each of said one or more server computers;

v) execute a hypertext transfer protocol request, comprising a put command, to write said file to said one or more locations calculated to maximize computing power for each of said one or more server computers; and

vi) communicate, to said one or more tracker server modules, said one or more locations and a confirmation that said file has been successfully written to said one or more server computers.

4. The system of claim 3 wherein said application programming interface is further configured to:

i) receive, from said client computer, input from a user indicating selection, via a file upload interface on said client computer, of said file; and

ii) receive, from said client computer, a customized metadata about said file.

5. The system of claim 4 wherein said file upload interface further comprises:

i) a file upload dialog comprising any combination of one or more checkboxes, one or more drop-down lists or a plurality of drag and drop technology configured to move said file from said client computer to said one or more server computers;

ii) a text box configured to receive from said user said customized metadata about said file; and

iii) one or more software instructions to send said file, and said customized metadata, from said client computer to said application programming interface.

6. The system of claim 3 wherein said application programming interface is further configured to:

i) receive said file from said client computer;

ii) determine a size of said file upon receipt of said file;

iii) determine whether said size of said file exceeds a predetermined threshold for a file size according to a class for said file and a replication policy for said class; and

iv) responsive to a determination that said file exceeds said predetermined threshold for said file size;

a) determine a file piece size for a plurality of file pieces to be divided from said file, said file piece size being based on said size of said file;

b) divide said file into said plurality of file pieces comprising a file parent piece and one or more file children pieces;

c) manage a storage of said plurality of file pieces to facilitate a retrieval or a streaming of said file;

d) store each of said plurality of file pieces at said one or more locations having been calculated to maximize computing power for each of said one or more server computers; and

e) replicate said plurality of file pieces according to said replication policy determined by said class of said file.

7. The system of claim 3 further comprising said one or more tracker server modules configured to:

i) manage communications between said application programming interface and said one or more server computers;

ii) receive, from said application programming interface, a plurality of data identifying said file, and said customized metadata about said file;

iii) request and receive, from said database, said one or more available locations on said one or more server computers to store said file, said one or more available locations having been calculated to maximize computing power for each of said one or more server computers;

iv) transmit, to said application programming interface, said one or more locations;

v) receive from said application programming interface;

a) confirmation that said file has been successfully written to said one or more server computers; and

b) said one or more locations to which said file was written;

vi) store, in said database, said one or more locations to which said file was written;

vii) store said customized metadata about said file to said database;

viii) initialize said plurality of query worker modules; and

ix) replicate said file among said one or more server computers according to a replication policy determined by a class to which said file belongs.

8. The system of claim 7 further comprising one or more storage node servers comprising:

i) a server cluster of one or more application servers comprising a large distributed hypertext-transfer-protocol-backed data store configured to provide and control a redundancy, a replication and a file retention for said file; and

ii) a cloud storage file system which is;

a) accessed online;

b) configured to store said file;

c) accessible to verify the contents of said cloud storage file system;

d) configured to handle one or more request messages to store or serve a plurality of web content from said application programming interface; and

e) configured to act as a web server to store and serve said file and a plurality of other web content configured to handle hypertext transfer protocol request functionality.

9. The system of claim 8 wherein said one or more storage node servers are configured to run one or more parent processes and one or more child processes comprising:

i) one or more disk manager initializer worker modules configured to format one or more disks used by said one or more storage node servers;

ii) one or more disk manager worker modules configured to;

i) manage the health of said one or more disks and report storage statistics of said one or more disks;

ii) facilitate configuration, remote monitoring and management of said server cluster;

iii) one or more file builder worker modules configured to handle one or more requests to build one or more files on said one or more server computers;

iv) one or more file count worker modules configured to count said one or more files stored on said one or more server computers;

v) one or more file delete worker modules configured to determine said one or more files flagged for deletion and delete said one or more files flagged for deletion;

vi) one or more input output statistic worker modules configured to report the performance or status of said one or more server computers in a regular interval; and

vii) one or more storage sync worker modules configured to search through said one or more server computers and validate that a plurality of data about each of said one or more files is located in said database.

10. The system of claim 3 wherein said one or more process management validator worker modules are further configured to:

i) manage communications from said application programming interface;

ii) ensure that all of said one or more query worker modules are running and functioning properly; and

iii) ensure that all of said one or more servers are operational.

11. The system of claim 1 wherein:

i) said one or more job master worker modules comprise one or more event-based parent processes comprising a message bus configured to manage all communication from one or more applications on a client computer requesting operations to be performed;

ii) said one or more query worker modules comprise one or more child processes created, managed, handled, communicated with, stopped, reset or deleted by said one or more event-based parent processes, each of said one or more child processes comprising software or scripts executed on said one or more server computers and configured to accomplish one or more designated tasks requested by an application programming interface on said one or more server computers; and

iii) said one or more job master worker modules transmits one or more requests to one or more idle query workers for processing wherein, responsive to none of said one or more idle workers being available, said one or more requests are queued in said system queue for later processing by said one or more idle query workers.

12. The system of claim 1 wherein said one or more sync worker modules are further configured to:

i) monitor said one or more system tasks in said one or more system queues;

ii) load balance said one or more system tasks in said one or more system queues; and

iii) ensure that each of one or more files uploaded to said one or more server computers has a valid replication policy.

13. The system of claim 1 wherein said one or more meta store worker modules are further configured to ensure that meta information is stored about each of one or more files stored on said one or more server computers.

14. The system of claim 1 further comprising one or more monitor worker modules configured to interface with said one or more process management validator worker modules to:

i) monitor the health and status of said one or more servers, said one or more query worker modules and said database;

ii) ensure that one or more storage node servers storing one or more files, including said file, are in contact with said network;

iii) verify that each copy of said one or more files is stored in one or more locations stored in said database;

iv) verify that said one or more files exist and are the correct length;

v) responsive to a determination that said one or more files are missing, indicated by no working paths to said one or more server computers within said network, attempt to find said one or more files on any of said one or more server computers;

vi) constantly contact said one or more server computers to determine which of said one or more server computers are available and online; and

vii) detect one or more changes to said one or more query worker modules.

15. The system of claim 1 further comprising one or more reaper and rebalance worker modules configured to:

i) ensure that a plurality of traffic in said one or more system queues does not meet a configurable threshold;

ii) calculate a number of bytes created by said plurality of traffic, and flowing through said network;

iii) responsive to a determination that said number of bytes is less than said configurable threshold, add one or more of said query worker modules to said one or more system queues;

iv) responsive to a determination that said number of bytes is greater than or equal to said configurable threshold, wait until said one or more system queues empty before adding one or more of said query workers to said one or more system queues;

v) ensure one or more files, from a non-functional server among said one or more server computers, are copied to one or more valid locations on said one or more servers to fulfill a replication policy for said one or more files;

vi) delete, from said database, one or more non-valid locations for said non-functional server; and

vii) schedule a replication of said one or more files from said non-functional server.

16. The system of claim 1 further comprising one or more delete worker modules configured to flag one or more files for deletion based on an archiving algorithm, said archiving algorithm being configured to process one or more delete queues among said one or more system queues to permanently remove one or more deleted files, one or more overwritten files or one or more temporary files from said one or more server computers.

17. The system of claim 1 further comprising one or more replication worker modules configured to:

i) automatically check said file against a replication policy;

ii) replicate one or more copies of said file on said one or more server computers according to said replication policy; and

iii) manage a drain and a rebalance of said one or more server computers.

18. The system of claim 17 wherein said replication policy:

i) comprises a desired state comprising a desired count of one or more replicated files for said file, said desired count comprising;

a) a property of a class to which said file belongs; and

b) a definition of how many times one or more files in said class need to be replicated within said one or more server computers to ensure redundancy and to prevent data loss within said one or more files;

ii) requires a predetermined number of valid copies of said one or more replicated files to exist prior to transmitting a confirmation of a successful write of said file,

iii) satisfies the desired state; and

iv) provides and controls a retention of said file.