Verifying and balancing server resources via stored usage data

• US 9,286,331 B2
• Filed: 06/27/2013
• Issued: 03/15/2016
• Est. Priority Date: 05/06/2010
• Status: Active Grant

• Current Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Original Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Inventors: Knapp, Adam, Strange, Brent
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Hoffler, Raheem
• Application Number:
  US13/928,664
• Publication Number:
  US 20130290284A1
• Time in Patent Office: 992 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems and methods of the present invention provide for a user uploading a file and a customized metadata about the file to a file storage system. This information may be forwarded to an application, possibly an API, which may divide the file into a plurality of file chunks if the file is greater than a predetermined file size. One or more tracker servers may check a database to locate optimal file locations for the file or file chunks to be stored within the file storage system. These optimal locations may be returned from the tracker servers to the application/API which may perform an HTTP request to write the file to the file storage system. The file write may be validated and finalized, and the successful write may be displayed to the user on a user interface on a client.

Claims

1. A method, comprising the steps of:

receiving, by a server computer communicatively coupled to a network, a request to store a file, a customized metadata about the file, a stored checksum associated with the file and at least one file copy on the server computer;

populating, by the server computer, at least one system queue on the server computer with at least one system task to write the file, the customized metadata, the stored checksum and the at least one file copy to the server computer;

generating at runtime, by the server computer;

at least one query worker module configured to load balance the at least one system task; and

at least one checksum worker module configured to;

identify a modification to the file;

generate, from the file, a checksum;

determine whether the checksum matches the stored checksum; and

responsive to a determination that the checksum does not match the stored checksum;

remove the file from the server computer; and

generate an additional file copy from the at least one file copy; and

responsive to a completion of the at least one system task, deleting, by the sever computer, the at least one query worker module and the at least one checksum worker module.

2. The method of claim 1, wherein the stored checksum comprises a plurality of checksum data and is stored within:

a database communicatively coupled to the network;

or

a metadata file stored on the at least one server computer and associated with the file via a file identification.

3. The method of claim 2, wherein the plurality of checksum data in the database is compared with a second plurality of checksum data generated in the checksum.

4. The method of claim 1, wherein generating the at least one query worker module and the at least one checksum worker module further comprises the steps of:

queuing, by the server computer, into a checksum server process queue on the server computer, at least one checksum server process associated with the file; and

executing, by the server computer, the at least one checksum server process.

5. The method of claim 1, further comprising the steps of:

storing, by the server computer;

a metadata file associated with the file and comprising a file identification; and

a metadata;

stored in a database communicatively coupled to the network;

associated with the file; and

comprising a corresponding file identification;

determining, by the server computer, whether the file identification and the corresponding file identification comprise asynchronous data; and

responsive to a determination that the file identification and the corresponding file identification comprise asynchronous data;

removing, by the server computer, the file from the server computer; and

generating, by the server computer, a second copy of the file from the at least one file copy.

6. The method of claim 5, wherein the metadata file and the metadata stored in the database comprise:

a full path to the file; and

a REST layer to search the database for a data within the full path.

7. The method of claim 1, further comprising the steps of:

storing, by the server computer;

a metadata file;

associated with the file via a file identification; and

comprising a file data about the file; and

a metadata;

stored in a database communicatively coupled to the network;

associated with the file via the file identification; and

comprising a corresponding file data about the file;

identifying, by the server computer, an asynchronous data between the file data and the corresponding file data;

identifying, by the server computer, the file data within the metadata file as a master data; and

replacing, by the server computer, the asynchronous data in the database with the master data from the metadata file.

8. The method of claim 1, further comprising the steps of:

storing, by the server computer, at least one replication policy for replicating at least one file stored on the server computer;

receiving, by the server computer, the file to be stored on the server computer;

replicating, by the server computer, the at least one file copy according to the at least one replication policy;

determining, by the server computer, whether the server computer is actively replicating the at least one file according to the at least one replication policy; and

responsive to a determination that the server computer is not actively replicating the at least one file according to the at least one replication policy, automatically replicating, by the server computer, the at least one the copy to fulfill the at least one replication policy.

9. The method of claim 8, wherein replicating the at least one file copy is accomplished using at least one replication server process running on the server computer.

10. The method of claim 8, wherein populating the at least one system queue further comprises the steps of:

storing, by the server computer, a file division size;

receiving, by the server computer, the file via a data stream, wherein the file comprises an unknown file size;

determining, by the server computer, whether the data stream for the file is equal to the file division size; and

responsive to a determination that the data stream for the file is equal to the electronic file division size, storing, by the server computer, a split file of the file division size comprising the data stream of the file.

11. A method, comprising the steps of:

receiving, by a server computer communicatively coupled to a network a request to store a file, a customized metadata about the file and a threshold value for a server usage metric on the server computer;

populating, by the server computer, at least one system queue on the server computer with at least one system task to write the file, the customized metadata and the threshold value to the server computer;

generating at runtime, by the server computer, at least one query worker module configured to load balance the at least one system task by;

monitoring the server usage metric for a first server computer and a second server computer;

determining whether the server usage metric for the first server computer exceeds the threshold value;

responsive to a determination that the server usage metric for the first server computer exceeds the threshold value, generating, at runtime, at least one rebalance worker module configured to;

determine whether the server usage metric for the second server computer exceeds the threshold value; and

responsive to a determination that the server usage metric for the second server computer does not exceed the threshold value, run, on the second server computer, a server process to change the server usage metric; and

responsive to a completion of the at least one system task, deleting, by the server computer, the at least one query worker module and the at least one rebalance worker module.

12. The method of claim 11 wherein the server usage metric comprises a usage load on the first server computer and the second server computer.

13. The method of claim 12 wherein the server process comprises one of a plurality of requested idle server processes within a server process queue tasked with load balancing the usage load.

14. The method of claim 13 wherein the server process is added to the server process queue on the second server computer responsive to the usage load on the first server computer being greater than the usage load on the second server computer.

15. The method of claim 14 wherein at least one additional server process is added to the server process queue on the second server computer if the server usage metric for the second server computer does not exceed the threshold value.

16. The method of claim 11 wherein the server usage metric comprises:

a network traffic metric comprising a total bandwidth used on the network;

a disk storage usage metric comprising a percentage of disk space available on the first server computer or the second server computer;

a server process thread metric comprising a total of all server process threads on the first server computer or the second server computer;

or

a server process queue metric comprising a total of all server processes in a server process queue of the first server computer or the second server computer.

17. The method of claim 11 wherein the server process comprises:

writing a file to the first server computer; and

replicating the file on the first server computer.

18. The method of claim 17 wherein the server process of writing the file to the first server computer takes precedence over any other server process in a server process queue.

19. The method of claim 11, wherein responsive to a determination that the server usage metric for the second server computer exceeds the threshold value:

deferring, by the server computer, the server process until at least one server process in a server process queue is removed;

or

removing, by the server computer, from the server process queue, the at least one server process.

20. The method of claim 11, wherein the server process comprises:

identifying a network device that is no longer accessible through the network;

deleting, from a database communicatively coupled to the network, a location of at least one file stored on the network device; and

replicating, according to a replication policy on the network device, a copy of the at least one file from a second network device accessible through the network.