Verifying and Balancing Server Resources via Stored Usage Data

• US 20130290284A1
• Filed: 06/27/2013
• Published: 10/31/2013
• Est. Priority Date: 05/06/2010
• Status: Active Grant

• Current Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Original Assignee: Go Daddy Operating Company LLC (GoDaddy, Inc.)
• Inventors: Knapp, Adam, Strange, Brent
• Granted Patent: US 9,286,331 B2
• Time in Patent Office: Days
• US Class Current: 707/697
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems and methods of the present invention provide for a user uploading a file and a customized metadata about the file to a file storage system. This information may be forwarded to an application, possibly an API, which may divide the file into a plurality of file chunks if the file is greater than a predetermined file size. One or more tracker servers may check a database to locate optimal file locations for the file or file chunks to be stored within the file storage system. These optimal locations may be returned from the tracker servers to the application/API which may perform an HTTP request to write the file to the file storage system. The file write may be validated and finalized, and the successful write may be displayed to the user on a user interface on a client.

Claims

1. A method, comprising the steps of:

A) storing, by a server computer communicatively coupled to a network;

i) an electronic file;

ii) a file identification for the electronic file;

iii) a stored checksum associated with the file identification; and

iv) at least one copy of the electronic file;

B) identifying, by the server computer, a modification to the electronic file;

C) generating, by the server computer, from the electronic file, a second checksum;

D) determining, by the server computer, whether the second checksum matches the stored checksum; and

E) responsive to a determination that the second checksum does not match the stored checksum;

i) removing the electronic file from the at least one server computer; and

ii) generating a second copy of the electronic file from the at least one copy of the electronic file.

2. The method of claim 1, wherein the stored checksum comprises a plurality of checksum data and is stored within:

i) a database communicatively coupled to the network;

or

ii) a metadata file stored on the at least one server computer and associated with the electronic file via the file identification.

3. The method of claim 2, wherein the checksum data in the database is compared with a second plurality of checksum data generated in the second checksum.

4. The method of claim 1, wherein generating step C) further comprises the steps of:

i) queuing, by the server computer, into a checksum server process queue on the server computer, at least one checksum server process associated with the file identification; and

ii) executing, by the server computer, the at least one checksum server process.

5. The method of claim 1, further comprising the steps of:

F) storing, by the server computer;

i) a metadata file associated with the electronic file and comprising the file identification; and

ii) a metadata;

a) stored in a database communicatively coupled to the network;

b) associated with the electronic file; and

c) comprising a corresponding file identification;

G) determining, by the server computer, whether the file identification and the corresponding file identification comprise asynchronous data; and

I) responsive to a determination that the file identification and the corresponding file identification comprise asynchronous data;

i) removing, by the server computer, the electronic file from the server computer; and

ii) generating, by the server computer, a second copy of the electronic file from the at least one copy of the electronic file.

6. The method of claim 5, wherein the metadata file and the metadata stored in the database comprise:

i) a full path to the electronic file; and

ii) a REST layer to search the database for a data within the full path.

7. The method of claim 1, further comprising the steps of:

F) storing, by the server computer;

i) a metadata file;

a) associated with the electronic file via the file identification; and

b) comprising file data about the electronic file; and

ii) a metadata;

a) stored in a database communicatively coupled to the network;

b) associated with the electronic file via the file identification; and

c) comprising a corresponding file data about the electronic file;

G) identifying, by the server computer, an asynchronous data between the file data and the corresponding file data;

H) identifying, by the server computer, the file data within the metadata file as a master data; and

I) replacing, by the server computer, the asynchronous data in the database with the master data from the metadata file.

8. The method of claim 1, further comprising the steps of:

F) storing, by the server computer, at least one replication policy for replicating at least one electronic file stored on the server computer;

G) receiving, by the server computer, the electronic file to be stored on the server computer;

H) replicating, by the server computer, the at least one copy of the electronic file according to the at least one replication policy;

I) determining, by the server computer, whether the at least one server computer is actively replicating the at least one electronic file according to the at least one replication policy; and

J) responsive to a determination that the at least one server computer is not actively replicating the at least one electronic file according to the at least one replication policy, automatically replicating, but the server computer, the at least one copy of the electronic file to fulfil the at least one replication policy.

9. The method of claim 8, wherein replicating the at least one copy of the electronic file is accomplished using at least one replication server process running on the server computer.

10. The method of claim 8, wherein storing step A) further comprises the steps of:

i) storing, by the server computer, an electronic file division size;

ii) receiving, by the server computer, the electronic file via a data stream, wherein the electronic file comprises an unknown file size;

iii) determining, by the server computer, whether data stream for the electronic file is equal to the electronic file division size; and

iv) responsive to a determination that the data stream for the electronic file is equal to the electronic file division size, storing, by the server computer, a split file of the electronic file division size comprising the stream of the electronic file.

17. The method of claim 1 wherein the server process comprises:

i) writing an electronic file to the first server computer; and

ii) replicating the electronic file on the first server computer.

18. The method of claim 17 wherein the server process of writing the electronic file to the first server computer takes precedence over any other server process in a server process queue.

11. A method, comprising the steps of:

A) storing, by a server computer communicatively coupled to a network, in a database communicatively coupled to the network, a threshold value for a server usage metric;

B) monitoring, by the server computer, the server usage metric for a first server computer and a second server computer;

C) determining, by the server computer, whether the server usage metric for the first server computer exceeds the threshold value;

D) responsive to a determination that the server usage metric for the first server computer exceeds the threshold value;

i) determining, by the server computer, whether the server usage metric for the second server computer exceeds the threshold value; and

ii) responsive to a determination that the server usage metric for the second server computer does not exceed the threshold value, running, by the server computer, on the second server computer, a server process to change the server usage metric.

12. The method of claim 11 wherein the server usage metric comprises a usage load on the first server computer and the second server computer.

13. The method of claim 12 wherein the server process comprises one of a plurality of requested idle server processes within a server process queue tasked with load balancing the usage load.

14. The method of claim 13 wherein the at least one server process is added to the server process queue on the second server computer responsive to the usage load on the first server computer being greater than the usage load on the second server computer.

15. The method of claim 14 wherein at least one additional server process is added to the server process queue on the second server computer if the server usage metric for the second server computer does not exceed the threshold value.

16. The method of claim 11 wherein the server usage metric comprises:

i) a network traffic metric comprising a total bandwidth used on the network;

ii) a disk storage usage metric comprising a percentage of disk space available on the first server computer or the second server computer;

iii) a server process thread metric comprising a total of all server process threads on the first server computer or the second server computer;

or

iv) a server process queue metric comprising a total of all server processes in a server process queue of the first server computer or the second server computer.

19. The method of claim 11, wherein responsive to a determination that the server usage metric for the second server computer exceeds the threshold value:

i) deferring, by the server computer, the server process until at least one server process in a server process queue is removed;

or

ii) removing, by the server computer, from the server process queue, the at least one server process.

20. The method of claim 11, wherein the server process comprises:

i) identifying a network device that is no longer accessible through the network;

ii) deleting from a database communicatively coupled to the network, a location of at least one electronic file stored on the network device; and

iii) replicating, according to a replication policy on the network device, a copy of the at least one electronic file from a second network device accessible through the network.