Multi-level data staging for low latency data access

• US 9,609,050 B2
• Filed: 01/31/2013
• Issued: 03/28/2017
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Rash, Samuel, Borthakur, Dhruba, Shao, Zheng, Shen, Guanghao
• Primary Examiner(s): Tiv, Backhean
• Application Number:
  US13/756,304
• Publication Number:
  US 20140215007A1
• Time in Patent Office: 1,517 Days
• Field of Search: 709/217, 709/224
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/567 Integrating service provisioning from a plurality of service providers

Abstract

Techniques for facilitating and accelerating log data processing are disclosed herein. The front-end clusters generate a large amount of log data in real time and transfer the log data to an aggregating cluster. When the aggregating cluster is not available, the front-clusters write the log data to local filers and send the data when the aggregating cluster recovers. The aggregating cluster is designed to aggregate incoming log data streams from different front-end servers and clusters. The aggregating cluster further sends the aggregated log data stream to centralized NFS filers or a data warehouse cluster. The local filers and the aggregating cluster stage the log data for access by applications, so that the applications do not wait until the data reach the centralized NFS filers or data warehouse cluster.

Claims

1. A method, comprising:

producing, at a plurality of front end servers, log data based on real-time user activities;

in an event that an aggregating server is unavailable, staging the log data at a front end staging area in at least one of the plurality of front end servers for providing a back end server real-time access to the log data;

in an event that the aggregating server is available, transmitting the log data to the aggregating server;

aggregating the log data at the aggregating server;

staging the log data at the aggregating server and providing a back end server with real-time access to the log data;

after providing the back end server real-time access to the log data, sending the log data from the aggregating server to a data warehouse; and

processing the log data at the data warehouse so that the data warehouse can respond to a data query based on the processed log data.

2. The method of claim 1, further comprising:

feeding the log data from the aggregating server directly to the back end server in real time.

3. The method of claim 1, further comprising:

feeding the log data from the front end staging area to the back end server in real time.

4. The method of claim 1, wherein the step of transmitting comprises:

when the aggregating server is available, transmitting the log data to the aggregating server.

5. The method of claim 1, wherein the plurality of front end servers further provide web content to users.

6. The method of claim 1, wherein the step of aggregating comprises:

reorganizing the log data by combining files of the log data at the aggregating server.

7. The method of claim 1, further comprising:

compressing the log data at the aggregating server.

8. The method of claim 1, further comprising:

removing the log data from the aggregating server after a predetermined time period.

9. The method of claim 1, further comprising:

dividing the log data into a plurality of tiers;

and wherein the step of transmitting comprises;

transmitting the log data in a top tier of the plurality of tiers to an aggregating server, wherein the log data in the top tier has a higher priority to be transmitted to the data warehouse and the back end server.

10. The method of claim 1, further comprising:

receiving an instruction from the back end server regarding a staging location for the log data.

11. The method of claim 1, wherein the back end server runs a data consuming application that retrieves the log data in real time and processes the retrieved log data to generate an analysis result.

12. A computer-implemented system, comprising:

a plurality of front end servers configured to produce log data based on real-time user activities;

at least one aggregating server configured to aggregate the log data received from at least some of the front end servers, the aggregating server being connected with at least some of the front end servers via a network;

wherein the aggregating server includes a data staging area configured to stage the log data at the aggregating server and providing a back end server with real-time access to the log data;

a data warehouse, wherein the aggregating server is further configured to periodically send the log data to the data warehouse after providing the back end server real-time access to the log data, and the data warehouse is configured to process the log data and to respond a data query based on the processed log data; and

at least one second level aggregating server configured for aggregating the log data received from the aggregating server, the second level aggregating server being connected with the aggregating server, wherein the second level aggregating server includes a second level data staging area configured for staging the log data so that the back end server can access the log data in real time.

13. The computer-implemented system of claim 12, wherein at least one front end server of the front end servers includes a front end staging area, the front end staging area is configured for staging the log data so that the back end server can access the log data from the front end staging area in real time.

14. The computer-implemented system of claim 13, wherein the front end staging area starts to stage the log data and feed the log data to the back end server when the aggregating server is unavailable.

15. The computer-implemented system of claim 12, wherein the back end server can request to retrieve the log data in real from either the aggregating server or the second level aggregating server, depending on which of the aggregating server or the second level aggregating server is closer to the back end server in a network topology.

16. An aggregating server, comprising:

a processor;

a network interface, coupled to the processor, through which the aggregating server can communicate with a plurality of front end servers;

a data storage including a data staging area;

a memory storing instructions which, when executed by the processor, cause the aggregating server to perform a process including;

receiving log data from the front end servers, wherein the front end servers produce the log data based on real-time user activities,

aggregating the log data, and

staging the log data at the data staging area of the aggregating server and providing a back end server with real-time access to the log data;

splitting the log data into multiple log data streams based on hash values calculated based on entries of the log data; and

after providing the back end server real-time access to the log data, sending the log data from the aggregating server to the data warehouse, wherein the data warehouse processes the log data and responds to data queries based on the processed log data.

17. The aggregating server of claim 16, wherein the front end servers stage the log data at front end staging area of the front end servers when the aggregating server is unavailable.