DATA STREAM SPLITTING FOR LOW-LATENCY DATA ACCESS

• US 20140214752A1
• Filed: 01/31/2013
• Published: 07/31/2014
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Hwang, Eric, Rash, Samuel, Borthakur, Dhrubajyoti, Shao, Zheng
• Granted Patent: US 10,223,431 B2
• Time in Patent Office: Days
• US Class Current: 707/600
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

Techniques for facilitating and accelerating log data processing by splitting data streams are disclosed herein. The front-end clusters generate large amount of log data in real time and transfer the log data to an aggregating cluster. The aggregating cluster is designed to aggregate incoming log data streams from different front-end servers and clusters. The aggregating cluster further splits the log data into a plurality of data streams so that the data streams are sent to a receiving application in parallel. In one embodiment, the log data are randomly split to ensure the log data are evenly distributed in the split data streams. In another embodiment, the application that receives the split data streams determines how to split the log data.

Claims

1. A method comprising:

producing, at a plurality of front end servers, log data based on real-time user activities;

transmitting the log data to an aggregating server;

aggregating the log data at the aggregating server;

splitting the log data into a plurality of log data streams; and

feeding the log data streams to at least one back end server in parallel.

2. The method of claim 1, wherein the step of splitting comprises:

splitting the log data randomly so that the log data are evenly distributed to a plurality of log data streams.

3. The method of claim 1, further comprising:

receiving an instruction from the back end server regarding how to split the log data into the plurality of log data streams.

4. The method of claim 1, wherein the log data includes a plurality of log data entries, each log data entry includes an application identification and a category field.

5. The method of claim 4, further comprising:

for each entry of the log data, calculating a bucket number by a hash function of the application identification and the category field modulo a total number of buckets, wherein the total number of buckets is a total number of the plurality of log data streams; and

assigning that entry of the log data to a log data stream identified by the bucket number.

6. The method of claim 5, wherein the category field includes a high level description of an intended destination of the log data entry.

7. The method of claim 5, wherein the application identification identifies a data consuming application for processing the log data entry.

8. The method of claim 4, wherein the log data includes a plurality of log data entries;

and

the method further comprising;

for each log data entry, randomly generating a integer from 1 to a total number of buckets, wherein the total number of buckets is a total number of the plurality of log data streams; and

assigning that log data entry to a log data stream identified by the bucket number.

9. The method of claim 4, wherein the total number of buckets is determined by a number of back end servers that are available to receive the log data streams and a number of connections that each back end server is capable of handling.

10. The method of claim 4, wherein the total number of buckets is instructed by a data consuming application running on at least one back end server.

11. The method of claim 9, wherein the back end servers are equally loaded when the back end servers receive and process the log data streams.

12. The method of claim 1, further comprising:

examining prefixes of entries of the log data to determine the log data stream that the entries are assigned to.

13. The method of claim 1, further comprising:

sending the log data to a data warehouse; and

processing the log data at the data warehouse so that the data warehouse can respond to data queries based on the processed log data.

17. The method of claim 4, wherein the total number of buckets is instructed by a data consuming application running on one or more back end servers, and the total number of buckets is determined by a number of the back end servers that are available to receive the log data streams and a number of connections that each back end server of the back end servers is capable of handling.

14. A computer-implemented system, comprising:

a plurality of front end servers configured for producing log data based on real-time user activities; and

at least one aggregating server configured for aggregating the log data received from at least some of the front end servers, the aggregating server being connected with at least some of the front end servers via a network;

wherein the aggregating server includes a data staging area configured for staging the log data, and the aggregating server is configured for splitting the log data into a plurality of log data streams so that one or more back end servers can retrieve the log data streams in parallel.

15. The computer-implemented system of claim 14, wherein the log data includes a plurality of log data entries, each log data entry includes an application identification and a category field;

and

wherein the aggregating server is further configured for;

for each entry of the log data, calculating a bucket number by a hash function of the application identification and the category field modulo a total number of buckets, wherein the total number of buckets is a total number of the plurality of log data streams, and

assigning that entry of the log data to a log data stream identified by the bucket number.

16. The computer-implemented system of claim 14, wherein the log data includes a plurality of log data entries;

and

wherein the aggregating server is further configured for;

for each log data entry, randomly generating a integer from 1 to a total number of buckets, wherein the total number of buckets is a total number of the plurality of log data streams; and

assigning that log data entry to a log data stream identified by the bucket number.

18. An aggregating server, comprising:

a processor;

a network interface, coupled to the processor, through which the aggregating server can communicate with a plurality of front end servers;

a data storage including a data staging area; and

a memory storing instructions which, when executed by the processor, cause the aggregating server to perform a process including;

receiving log data from the front end servers, wherein the front end servers produce the log data based on real-time user activities,

aggregating the log data,

staging the log data at the data staging area, and

splitting the log data into a plurality of log data streams so that one or more back end servers can retrieve the log data streams in parallel.

19. The aggregating server of claim 18, wherein the log data includes a plurality of log data entries, each log data entry includes an application identification and a category field;

and

wherein the process further includes;

for each entry of the log data, calculating a bucket number by a hash function of the application identification and the category field modulo a total number of buckets, wherein the total number of buckets is a total number of the plurality of log data streams, and

assigning that entry of the log data to a log data stream identified by the bucket number.

20. The aggregating server of claim 18, wherein the log data includes a plurality of log data entries;

and

wherein the process further includes;

for each log data entry, randomly generating a integer from 1 to a total number of buckets, wherein the total number of buckets is a total number of the plurality of log data streams, and

assigning that log data entry to a log data stream identified by the bucket number.