DATA STREAM CONSOLIDATION IN A SOCIAL NETWORKING SYSTEM FOR NEAR REAL-TIME ANALYSIS

• US 20170142200A1
• Filed: 11/12/2015
• Published: 05/18/2017
• Est. Priority Date: 11/12/2015
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Kodner, Neil A., Sundram, Jason, Arpat, Guven Burc
• Granted Patent: US 10,057,349 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 4/21 for social networking applications

Abstract

Some embodiments include a stream consolidation engine in a social networking system. The stream consolidation engine can receive two or more input data streams (e.g., an activity record data stream and an application service output stream) from the social networking system. The stream consolidation engine can merge an activity record from the activity record data stream with at least a data record from the input data streams (e.g., from the activity record data stream or the application service output stream) to create a conglomerate data record. The stream consolidation engine can supplement the conglomerate data record with asynchronous information from a data storage or derivative data computed based on content in or referenced by the conglomerate data record. The stream consolidation engine can publish the conglomerate data record in a consolidated data stream. The consolidated data stream can be substantially synchronous to at least one of the input data streams.

Claims

1. A computer-implemented method, comprising:

receiving two or more input data streams from a social networking system, wherein the input data streams include an activity record data stream and an application service output stream;

merging an activity record from the activity record data stream with a data record from the input data streams to create a conglomerate data record;

extracting asynchronous information from a data storage device, wherein the asynchronous information corresponds to the activity record and is generated prior to a timestamp of the activity record;

supplementing the conglomerate data record with the extracted asynchronous information; and

publishing the conglomerate data record in a consolidated data stream.

2. The computer-implemented method of claim 1, further comprising, prior to said merging, filtering the activity record data stream to remove one or more activity records that have no media or text content.

3. The computer-implemented method of claim 1, further comprising, piping the activity record data stream to an application service to generate the application service output stream.

4. The computer-implemented method of claim 1, wherein the asynchronous information is user profile information.

5. The computer-implemented method of claim 1, further comprising:

computing, via the application service, derivative data from media or text content in or referenced by the activity record;

generating a derivative data record containing the computed derivative data; and

publishing the generated derivative data record in the application service output stream, wherein the derivative data record is the data record merged with the activity record.

6. The computer-implemented method of claim 1, wherein the input data streams are unbounded data streams that are open-ended sequences of data.

7. The computer-implemented method of claim 1, wherein extracting the asynchronous information includes querying a key-value data cache.

8. The computer-implemented method of claim 1, further comprising selecting the data record to merge with the activity record by identifying matching values between corresponding data fields of the activity record and the data record.

9. The computer-implemented method of claim 1, further comprising selecting the data record to merge with the activity record by identifying a reference number or symbol in the data record that references the activity record or a reference number or symbol in the activity record that references the data record.

10. The computer-implemented method of claim 1, wherein the activity record is the most recent activity record in the activity record data stream.

11. The computer-implemented method of claim 1, further comprising:

computing derivative data from content in or referenced by the conglomerate data record; and

supplementing the derivative data into the conglomerate data record.

12. The computer-implemented method of claim 11, wherein said supplementing includes inserting the derivative data in an existing data field of the conglomerate data record as a new entry or as a new data field of the conglomerate data record.

13. The computer-implemented method of claim 11, wherein the derivative data indicates overall sentiment of text content in the activity record.

14. The computer-implemented method of claim 1, wherein the activity record data stream is a heterogeneous data stream containing activity records of different data structures, and the data record merged with the activity record is from the activity record data stream and of a different type from the activity record.

15. The computer-implemented method of claim 1, further comprising:

monitoring the input data streams for a potential data record to merge with the activity record during a preset synchronization period starting from when the activity record is received.

16. The computer-implemented method of claim 1, further comprising:

determining that the activity record or the data record satisfies a preset condition; and

wherein said merging occurs only when the preset condition is satisfied.

17. A computer readable data memory storing computer-executable instructions that, when executed by a computer system, cause the computer system to perform a computer-implemented method, the instructions comprising:

instructions for receiving two or more input data streams from a social networking system, wherein the input data streams include an activity record data stream and an application service output stream;

instructions for merging the activity record data stream and the application service output stream into a consolidated data stream by at least merging an activity record from the activity record data stream with a data record from the input data streams to create a conglomerate data record;

instructions for computing derivative data from content in or referenced by the conglomerate data record;

instructions for supplementing the derivative data into the conglomerate data record; and

instructions for publishing the conglomerate data record in the consolidated data stream.

18. The computer readable data memory of claim 17, wherein the instructions further comprising:

instructions for piping at least a portion of the consolidated data stream to a chatter analysis engine; and

instructions for generating, by the chatter analysis engine, a dashboard dynamically configurable to discover a statistical attribute of digital chatter represented by the consolidated data stream that is occurring in the social networking system in real-time.

19. A social networking system, comprising:

an application service engine configured to process an activity record data stream to generate an application service output stream;

a stream consolidation engine configured to;

receive two or more input data streams from a social networking system, wherein the input data streams include the activity record data stream and the application service output stream;

merge the activity record data stream and the application service output stream into a consolidated data stream by merging an activity record from the activity record data stream with a data record from the input data streams to create a conglomerate data record; and

publish the conglomerate data record in the consolidated data stream; and

a real-time chatter analysis engine configured to re-route data records in the consolidated stream to a digital chatter analysis engine according to content concept identifiers in the data records that correspond to content concept identifiers associated with the digital chatter analysis engine.

20. The social networking system of claim 19, wherein the application service is an image tagger engine configured to analyze subject matter of media content of an activity record or a topic tagger engine configured to analyze subject matter of text content of an activity record.