APPARATUS, SYSTEMS, AND METHODS FOR BATCH AND REALTIME DATA PROCESSING

• US 20140289188A1
• Filed: 03/14/2014
• Published: 09/25/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Foursquare Labs, Inc.
• Original Assignee: Factual Incorporated (Foursquare Labs, Inc.)
• Inventors: SHIMANOVSKY, Boris, RANA, Ahad, KOK, Chun
• Granted Patent: US 9,317,541 B2
• Time in Patent Office: Days
• US Class Current: 707/609
• CPC Class Codes: G05B 13/0265 the criterion being a learning criterion
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/282 Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0282 Rating or review of business operators or products
• CPC Class Codes: G06Q 50/01 : Social networking
• CPC Class Codes: H04L 41/14 : Network analysis or design
• CPC Class Codes: H04W 16/00 : Network planning, e.g. coverage or traffic planning tools; Network deployment, e.g. resource partitioning or cells structures
• CPC Class Codes: H04W 16/24 : Cell structures
• CPC Class Codes: H04W 16/30 : Special cell shapes, e.g. doughnuts or ring cells
• CPC Class Codes: H04W 16/32 : Hierarchical cell structures
• CPC Class Codes: H04W 4/02 : Services making use of location information
• CPC Class Codes: H04W 4/021 : Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/025 : using location based information parameters
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/50 : Service provisioning or reconfiguring
• CPC Class Codes: H04W 64/00 : Locating users or terminals or network equipment for network management purposes, e.g. mobility management
• CPC Class Codes: H04W 64/003 : locating network equipment
• CPC Class Codes: H04W 76/38 : triggered by timers
• CPC Class Codes: H04W 8/08 : Mobility data transfer
• CPC Class Codes: H04W 8/16 : selectively restricting mobility data tracking
• CPC Class Codes: H04W 8/18 : Processing of user or subscriber data, e.g. subscribed services, user preferences or user profiles; Transfer of user or subscriber data
• CPC Class Codes: H04W 88/00 : Devices specially adapted for wireless communication networks, e.g. terminals, base stations or access point devices
• CPC Class Codes: H04W 88/02 : Terminal devices

Abstract

A traditional data processing system is configured to process input data either in batch or in real-time. On one hand, a batch data processing system is limiting because the batch data processing often cannot take into account any data received during the batch data processing. On the other hand, a real-time data processing system is limiting because the real-time system often cannot scale. The real-time data processing system is often limited to dealing with primitive data types and/or a small amount of data. Therefore, it is desirable to address the limitations of the batch data processing system and the real-time data processing system by combining the benefits of the batch data processing system and the real-time data processing system into a single data processing system.

Claims

1. A computing system for generating a summary data of a set of data, the computing system comprising:

one or more processors configured to run one or more modules stored in non-tangible computer readable medium, wherein the one or more modules are operable to;

receive a first set of data and a second set of data, wherein the first set of data comprises a larger number of data items compared to the second set of data;

process the first set of data to format the first set of data into a first structured set of data;

generate a first summary data using the first structured set of data by operating rules for summarizing the first structured set of data, and store the first summary data in a data store;

process the second set of data to format the second set of data into a second structured set of data;

generate a second summary data based on the first structured set of data and the second structured set of data by operating rules for summarizing the first structured set of data and the second structured set of data;

determine a difference between the first summary data and the second summary data; and

update the data store based on the difference between the first summary data and the second summary data.

2. The computing system of claim 1, wherein the second set of data comprises real-time data submissions, and the one or more modules are operable to process the second set of data to format the second set of data into the second structured set of data in response to receiving the second set of data.

3. The computing system of claim 1, wherein the one or more modules are operable to process the first set of data to format the first set of data into the first structured set of data at a first time interval, which is substantially longer than a second time interval at which the second set of data is formatted into the second structured set of data.

4. The computing system of claim 1, wherein each of the first summary data and the second summary data comprises an entity identifier and a value associated with the entity identifier, and wherein the one or more modules are operable to determine the difference between the first summary data and the second summary data by:

determining that the first summary data and the second summary data include an identical entity identifier, and

comparing values associated with the identical entity identifiers in the first summary data and the second summary data.

5. The computing system of claim 4, wherein the one or more modules are further operable to provide the difference between the first summary data and the second summary data to other authorized computing systems.

6. The computing system of claim 5, wherein the one or more modules are further operable to provide the difference to other authorized computing systems via an application programming interface.

7. The computing system of claim 5, wherein the one or more modules are further operable to provide the difference to other authorized computing systems as a file.

8. The computing system of claim 1, wherein the one or more modules are operable to:

combine at least the first set of data and the second set of data to generate a third set of data;

process the third set of data to format the third set of data into a third structured set of data based on new rules for formatting a set of data; and

generate a third summary data using the third structured set of data.

9. The computing system of claim 8, wherein the first set of data and the third set of data each includes a first data element, and wherein the first data element is associated with a first entity in the first summary data identified by the first entity identifier, wherein the first data element is associated with a second entity in the third summary data, and wherein the one or more modules are further operable to associate the first entity identifier to the second entity in the third summary data so that the first data element maintains its association with the first entity identifier in the third summary data.

10. The computing system of claim 1, wherein the first structured set of data comprises a grouping of data items based on an entity identifier associated with the data items.

11. The computing system of claim 1, wherein the computing system comprises at least one server in a data center.

12. The computing system of claim 1, wherein the data store comprises a plurality of database systems, each of which is associated with a view, and wherein the one or more modules are operable to select one of the plurality of database systems in response to a query based on the view associated with the query.

13. The computing system of claim 1, wherein the one or more modules are operable to identify a third set of data received after the generation of the second summary data;

generate a third summary data based on the third set of data, the first structured set of data, and the second structured set of data by operating rules for summarizing the first structured set of data, the second structured set of data, and the third summary data;

determine a difference between the second summary data and the third summary data; and

update the data store based on the difference between the second summary data and the third summary data.

14. A method for generating a summary data of a set of data, the method comprising:

receiving, at an input module operating on a processor of a computing system, a first set of data and a second set of data, wherein the first set of data comprises a larger number of data items compared to the second set of data;

processing, at a first input processing module of the computing system, the first set of data to format the first set of data into a first structured set of data;

generating, at a first summary generation module of the computing system, a first summary data using the first structured set of data by operating rules for summarizing the first structured set of data;

maintaining the first summary data in a data store in the computing system;

processing, at a second input processing module of the computing system, the second set of data to format the second set of data into a second structured set of data;

generating, at a second summary generation module of the computing system, a second summary data using the first structured set of data and the second structured set of data by operating rules for summarizing the first structured set of data and the second structured set of data;

determining, at a difference generation module of the computing system, a difference between the first summary data and the second summary data; and

updating, by the computing system, the data store based on the difference between the first summary data and the second summary data.

15. The method of claim 14, wherein receiving the second set of data comprises receiving real-time data submissions, and wherein processing the second set of data to format the second set of data into the second structured set of data comprises processing the second set of data in response to receiving the second set of data.

16. The method of claim 14, wherein each of the first summary data and the second summary data comprises an entity identifier and a value associated with the entity identifier, and wherein determining the difference between the first summary data and the second summary data comprises:

determining that the first summary data and the second summary data include an identical entity identifier, and

comparing values associated with the identical entity identifiers in the first summary data and the second summary data.

17. The method of claim 14, further comprising:

combining at least the first set of data and the second set of data to generate a third set of data,

processing the third set of data to format the third set of data into a third structured set of data based on new rules for formatting a set of data; and

generating a third summary data using the third structured set of data.

18. The method of claim 14, wherein a first data element is a part of the first set of data and the third set of data, and is associated with a first entity, identified by the first entity identifier, in the first summary data, wherein the first data element is associated with a second entity in the third summary data, and wherein the method further comprises associating the first entity identifier to the second entity in the third summary data so that the first data element maintains its association with the first entity identifier in the third summary data.

19. The method of claim 14, wherein the computing system comprises at least one server in a data center.

20. A computer program product, tangibly embodied in a non-transitory computer-readable storage medium, the computer program product including instructions operable to cause a data processing system to:

receive a first set of data and a second set of data, wherein the first set of data comprises a larger number of data items compared to the second set of data;

process the first set of data to format the first set of data into a first structured set of data;

generate a first summary data using the first structured set of data by operating rules for summarizing the first structured set of data, and store the first summary data in a data store;

process the second set of data to format the second set of data into a second structured set of data;

generate a second summary data using the first structured set of data and the second structured set of data by operating rules for summarizing the first structured set of data and the second structured set of data;

determine a difference between the first summary data and the second summary data; and

update the data store based on the difference between the first summary data and the second summary data.

21. The computer program product of claim 20, wherein the second set of data comprises real-time data submissions, and wherein the instructions are operable to cause the data processing system to process the second set of data to format the second set of data into the second structured set of data in response to receiving the second set of data.

22. The computer program product of claim 20, wherein each of the first summary data and the second summary data comprises an entity identifier and a value associated with the entity identifier, and wherein the instructions are operable to cause the data processing system to determine the difference between the first summary data and the second summary data by:

determining that the first summary data and the second summary data include an identical entity identifier, and

comparing values associated with the identical entity identifiers in the first summary data and the second summary data.

23. The computer program product of claim 20, wherein the instructions are operable to cause the data processing system to:

combine at least the first set of data and the second set of data to generate a third set of data,

process the third set of data to format the third set of data into a third structured set of data based on new rules for formatting a set of data; and

generate a third summary data using the third structured set of data.