SYSTEMS AND METHODS FOR AUTOMATIC TRANSFERRING OF BIG DATA COMPUTATIONS FROM CENTRALIZED SYSTEMS TO AT LEAST ONE OF MESSAGING SYSTEMS AND DATA COLLECTION SYSTEMS

• US 20180069925A1
• Filed: 09/08/2017
• Published: 03/08/2018
• Est. Priority Date: 09/08/2016
• Status: Active Application

• Current Assignee: Bigstream Solutions, Inc.
• Original Assignee: Bigstream Solutions, Inc.
• Inventors: LAVASANI, Maysam
• Application Number:
  US15/699,999
• Publication Number:
  US 20180069925A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/28 Databases characterised by their database models, e.g. relational or object models
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system is disclosed that includes machines for performing big data applications. In one example, a centralized system for big data services comprising storage to store data for big data services and a plurality of servers coupled to the storage. The plurality of servers perform at least one of ingest, transform, and serve stages of data. A sub-system has an auto transfer feature to perform program analysis on computations of the data and to automatically detect computations to be transferred from within the centralized system to at least one distributed node that includes at least one of messaging systems and data collection systems.

Claims

1. A centralized system for big data services comprising:

storage to store data for big data services;

a plurality of servers coupled to the storage, the plurality of servers to perform at least one of ingest, transform, and serve stages of data; and

a sub-system having an auto transfer feature to perform program analysis on computations of the data and to automatically detect computations to be transferred from within the centralized system to at least one distributed node that includes at least one of messaging systems and data collection systems.

2. The centralized system of claim 1, wherein the auto transfer feature causes computations of data that are normally performed with the plurality of servers of the centralized system to be transferred and performed with resources of the at least one distributed node including at least one of messaging systems and data collection systems.

3. The centralized system of claim 2, wherein the at least one distributed node receives data from event producers and filters the collected data to generate a reduced set of data.

4. The centralized system of claim 3, wherein the plurality of servers to perform at least one of ingest, transform, and serve stages for the reduced set of data instead of the collected data received by the at least one distributed node to reduce networking congestion for data transferred between the centralized system and the distributed nodes.

5. The centralized system of claim 1, wherein the auto transfer feature causes computations of data that are normally performed with the plurality of servers of the centralized system to be performed with devices of the event producers.

6. The centralized system of claim 5, wherein the devices of the event producers filter the data to produce a reduced set of data that is received by the at least one distributed node.

7. The centralized system of claim 6, wherein the plurality of servers to perform at least one of ingest, transform, and serve stages for the reduced set of data that is filtered by the devices of the event producers to reduce networking congestion for data transferred between the event producers, at least one distributed node, and the centralized system.

8. A machine comprising:

storage to store data; and

an Input/output (I/O) processing unit coupled to the storage, the I/O processing unit having an in-line accelerator that is configured for in-line stream processing of distributed multi stage dataflow based computations including automated controlled intelligence for transferring of data computations from the machine to at least one distributed node including at least one of messaging systems and data collection systems.

9. The machine of claim 8, wherein the in-line accelerator is further configured to perform at least one of ingest, transform, and serve operations.

10. The machine of claim 8, wherein the in-line accelerator is further configured to receive a reduced set of data from at least one of messaging systems and data collection systems that perform the transferred data computations.

11. The machine of claim 8, further comprising:

a general purpose instruction-based processor coupled to the I/O processing unit, wherein the in-line accelerator is configured to perform operations of multiple stages without utilizing the general purpose instruction-based processor.

12. The machine of claim 8, wherein the in-line accelerator is implemented on a Field Programmable Gate Array (FPGA), a many-core, a graphical processing unit (GPU), or an application specific integrated circuit (ASIC).

13. The machine of claim 8, wherein the automated controlled intelligence causes data computations that are normally performed with the machine to be transferred and performed with resources of the at least one distributed node including at least one of messaging systems and data collection systems.

14. The machine of claim 13, wherein the at least one distributed node receives data from event producers and filters the collected data to generate a reduced set of data.

15. The machine of claim 14, wherein the in-line accelerator performs at least one of ingest, transform, and serve stages for the reduced set of data instead of the collected data received by the at least one distributed node to reduce networking congestion for data transferred between the centralized system and the distributed nodes.

16. A computer-implemented method comprising:

automatically detecting computations to be performed on a centralized resource for ingestion and enrichment operations or intelligent extractions;

determining whether any of the detected computations to be performed on the centralized resource can be transferred to a distributed resource of a messaging system or data collection system; and

transferring at least one detected computation of the detected computations to be performed on the centralized resource to a distributed resource of a messaging system or data collection system when possible.

17. The computer-implemented method of claim 16, further comprising:

performing the at least one detected computation that has been transferred to the distributed resource of the messaging system or data collection system to generate a reduced set of data.

18. The computer-implemented method of claim 17, further comprising:

receiving with the centralized resource the reduced set of data from the distributed resource; and

performing at least one of ingest, transform, and serve stages for the reduced set of data instead of the collected data received by the distributed resource to reduce networking congestion for data transferred between the centralized resource and the distributed resource.

19. A distributed resource comprising:

storage to store data for at least one of messaging and data collection services; and

an Input/output (I/O) processing unit coupled to the storage, the I/O processing unit having an in-line accelerator that is configured for in-line stream processing of data computations including automated controlled intelligence for transferring of data computations from a centralized resource to the distributed resource.

20. The distributed resource of claim 19, wherein the automated controlled intelligence causes data computations that are normally performed with the centralized resource to be transferred and performed with the in-line accelerator of the distributed resource.

21. The distributed resource of claim 20, wherein the distributed resource receives data from event producers and filters the collected data to generate a reduced set of data.

22. The distributed resource of claim 20, wherein the distributed resource to transfer the reduced set of data to the centralized resource for additional processing with the reduced set of data reducing networking congestion for data transferred between the centralized resource and the distributed resource.